VL805 PCIe to USB

I am developing a custom PCIe to USB board to enable the USB hub controller for my Raspberry pi CM4. I am having trouble booting when I enabled the firmware loading to vl805 via PCIe in the RPI boot config. I am wondering if anyone had a similar issue. The boot config allows the CM4 to load vl805 through PCIe rather than from the external EEPROM via SPI. To test the configuration, I de-soldered the EEPROM on Dual Gigabit Ethernet Carrier Board. It successfully loaded the firmware through PCIe and USB ports were enabled. However, on my custom board, the CM4 fails to boot up and terminates with some error code.

here is the link to my schematic. The schematic is based on the Dual Gigabit Ethernet Carrier Board.