The XIAO_5V pin must be used as either a 5V input pin or a 5V output pin. When USB is connected, the XIAO_5V pin functions as a 5V output pin, so do not connect an external 5V power source. When USB is not connected, the XIAO_5V pin functions as a 5V input pin, allowing you to connect an external power source to supply power to the XIAO.
How is your wiring configured?