@SeeedUser - I have been using 5V connected directly to the VBus pin (14) and GND (13) of my XIAO ESP32S3 (and others) without any issues. My 5V “power source” limits voltage injected back (eg a diode).
If the battery is low, it gets charged. If I connect USB the voltage from the USB doesn’t affect my 5V power supply other than reduce its current consumption.
This is dependent on the USB voltage that is supplied (in most cases it’s more than my “5V” supply from my solar charger).
If I don’t use the USB, I don’t include the diode. In any case, the ESP32S3 just chugs along regardless.