I had same problem,so I asked about this issue to seeed directry.
thay said that
my question
This diode is not found on the XIAO PCB.
current flows back from the + 5V terminal side to the USB connector.
Seeed’s answer
I checked our PCB update log and in the new version xiao we removed this diode in order to get more current.
So I removed the 3.3V regulator from the board and installed an external regulator.
This made it easier to control the power supply.