Hi there,
So I don’t like the schematic you show,
How can you connect 3.3V INTO the Xiao when it’s an OUTPUT?
here is the chunk from the schematic
I think hook the Servo to 5V only too… disconnect the 3.3v wire and just use the USB to power the Xiao and be sure the grounds are connected.
it will work I think. I have some stuff I’ll have to set something up, be sure batteries are fully charged. 
HTH
GL 
PJ