Hi there,
Yea, if it were only that easy…LOL
The Xiao has built in charge control chip, so if you use the usb for external connection to power it. The Xiao charges and manages the battery for you.(provided you use the BATT pads on the bottom of the PCB for connection. It doesn’t do that if you use the 5V pin connection.
HTH
GL 
PJ