I don’t see the charging circuit of the LiPo battery on the schematic.
If you plan to use the battery charging chip of the Xiao nRF52840, you can’t use a switch to disconnect the battery.
As most embedded systems operate, the whole circuit is always on but the nRF52840 enters deep sleep. Use a temporary push button connected to an interrupt-capable GPIO as trigger to wake it up.