Hi there,
So pretty sure OP Was using just base Nrf54840 Xiao and NOT the “Sense” version. so no IMU or PDM… Plenty O’ Room…
WARNING …
I can’t emphasize this enough, look VERY, VERY closely at that end of the board. There LIVES a LONE resistor, probably the smallest Known to man I’m betting it’s almost not visible to the Naked eyeball,
so close to the edge , it should be Illegal 
Beware of it and the traces around it. 
GL mr. Phelps,
You have been prepared.
HTH
GL 
PJ 
it’s to the right or left of the PDM footprint 