So I ported the code back to a Pro Micro and it works as it should; no timing issues whatsoever. The only changes I made were a couple of pin assignments to support the interrupts and a PWM output and to go back to using delay() instead of my Pause function which I gather is blocking the same as delay() would anyway.
My reason for wanting to go with the XIAO was the size factor. The Pro Micro is double the size of the XIAO.