That was it! I was using espressif BSP 2.0.17. It turns out that only v2.0.8 will work with these displays.
Thank you for the final piece of the puzzle. The Xiao wiki really needs to get updated with this specific information.
Thanks again @PJ_Glasso