The maximum SPI clock for XIAO_SAMD21 is 12 MHz.
I checked with a digital oscilloscope, and SPI transfers byte-data with 12MHz clock, but only once every 5.25uS. The real throughput is up to 190 kByte/sec.
There is also the problem in SPI.h. You have to change line 57 from 12000000 to 24000000 to get the maximum 12MHz. In your sketch, it might be better to explicitly set the SPI clock to 12MHz, as it might be set to 1/2 if you use the default.