Had this problem too and was only confused by this thread, so here’s what I ultimately figured out:
Seeed XIAO does have internal pullups and pulldowns on all pins
The regular arduino analogRead() function will turn them off while reading, so you have to use the SAMD21 registers to configure and read your ADC.
As determined in this thread, digitalRead() works fine with pullups and pulldowns.
Here is the help thread that helped me: Arduino Zero Pull-Up and ADC reading - Arduino Zero - Arduino Forum