I have I think the same issue, but my view is the compile software for the XIAO is broken.
I can use the INPUT_PULLUP in pinmode for D0 and D1 and they work fine with buttons.
D2 is another story. INPUT and an externally attached pullup work fine, but the INPUT_PULLUP with D2 causes the compilation to fail in different random places. The rare compile that completes does not act like it works.
The chip’s specification appears to show all pins should have the internal pullup available, so this must be a software issue from SEEED studio.