I like the idea of expansion boards but I don’t think I would go too big. For me the reason I picked the Xiao was because it was super small. I would probably pick another board if I was doing a more complex project. In addition, the small amount of flash memory on the Xiao makes it really hard to do complicated projects as you quickly run out of memory for library files (in my project I could only just get the HID library on it - didn’t even have room for the neopixel library).
I would take a look at all the connectors you’ve added and then see if you can actually have room to include all of the libraries on the Xiao. If not, thats probably a good sign that you need to reduce the amount of functionality it has.
PS… if you released a Xiao+ with more memory it would be my go to dev board for SO MANY of my projects that I currently use adafruit products for.