Hi there,
And Welcome here…
So it’s a common question about the ESP32C3 it is not like the others
If you have a Link to the super mini Board you tried I could look at the specs and perhaps share some thoughts on its IO.
I can offer this ,
1).The GPIO pin numbers may give better results , the Dx,Ax Pin macros differ in some BSP’s and boards.
2). The Xiao is so fast it see’s many INTERRUPTS, 
Debounce the switch in Software or add it to the hardware or disable the interrupt for a short time period…use a counter?
You can POLL for the Digital inputs or latch them and read them, but the Interrupt is better, your code can do other things than just read the IO.
Some notes on Interrupt, You should set a flag or just increment the counter and let the LOOP function do the printing… Bad things can happen when you print in the Interrupt Handler routine, FYI.
Check the search out there is a demo and how it uses the button and Tap interrupts too.
HTH
GL 
PJ 