The Bluefruit is a library built to work with the Nordic softdevice. If you want to use BLE then you have three choices. Write your own register level code. Use the Mbed version of the XIAO BLE Sense library for BLE or use the Adafruit (version 1) of the XIAO BLE Sense library, which is Bluefruit.
Many people seem to have problems with the Mbed libraries and use the Adafruit instead because they are much better documented.
In most cases, the Mbed library, and the Adafruit library are not compatible. You have to make a choice.