So this is quite straightforward with the XIAO ESP32-S3 boards. I just use a standard PC keyboard as a OTG device but should be similar for your “keypad”.
To add the 5V you can either inject (special cable or diy) or just add the 5V to the XIAO ESP32-S3’s VBus pin (14 or J1 Pin 7).
I haven’t noticed anything “untoward” with that approach.
For uploading, I use the secondary serial port connections D6/TX and D7/RX connected to a standard USB-232 device. With some USB-232’s there is 5V available, so you could potentially use that (but I haven’t tried).
Just make sure USB CDC on boot is “Disabled”. No issues with BSP - I used latest (same as yours).