Hi there,
Sounds like a fun project I would recommend you scale it back and try two approaches.
First just use one Letter, Train your model with several samples or fonts. Then add the "Gesture "
looking for the finger point gesture. Scale it back and get the first thing working first then proceed.
Add the ESP32S3 as the Xiao more ram and Horsepower in that. After some success copy the models to the flash and try inferencing that!
would be my first recommend.
HTH
GL 
PJ 