Hi there,
What are the Connection parameters ? Are you setting things like advertisement time, Latency, etc. What speed PHY are you using? NimBLE allows tuning of them so You’ll need to adjust those to get close to real time response.
Cool project.
HTH
GL 
PJ 