Voltage of 5V pin when only connected to battery

I don’t think a diode is necessary.

As you can see in the link below, the sleep current of ESP32C3 is less than 50uA. if you have PPK2, you should check the current consumption of XIAO and display unit respectively.
The sleep current of the display unit chip is 3uA, so the rest might be the display panel and the boost circuit. Or maybe it is not sleeping properly.