Xiao ESP32S3 & SX1262 - Deep Sleep Current problem

So could that be the same problem I’m having when I use the user button on the sx1262 to wake up the system from deep sleep? When I add a esp_sleep_enable_ext1_wakeup(1ULL << 21, ESP_EXT1_WAKEUP_ANY_LOW); to my code the sleep current shoots up by a whole 1mA.