I found the culprit. I had the USB port connected (using a cable with the power removed though, so I thought it wouldn’t make a difference). After detaching it the XIAO consumes just 40uA 
As a side note, I had also removed removed C39 but it didn’t make a difference