I made some progress, I think I had 1 or 2 major wiring issues.
First I only had this wiring:
I2S_DIN_XIAO(In) → GPIO 21
I2S_DIN_SEC(Out) → GPIO 20
I2S_LRCLK → GPIO 19
I2S_BCLK → GPIO 18
Seeed released the pinout some months ago, and today I noticed the XU316_RST Pin, shorting it once again GND seems to “enable” the respeaker.
I can now with the overlay mentioned by @jiewei.ye (Thanks again) record and playback.
But playback is still not 100% working, its kind of slow, I got the feeling its only playback with half of normal speed.
I also connected the I2C Pins with the raspberry, its seems to work without it, but I’ve got static while Playback.
SDA → GPIO 2
SCL → GPIO 3
Update:
The slow playback is fixed now, turned out, my test files had a samplerate of 48khz, after flashing the 48khz firmware, playback is working fine now.
Also, after installing 32 Bit firmware of Raspberry OS, it seems like I don’t need to short the reset pin anymore.