After many evenings at the computer and even more head-shaking, I have finally found a solution.
You have to use the HinTakt driver in v6.6.
You can retrieve it from GitHub with the following command:
git clone -b v6.6 --single-branch GitHub - HinTak/seeed-voicecard: This is an enhancement fork with the explicit aim of supporting current shipping Raspbian/Ubuntu kernels without requiring downgrading. Please donate at https://hintak.github.io/ if it works for you. Use vX.Y branch for kernel version vX.Y
If you need further help with the installation of Assist with HomeAssistant, please feel free to contact us.