After a while i figured that`s what broke it. I did a sudo apt full-upgrade at some point and after that it broke. Atleast i think so, this was quite a while ago 
I think it has something to do with the firmware on the board the CM4 sits on which does not like the āapt upgradeā. If i remember correctly it was somewhat broken after that.
Im not sure, but is there any fix to when someone did that mistake with āsudo apt-upgradeā or does it mean that the device now is permanently broken?