Looks like @PJ_Glasso found an explanation for the problem. See info here: XIAO nrf54l15 Matter App Recipe
The issue appears to be with uart21 being enabled. Including the following in your .overlay file should prevent the issue.
&uart21 {
status = "disabled";
};
I have been running a day or so with uart21 disabled and haven’t seen the issue.