Thanks @PJ_Glasso for the excellent info!
What you describe seems to align with what I was seeing. There was a point where turning CONFIG_ASSERT on/off would make the problem appear/disappear.
This issue seems unfortunate. It is natural to turn on debug config when you have an issue like a bus fault. The bus faults with this issue might make it tricky to debug similar looking real issue bus faults and send some on a wild goose chase like my own.