Thanks for figuring out this important issue. Actually, we have a plan to break out the Cortex-M4 to the raspberry pi standard 40-pin connector, and will provide an Arduino HAT in the future, do you think this is an acceptable solution for you?
From the block diagram in the datasheet, we can see that the data transfers between the A7 and M4 via multi-AHB bus and AXI bus. Theoretically, the M4 and A7 can’t access the SRAM at the exact same time but might be able to access and switch in a very short time.