Could you please send, how did you exactly connected the boards and the battery pack?
Just because the led matrix board must have survived your 3x AA batteries: it has a power regulator and it must work up to 6V! (I checked the circuit, datasheet of the regulator here farnell.com/datasheets/2012663.pdf)
Maybe you forgot to add a proper common ground, therefore the communication fails - hopefully.