Yepp, the led matrix MUST be ok: “Operating voltage: 4.5V~5.5V” from the dathasheet (adafruit.com/datasheets/ht16K33v110.pdf)
The faces are somewhat OK, so the rephone DOES work, but the question is: why does it mess up?
Can you add some delay inbetween sending the command data? If not, please send your whole code - maybe a little pause must be inserted.