Figured it out … I was (kind of) stupid …
When setting up slave you have to provide 4 params, I was giving 3 (which sets up a master)
Wire.begin(SLAVE_ADDRESS_1,5,6,100000UL);
Wire1.begin(SLAVE_ADDRESS_2,1,2,100000UL);
Hope it helps someone …
RTFM