Orin NX GPIO with level shifter doesn't output 0V

I am trying to interface a device on 5V with my Orin NX J4012.
To do so, I am using a level shifter to go from 3.3V output of the GPIO pin to 5V.

When I set the GPIO pin to HIGH, the reading is correctly 5V.
However, when I set it to LOW, the reading is ~1V. Way too high for my usecase (low needs to be 0.5V at max).

Additionally:

  • the voltages before the level shifter (pure GPIO output) are correct: 3.3V for high and 0V for low
  • I tested out the same level shifter on a Raspberry Pi 4 and there I am able to get 5V for high and 50mV for low (good enough).

So, what can be the problem in driving the shifted low voltage to 1V instead of 0V?