Hi,
You just connect the AU_HPL(which mean audio_headphone_left_channel) to the amplifier? I think may be the circuit lack of some filter circuit, especially the blocking capacitors. May be you can reference to the headphone circuit of LinkIt One, add some filter before you connect the amplifier.