On some AIX systems, we have found that the /usr/dt/bin/Xsession script uses the xmodmap command to tamper with the keyboard layout, causing this kind of problem.

Have a look at your ~/.vnc/xstartup file, or the system-wide /etc/vnc/xstartup file. If it runs /usr/dt/bin/Xsession first, you could try changing it to run /usr/dt/bin/dtsession instead, which might bypass the keyboard-tampering.

Otherwise, you could try exporting these variables

XSTATION=1 DTXSERVERLOCATION=local

before running /usr/dt/bin/Xsession . So the lines would look something like this:

elif [ -x /usr/dt/bin/Xsession ]; then
  XSTATION=1 DTXSERVERLOCATION=local /usr/dt/bin/Xsession
elif [ -x /usr/dt/bin/dtsession ]; then /usr/dt/bin/dtsession

This change has been reported to fix keyboard problems on AIX.