VNC Keyboard Headaches

VNC is a really cool utility to connect to machines.  It works on various OSes and either gives you a new desktop, or lets you monitor or take over an existing desktop.

This makes it great for support, and for teaching.  It's also fairly established and accepted.  It's really a wonderful utility, and we use it.

We have recently stumbled across two keyboard problems.

1. Connecting from a Windows client to a Linux server, Numlock and the numeric keypad do not always operate as expected.
2. From any machine to a Linux server, the keyboard sometimes gets stuck in auto-repeat mode, so you'll get text like helllllllo woooooorld.  This problem is exaggerated when mixing case, as in Hhhhhello Wwwwworld.

These problems are different depend on the VNC server, the X server, and the VNC client.  So take note  that this is for VNC Server 4.1.2 and Xorg 1.1.1 as shipped with the RedHat Enterprise 5 series.

Numlock

Problem one appears to be caused by various design decisions:

• NumLock really sends Shift-Home for 7, Shift-Up for 8, Shift-PgUp for 9, etc.  It behaves much like CapsLock on the left half of the keyboard.  VNC assumes that neither CapsLock nor Numlock is set -- because they might be different between the server and workstation.
• The client keyboard can be vastly different from the server keyboard.  They don't both need to be QWERTY, and the server could happily have a 105-key keyboard, and the client a laptop or netbook keyboard.  VNC has to make some assumptions and assume some sane defaults.

So we map the keys, either in xorg.conf:

Section "Module"
 Load "vnc"
EndSection

Section "Screen"
 Option "RemapKeys" "0xffae->0x2e,0xffb0->0x30,0xffb1->0x31,0xffb2->0x32,0xffb3->0x33,0xffb4->0x34,0xffb5->0x35,0xffb6->0x36,0xffb7->0x37,0xffb8->0x38,0xffb9->0x39"
EndSection

Or you can run vncconfig -set RemapKeys="0xffae->0x2e,0xffb0->0x30,0xffb1->0x31,0xffb2->0x32,0xffb3->0x33,0xffb4->0x34,0xffb5->0x35,0xffb6->0x36,0xffb7->0x37,0xffb8->0x38,0xffb9->0x39" on the command line after you are connected

To find other keystrokes that you want to remap, use xev, and look for the keysym values.

Keyboard Repeat

Keyboard repeat is normally switched off with xset r off.  However, that leaves it off for the keyboard physically attached to the server too, so that isn't ideal.  The vnc-server package shipped with RedHat Enterprise is Real VNC and that has this bug.  So here is a patch to switch keyboard repeat off when someone connects via VNC, and switch it back on when all connections close.

The version of vnc-server shipped with RedHat Enterprise Linux is GPL.

--- common/rfb/VNCServerST.h.orig 2010-05-27 15:41:38.000000000 +0200
+++ common/rfb/VNCServerST.h 2010-05-27 15:42:10.000000000 +0200
@@ -219,6 +219,7 @@
     QueryConnectionHandler* queryConnectionHandler;
     KeyRemapper* keyRemapper;
     bool useEconomicTranslate;
+    bool savedRepeat;
   };
 
 };
--- common/rfb/VNCServerST.cxx.orig 2010-05-27 15:42:20.000000000 +0200
+++ common/rfb/VNCServerST.cxx 2010-05-27 15:42:53.000000000 +0200
@@ -79,6 +79,7 @@
     useEconomicTranslate(false)
 {
   slog.debug("creating single-threaded server %s", name.buf);
+  savedRepeat = true;
 }
 
 VNCServerST::~VNCServerST()
--- common/rfb/InputHandler.h.orig 2010-05-27 12:09:44.000000000 +0200
+++ common/rfb/InputHandler.h 2010-05-27 15:40:29.000000000 +0200
@@ -32,9 +32,10 @@
   public:
     virtual ~InputHandler() {}
     virtual void keyEvent(rdr::U32 key, bool down) {}
+    virtual void keyRepeat(bool on) {}
+    virtual bool getKeyRepeat() { return false; }
     virtual void pointerEvent(const Point& pos, int buttonMask) {}
     virtual void clientCutText(const char* str, int len) {}
   };
-
 }
 #endif
--- common/rfb/VNCSConnectionST.cxx.down 2010-05-27 10:19:25.000000000 +0200
+++ common/rfb/VNCSConnectionST.cxx 2010-05-27 15:43:24.000000000 +0200
@@ -61,6 +61,8 @@
   std::set::iterator i;
   for (i=pressedKeys.begin(); i!=pressedKeys.end(); i++)
     server->desktop->keyEvent(*i, false);
+  if (server->clients.size() == 1)
+    server->desktop->keyRepeat(server->savedRepeat);
   if (server->pointerClient == this)
     server->pointerClient = 0;
 
@@ -308,6 +310,9 @@
 
   // - Mark the entire display as "dirty"
   updates.add_changed(server->pb->getRect());
+  // If we're the first connection, save the keyboard repeat
+  if (server->clients.size() == 1) server->savedRepeat = server->desktop->getKeyRepeat();
+  server->desktop->keyRepeat(false);
 }
 
 void VNCSConnectionST::queryConnection(const char* userName)
--- unix/xc/programs/Xserver/vnc/XserverDesktop.cc.orig 2010-05-27 11:56:16.000000000 +0200
+++ unix/xc/programs/Xserver/vnc/XserverDesktop.cc 2010-05-28 10:20:24.000000000 +0200
@@ -985,6 +985,29 @@
 }
 
 
+void XserverDesktop::keyRepeat(bool on)
+{
+  DeviceIntPtr dev = (DeviceIntPtr)LookupKeyboardDevice();
+  KeybdCtrl ctrl = dev->kbdfeed->ctrl;
+  if (on) {
+    vlog.info("Switching keyboard repeat on");
+    ctrl.autoRepeat = TRUE;
+  } else {
+    vlog.info("Switching keyboard repeat off");
+    ctrl.autoRepeat = FALSE;
+  }
+  dev->kbdfeed->ctrl = ctrl;
+  (*dev->kbdfeed->CtrlProc)(dev, &dev->kbdfeed->ctrl);
+}
+
+bool XserverDesktop::getKeyRepeat()
+{
+  DeviceIntPtr dev = (DeviceIntPtr)LookupKeyboardDevice();
+  KeybdCtrl ctrl = dev->kbdfeed->ctrl;
+  vlog.info("Keyboard repeat was: %s", ctrl.autoRepeat ? "on" : "off");
+  return ctrl.autoRepeat;
+}
+
 void XConvertCase(KeySym sym, KeySym *lower, KeySym *upper)
 {
     *lower = sym;
--- unix/xc/programs/Xserver/vnc/XserverDesktop.h.orig 2010-05-27 11:58:16.000000000 +0200
+++ unix/xc/programs/Xserver/vnc/XserverDesktop.h 2010-05-27 14:38:52.000000000 +0200
@@ -87,6 +87,8 @@
   // rfb::SDesktop callbacks
   virtual void pointerEvent(const rfb::Point& pos, int buttonMask);
   virtual void keyEvent(rdr::U32 key, bool down);
+  virtual void keyRepeat(bool on);
+  virtual bool getKeyRepeat();
   virtual void clientCutText(const char* str, int len);
   virtual rfb::Point getFbSize() { return rfb::Point(width(), height()); }

Unresolved

The keyboard repeat can come back.  If another process sets the keyboard repeat back on, it'll be on.  For example:

1. Lauch an X login program, like gdm
2. Connect with VNC, and the keyboard repeat is off.
3. Log in to gdm.
4. That then starts a session, and the keyboard repeat is back on.

The way to fix that is to grab (*dev->kbdfeed->CtrlProc); and set it to your own handler that prevents toggling the autoRepeat flag.  That's left as an exercise to the reader.