How can I control context menus in other (Windows) applications using Java Robot and Swing buttons

Question

I have written an extensive on-screen keyboard (OSK) in Java. This is intended for people with disabilities and needs to be able to do everything that a standard keyboard can do. One thing I've noticed that doesn't work is context menus - in (most) Windows programs. I want my OSK user to be able to click the 'Context Menu' key (normally beside right-Ctrl on standard keyboard) and use the arrow keys to navigate that menu. The context menu appears fine, but when I click back onto my Frame (to press an arrow key), the context menu disappears.

Anybody got any ideas as to how this might be achieved (probably a fairly monstrous 'hack' required I would think)? I'm pretty sure the context is disappearing because a mouse-click is occurring elsewhere on the screen (hard to get around that when you need to actually click something!).

Below is a test app. I can use it to navigate down through a context menu in NetBeans successfully, but not in any other application I've tried (e.g. Windows Explorer).

import java.awt.Robot;
import java.awt.event.ActionEvent;
import java.awt.event.ActionListener;
import java.awt.event.KeyEvent;
import javax.swing.JButton;
import javax.swing.JFrame;
import javax.swing.JPanel;
import javax.swing.WindowConstants;

public class RobotContextMenuTest extends JFrame implements ActionListener
{
    Robot R;
    JPanel JP;
    JButton JBmenu, JBdown;

    public RobotContextMenuTest()
    {
        try
        {
            R = new Robot();
        }
        catch(Exception e)
        {
            e.printStackTrace();
        }

        JP = new JPanel();
        getContentPane().add(JP);

        JBmenu = new JButton("Menu");
        JBmenu.addActionListener(this);
        JP.add(JBmenu);

        JBdown = new JButton("Down");
        JBdown.addActionListener(this);
        JP.add(JBdown);

        setFocusableWindowState(false);
        setAlwaysOnTop(true);
        setSize(200,80);
        setDefaultCloseOperation(WindowConstants.EXIT_ON_CLOSE);
        setVisible(true);
    }
    public static void main(String args[])
    {
        new RobotContextMenuTest();
    }

    public void actionPerformed(ActionEvent e)
    {
        if(e.getActionCommand().equals("Menu"))
        {
            R.keyPress(KeyEvent.VK_CONTEXT_MENU);
            R.keyRelease(KeyEvent.VK_CONTEXT_MENU);
        }
        else if(e.getActionCommand().equals("Down"))
        {
            R.keyPress(KeyEvent.VK_DOWN);
            R.keyRelease(KeyEvent.VK_DOWN);
        }
    }
}

Finally, here's what I've tried:

• Using MouseListener on the buttons instead of ActionListener, then 'consuming' the MouseEvent in the hope that the other application won't register it and hide the context menu.
• I made a Threaded test application that is able to show any context menu and automatically scan down through it (once every second) but that doesn't really solve my problem! I want the context menu navigation to be controlled by the user's mouse clicks.
• I've tried using JNativeHook as an alternative method of capturing mouse clicks but the result is the same.

Sorry for the long-winded question but it's a fairly complicated problem! Incidentally, the standard Windows 7 OSK can't do this either but WiViK can. Thanks.

Answer 1

Finally cracked this with a lot of help from a friend (thanks Chris Gregan!). After many failed attempts, the answer was relatively simple. I used JNA and Windows API to create a hook for low-level mouse events. It's all described with 2 sample classes given here in an answer by 'prunge'. Note: I had to include the jars for jna version 3.4.0 and jna platform version 3.4.0 (newer versions of these jars resulted in errors!).

Using the above linked MouseHook, you can prevent a low-level event from propagating (reaching other applications) by returning -1 from the LowLevelMouseProc callback.

Here is a snippet I added to the callback method in MouseHook to intercept/suppress left mouse button presses (thereby preventing context menus from disappearing!):

if(nCode >= 0 && wParam.intValue() == WinUserX.WM_LBUTTONDOWN)
    return new LRESULT(-1);

Thanks again to Alex Barker for taking the time to look at this and point me in the direction of JNA and Windows API.

Answer 2

• Using MouseListener on the buttons instead of ActionListener, then 'consuming' the MouseEvent in the hope that the other application won't register it and hide the context menu.

This doesn't currently work for you because consuming events in Java only effects your application, it cannot prevent the events propagation on the native system. JNativeHook 1.2 may have an undocumented method to allow you to do this on windows, although I am not sure if it will actually solve your problem. If you consume a click event on the native system, the jvm probably wont receive the click event on the osk button to send the arrow down events. There would be no way to detect if the java button is being clicked before you would need to consume the native event. In other words, the NativeInputEvent where you would call a consume method would probably be delivered before the Java AWT InputEvent. Please see https://code.google.com/p/jnativehook/issues/detail?id=22 for more information.

The only good solution I can think of right now would be to use JNA to call Windows API and grab the context menu handle then traverse it directly. There are API's available to do this sort of thing, I just don't remember what they are off the top of my head. Maybe start with http://msdn.microsoft.com/en-us/library/windows/desktop/ff468865(v=vs.85).aspx and see if you can create a small test program to manipulate the menu.

* Small Update: After a little more Googling I was able to locate the following post that pretty much outlines how to do this with C++. Porting this to JNA should be fairly trivial.