How to get MOUSE_PRESSED events on a touch screen?

[Stevens Miller]

The program below prints a message immediately upon clicking the JPanel with a mouse. Specifically, it receives a MOUSE_PRESSED event as soon as the left button on the mouse is pressed. The left button does not need to be released for this to happen.

My Windows 10, Asus laptop has a touch screen. When I press the JPanel via the screen, nothing happens until after I take my finger off of the screen.

Googling this has been largely fruitless, although a possibility might be that I need to call the Window API RegisterTouchWindow function with the Windows handle of my topmost JFrame (however, it has been said that this is no longer needed since Windows 8).

Now, a couple of interesting points: the JavaFX Node method, setOnTouchPressed is called immediately upon pressing, even though no visual feedback appears on the screen. Also, the behavior is observable, but not consistently, in Windows itself. For example, if I open the Control Panel, pressing a link to an applet does nothing, not even moving the focus to the link, until after I lift my finger from the screen. However, if I press the "X" in the Control Panel window's upper right, it turns red immediately upon contact with the screen.

Does anyone know how to make Java received MOUSE_PRESSED events from a Windows 10 touch screen immediately upon touching the screen?

import java.awt.Color;
import java.awt.event.MouseAdapter;
import java.awt.event.MouseEvent;
import javax.swing.GroupLayout;
import javax.swing.JPanel;
import javax.swing.WindowConstants;

/**
 *
 * @author smiller
 */
public class MainFrame extends javax.swing.JFrame
{

    /**
     * Creates new form MainFrame
     */
    public MainFrame()
    {
        initComponents();
    }

    /**
     * This method is called from within the constructor to initialize the form.
     * WARNING: Do NOT modify this code. The content of this method is always
     * regenerated by the Form Editor.
     */
    @SuppressWarnings("unchecked")
    // <editor-fold defaultstate="collapsed" desc="Generated Code">                          
    private void initComponents()
    {

        jPanel1 = new JPanel();

        setDefaultCloseOperation(WindowConstants.EXIT_ON_CLOSE);

        jPanel1.setBackground(new Color(255, 204, 0));
        jPanel1.addMouseListener(new MouseAdapter()
        {
            public void mousePressed(MouseEvent evt)
            {
                jPanel1MousePressed(evt);
            }
        });

        GroupLayout jPanel1Layout = new GroupLayout(jPanel1);
        jPanel1.setLayout(jPanel1Layout);
        jPanel1Layout.setHorizontalGroup(jPanel1Layout.createParallelGroup(GroupLayout.Alignment.LEADING)
            .addGap(0, 100, Short.MAX_VALUE)
        );
        jPanel1Layout.setVerticalGroup(jPanel1Layout.createParallelGroup(GroupLayout.Alignment.LEADING)
            .addGap(0, 100, Short.MAX_VALUE)
        );

        GroupLayout layout = new GroupLayout(getContentPane());
        getContentPane().setLayout(layout);
        layout.setHorizontalGroup(layout.createParallelGroup(GroupLayout.Alignment.LEADING)
            .addGroup(layout.createSequentialGroup()
                .addGap(137, 137, 137)
                .addComponent(jPanel1, GroupLayout.PREFERRED_SIZE, GroupLayout.DEFAULT_SIZE, GroupLayout.PREFERRED_SIZE)
                .addContainerGap(163, Short.MAX_VALUE))
        );
        layout.setVerticalGroup(layout.createParallelGroup(GroupLayout.Alignment.LEADING)
            .addGroup(layout.createSequentialGroup()
                .addGap(69, 69, 69)
                .addComponent(jPanel1, GroupLayout.PREFERRED_SIZE, GroupLayout.DEFAULT_SIZE, GroupLayout.PREFERRED_SIZE)
                .addContainerGap(131, Short.MAX_VALUE))
        );

        pack();
    }// </editor-fold>                        

    private void jPanel1MousePressed(MouseEvent evt)                                     
    {                                         
        System.out.println(evt.toString());
    }                                    

    /**
     * @param args the command line arguments
     */
    public static void main(String args[])
    {
        /* Set the Nimbus look and feel */
        //<editor-fold defaultstate="collapsed" desc=" Look and feel setting code (optional) ">
        /* If Nimbus (introduced in Java SE 6) is not available, stay with the default look and feel.
         * For details see http://download.oracle.com/javase/tutorial/uiswing/lookandfeel/plaf.html 
         */
        try
        {
            for (javax.swing.UIManager.LookAndFeelInfo info : javax.swing.UIManager.getInstalledLookAndFeels())
            {
                if ("Nimbus".equals(info.getName()))
                {
                    javax.swing.UIManager.setLookAndFeel(info.getClassName());
                    break;
                }
            }
        } catch (ClassNotFoundException ex)
        {
            java.util.logging.Logger.getLogger(MainFrame.class.getName()).log(java.util.logging.Level.SEVERE, null, ex);
        } catch (InstantiationException ex)
        {
            java.util.logging.Logger.getLogger(MainFrame.class.getName()).log(java.util.logging.Level.SEVERE, null, ex);
        } catch (IllegalAccessException ex)
        {
            java.util.logging.Logger.getLogger(MainFrame.class.getName()).log(java.util.logging.Level.SEVERE, null, ex);
        } catch (javax.swing.UnsupportedLookAndFeelException ex)
        {
            java.util.logging.Logger.getLogger(MainFrame.class.getName()).log(java.util.logging.Level.SEVERE, null, ex);
        }
        //</editor-fold>

        /* Create and display the form */
        java.awt.EventQueue.invokeLater(new Runnable()
        {
            public void run()
            {
                new MainFrame().setVisible(true);
            }
        });
    }

    // Variables declaration - do not modify                     
    private JPanel jPanel1;
    // End of variables declaration                   
}

[Rob Spoor]

Stevens Miller wrote:The program below prints a message immediately upon clicking the JPanel with a mouse. Specifically, it receives a MOUSE_PRESSED event as soon as the left button on the mouse is pressed. The left button does not need to be released for this to happen.

That sounds exactly like what it should do. The event is for when the mouse is pressed, not clicked. That has its own event, and mouse pressed has a companion event in mouse released. What happens if you click is a sequence of events: mouse pressed, mouse clicked and mouse released (although I don't know the order of the latter two). If you slightly move the mouse while clicking you might even get a mouse dragged (and possibly even mouse moved) as well.

My Windows 10, Asus laptop has a touch screen. When I press the JPanel via the screen, nothing happens until after I take my finger off of the screen.

Sounds more like something's wrong on the laptop.

[Stevens Miller]

Rob Spoor wrote:

Stevens Miller wrote:My Windows 10, Asus laptop has a touch screen. When I press the JPanel via the screen, nothing happens until after I take my finger off of the screen.

Sounds more like something's wrong on the laptop.

That's certainly possible.

Would someone out there who has a Windows 10 machine with a touch screen try this test for me?

1. Run the mspaint.exe program.

2. Select the "Fill" tool (the bucket).

3. Touch the canvas with your finger and hold it on the screen. (On my machine, this fills the canvas with black immediately.)

4. Touch one of the non-black colors in the color palette and hold your finger on the screen. (On my machine, this causes a tiny little cross-hairs to appear near the point of touch, but the current brush color does not change.)

5. Lift your finger from the screen. (On my machine, this causes the current brush color to change to the one you touched.)

Step 3 proves that Windows can react as I want it to, immediately.

Step 4 proves that, on my machine, Windows reacts immediately (by showing the little cross-hairs) but doesn't actually make the application do anything.

Step 5 proves that, in the same application, Windows reacts immediately to some touches, but not to others.

Now, if you use the mouse for Step 3-5, instead of your finger, the behavior is the same. At Step 3, the fill happens at once. At Step 4, the current brush color does not change. At Step 5, the current brush color does change.

Thus, on my machine, some things happen immediately upon "mouse pressed," either via the mouse or the touch screen, while others wait for "mouse released" (or "mouse clicked"), either via the mouse of the touch screen. What I'm trying to do is get Java to give me the same immediate reaction that I see in Step 3 in the above list, but I can't. Deep down, the messages Windows sends to an application are not the same. That is, it sends different messages for mouse events versus touch events. It's possible that AWT/Swing simply don't react to touch events, but JavaFX does. Still, what's frustrating is that Swing does get a MOUSE_PRESSED event when I touch the screen, but only after I lift my finger from the screen.

Oh, say, here's another test that would be a big help if anyone with Windows 10 and a touch screen would try it: run my code listed above and let me know if you see the message immediately, or only after you take your finger off the screen. If anyone sees it immediately, then maybe we can figure out what's different about our two setups.

[Dave Tolls]

You can't really compare a Windows exe, that can access the underlying Windows API easily, with a Swing application that has had touch capability bolted onto its mouse events.

As far as I can tell, a single tap is a pressed (and released?) event, so only fires when the finger is lifted.
A double tap is a click.
That's from stuff I read about this a few years ago, so no links I'm afraid.

So the limitation is with Swing.

With FX (as you say) you get the full touch support.

[Stevens Miller]

Dave Tolls wrote:With FX (as you say) you get the full touch support.

It's a mix. There are separate touchPressed and touchReleased events that JavaFX supports, but they don't act as alternatives to the mouse events. Instead, if you want to assign an "action" to a JavaFX Button, that action will fire when you click the button with the mouse, and it will also fire when you press the button through the touch screen. That's because the touch screen emulates a mouse, and generates the same MOUSE_PRESSED, MOUSE_RELEASED, and MOUSE_CLICKED events the mouse generates. That means the touch screen will also fire the action. But, owing to the silly way the events are timed (meaning that you don't get the MOUSE_PRESSED event until after you lift your finger), you can't make the touch screen behave like a real electromechanical button would behave. In fact, it doesn't even take the focus or change in any other way until you lift your finger (unless you move your finger while you touch the screen, and I cannot figure that behavior out at all).

Now, you can add touchPressed handlers to fire the action for you, but then you will fire the action twice: once when the touchPressed handler fires it, and once more when the touch screen sends the three mouse events after you lift your finger. You also have to take the focus and arm the button in the touchPressed handler, if you want the same visual feedback the mouse gives you.

To prevent the second firing, you can set a "skip" Boolean flag in the touchPressed handler, and have the action handler consult it. If it's true, set it false and just return. If it is false, execute the action.

But this gets worse...

If you press the button through the touch screen, take the focus, arm the button, fire the action, and set the skip flag, and then drag your finger off the button before taking your finger off the screen, the MOUSE_PRESSED event will occur, but the MOUSE_RELEASED event will not occur (for that button, anyway). This means the action will not fire a second time, because the necessary mouse events don't all happen. Thus, with the skip flag set, the next time you do press or click the button, the action will not be executed, because the skip flag is still set.

To get around this, I added a touchReleased handler. This is always called, even if you drag your finger off the button before taking it off the screen. In that handler, I can check to see if the event happened inside the visible shape of the button (there's actually a library method for that). If so, then (and only then) do I set the skip flag, since I can be confident that the mouse events will all be sent such that the action will be fired a second time (and, therefore, I want it to be skipped).

This all seems to work, but it also seems insane. This can't be how the JavaFX designers expected me to use the touch screen, by fighting with its dual behavior of sending touch events and mouse events. In this context, I'd rather just have the touch screen stop sending mouse events entirely. Or, as I started out, I would be happy if it just sent the pressed events for both mouse and touch like it does for touch: when, in fact, it is pressed. But it doesn't.

Is this really how touch screens work in Java?

(Adding this thread to JavaFX, since that's become a big part of the discussion.)

[Dave Tolls]

Ah.
That's what happens when you don't actually see these things in action.
That sounds a lot less useful than I thought, for FX.

I had the impression that it was a set of events akin to what you'd get from Android, for example.

[Stevens Miller]

Dave Tolls wrote:I had the impression that it was a set of events akin to what you'd get from Android, for example.

I bet you've just hit the nail on the thumb, Dave. I am working on an antique piece of equipment known as a "laptop computer." These required continuous shoveling of coal to keep their boilers hot and regular adjustment of their flywheel governors and gasket maintenance to avoid overheating or leaks. Contemporary software engineers, I am told, mostly concern themselves with something called a Mo' Belle de Fice. I've never actually seen one, though my son claims that's what he has lodged in his groin when he spends hours at a time, folded up like a jack-knife, staring into his lap, doing (as far as I can tell) nothing whatsoever. He seems to find it very satisfying. What little I know of such things suggests that, indeed, a de Fice's touch screen works reasonably, while the one on this steam engine I'm using works the cranky way it does because it had to get along with another obsolete contraption (a "mouse"), which the de Fice doesn't have. Given that the free market never fails to do what's best for all of us, I am forced to conclude that all my problems are my own fault, because the Java gods see no reason to make my coal-fired computer work any better than it does.

Gives me some comfort, at least, that my Rube Goldberg focus/arm/fire/skip-on-release approach is not as utterly absurd as it looks.

[Dave Tolls]

I knew there was a reason I never got a touch screen laptop.

Some things just shouldn't be mixed together...that way lies madness, I tell you!

[Stevens Miller]

Dave Tolls wrote:I knew there was a reason I never got a touch screen laptop.

Some things just shouldn't be mixed together...that way lies madness, I tell you!

Heh. Although it is about HTML5, and not Java, the problematic nature of mixing a touch interface and a mouse interface is addressed rather well in this article. I'm still not clear on why Windows doesn't pass a MOUSE_PRESSED event when it passes a TOUCH_PRESSED event, but I'm sure that's because I'm dumb. Regardless, that article at least confirms that the behavior I'm seeing (and the problems it causes) are normal.