Get x,y coords outside of windows/screen using mouselook technique

[Jonathan Janisch]

When you move your mouse all the way to the left - it hits the left edge of the screen and stops - even if the user keeps moving his mouse to the left.

In Java, there doesn't appear to be any way to detect that the user keeps moving his mouse "left".

Basically, if the current mouse coords are 50,100 and I move my mouse 10 feet to the left on my desk Java will still only report the position as 0,100.

My problem is, not only do I have to detect that the user moved the mouse to location -32535235,100 I also have to detect that position accurately.

In order to accomplish this, I'm doing the following:

1) Use robot class to forcefully move the cursor to the center of the screen.
2) Detect relative mouse movements and some Point which keeps track of the total movement
3) Repeat - go back to step 1

The code I'm using is basically straight out of a Java game development book "Developing Games in Java". This technique appears to work well for reporting simple changes in the relative x,y movement. It does not appear to be accurate in reporting actual x,y coordinates.

So my test case is this:

1) Start the application.
2) Move the mouse 1 foot to the left (left side of desk) - record the X location
3) Move the mouse 2 feet to the right (right side of desk)- record the X location.
4) Finally, move the mouse back to the left side of the desk and ensure it is the same (or very close) to the location recorded in step 2.

Currently when I do this, it's way off by hundreds of pixels.

Is there a simple error in the code that I'm missing? Any ideas on how I can accomplish this?

Here's the code:

import java.awt.BorderLayout;
import java.awt.Point;
import java.awt.Robot;
import java.awt.event.MouseEvent;
import java.awt.event.MouseMotionListener;
import javax.swing.JComponent;
import javax.swing.SwingUtilities;
 
 
public class SimpleRecenterTest extends JComponent implements MouseMotionListener {
 
    private Point location; // location relative to where the mouse cursor starts
    
    private Robot robot;    // use robot to recenter cursor
    
    
    private Point centerLocation;   // center of this component
    private Point lastActualLocation;   // last actual x y coords
    
    private boolean isRecentering;
    
    public SimpleRecenterTest() {
        location = new Point();
        centerLocation = new Point();
        lastActualLocation = new Point();
        
        try {
            robot = new java.awt.Robot();
        } catch (java.awt.AWTException ex) {
            ex.printStackTrace();
        }
    }
    
    public void mouseDragged(MouseEvent e) {
        mouseMoved(e);
    }
 
    public void mouseMoved(MouseEvent e) {
        
        // If the cursor reached the center, stop recentering...
        if (isRecentering && centerLocation.x == e.getX() && centerLocation.y == e.getY()) {
            isRecentering = false;
        } else {
            int dx = e.getX() - lastActualLocation.x;
            int dy = e.getY() - lastActualLocation.y;
            location.x += dx;
            location.y += dy;
            System.out.println("Location X,Y: " + location.x + "," + location.y);
            recenterMouse();
        }
        lastActualLocation.x = e.getX();
        lastActualLocation.y = e.getY();
    }
    
    public void init() {
        centerLocation.x = getWidth() / 2;
        centerLocation.y = getHeight() / 2;
        recenterMouse();
        addMouseMotionListener(this);
    }
 
    private synchronized void recenterMouse() {
        if (robot != null) {
            isRecentering = true;
            Point p = new Point(centerLocation);
            SwingUtilities.convertPointToScreen(p, this);
            robot.mouseMove(p.x, p.y);  // robot moves cursor using scren coordinates
        }
    }
    
    // Test it
    public static void main(String[] args) {
        java.awt.EventQueue.invokeLater(new Runnable() {
            public void run() {
                SimpleRecenterTest srt = new SimpleRecenterTest();
                javax.swing.JFrame f = new javax.swing.JFrame();
                f.setLayout(new BorderLayout());
                f.getContentPane().add(srt, BorderLayout.CENTER);
                f.setSize(800, 600);
                f.setDefaultCloseOperation(javax.swing.JFrame.EXIT_ON_CLOSE);
                f.setVisible(true);
                
                srt.init();
            }
        });
    }
}

[ August 14, 2007: Message edited by: Jonathan Janisch ]

[Craig Wood]

import java.awt.*;
import java.awt.event.*;
import javax.swing.*;
 
public class SRT extends JComponent implements MouseMotionListener,
                                               Runnable {
    private Point centerLocation;   // center of this component
    private Point screenLocation;   // last location on screen
    JLabel componentLabel;
    JLabel screenLabel;
 
    public void mouseDragged(MouseEvent e) {}
 
    public void mouseMoved(MouseEvent e) {
        Point p = e.getPoint();
        double d = p.distance(centerLocation);
        componentLabel.setText(String.format("[%4d, %4d]    %.1f",
                               p.x, p.y, d));
    }
 
    public void run() {
        boolean runMore = true;
        long delay = 40;
        int count = 0;
        while(runMore) {
            try {
                Thread.sleep(delay);
            } catch(InterruptedException e) {
                runMore = false;
                break;
            }
            Point p = showScreenData();
            if(!p.equals(screenLocation)) {
                screenLocation = p;
                count = 0;
                delay = 40;
            }
            // Slow timer down if mouse is not
            // moved for more than 1 second.
            if(count++ == 25) {
                delay = 1000;
            }
        }
    }
 
    protected void paintComponent(Graphics g) {
        Graphics2D g2 = (Graphics2D)g;
        g2.setRenderingHint(RenderingHints.KEY_ANTIALIASING,
                            RenderingHints.VALUE_ANTIALIAS_ON);
        centerLocation.x = getWidth() / 2;
        centerLocation.y = getHeight() / 2;
        g2.setPaint(Color.blue);
        g2.fillOval(centerLocation.x-2, centerLocation.y-2, 4, 4);
    }
 
    private Point showScreenData() {
        PointerInfo info = MouseInfo.getPointerInfo();
        Point screenLoc = info.getLocation();
        Point p = screenLoc.getLocation();
        SwingUtilities.convertPointFromScreen(p, this);
        double d = p.distance(centerLocation);
        screenLabel.setText(String.format("[%4d, %4d]    %.1f",
                            screenLoc.x, screenLoc.y, d));
        return screenLoc;
    }
 
    public void init() {
        centerLocation = new Point();
        addMouseMotionListener(this);
        Thread thread = new Thread(this);
        thread.setPriority(Thread.NORM_PRIORITY);
        thread.start();
    }
 
    private JPanel getLast() {
        componentLabel = new JLabel();
        screenLabel = new JLabel();
        Dimension d = new Dimension(125,25);
        JPanel panel = new JPanel(new GridBagLayout());
        GridBagConstraints gbc = new GridBagConstraints();
        gbc.insets = new Insets(2,2,2,2);
        addComponents(new JLabel("component"), componentLabel, panel, gbc, d);
        addComponents(new JLabel("screen"),    screenLabel,    panel, gbc, d);
        return panel;
    }
 
    private void addComponents(Component c1, JComponent c2, Container c,
                               GridBagConstraints gbc, Dimension d) {
        gbc.anchor = GridBagConstraints.EAST;
        gbc.gridwidth = GridBagConstraints.RELATIVE;
        c.add(c1, gbc);
        c2.setPreferredSize(d);
        gbc.anchor = GridBagConstraints.WEST;
        gbc.gridwidth = GridBagConstraints.REMAINDER;
        c.add(c2, gbc);
    }
 
    // Test it
    public static void main(String[] args) {
        EventQueue.invokeLater(new Runnable() {
            public void run() {
                SRT srt = new SRT();
                JFrame f = new JFrame();
                f.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
                System.out.printf("JFrame default layout = %s%n",
                                   f.getLayout().getClass().getName());
                f.getContentPane().add(srt);
                f.getContentPane().add(srt.getLast(), "Last");
                f.setSize(400, 400);
                f.setLocationRelativeTo(null);
                f.setVisible(true);
                System.out.printf("f.x = %d  f.y = %d%n", f.getX(), f.getY());
                srt.init();
            }
        });
    }
}

[Justin Chu]

The recentering seems to work fine, and your "location" print out seems to move around fine.

EDIT - Nvmd, I see a bias towards the negative X.

EDIT 2 - I think your code works fine, because of the differences in mouse acceleration. You typically cannot expect 1 foot on your table top = x number of pixels.

If I move the mouse back and forth multiple times, it seems to hover around the same point.
[ August 20, 2007: Message edited by: Chu Tan ]

[Jonathan Janisch]

Craig, I appreciate the time you put into your response. It's not really what I was attempting to accomplish though

Chu, thanks. This isn't really important - it's really just an experiment for fun. I wanted to make a cheap digital whiteboard using only a small mouse and a projector.

In order to do that I needed to be able to map positions on the whiteboard to screen coordinates. I attempted to solve that by first entering a configuration mode where the user clicks the top left, bottom left, and bottom right of the whiteboard - but I'm having issues getting those coordinates reliably.

[Brian Cole]

Originally posted by Jonathan Janisch:
So my test case is this:

1) Start the application.
2) Move the mouse 1 foot to the left (left side of desk) - record the X location
3) Move the mouse 2 feet to the right (right side of desk)- record the X location.
4) Finally, move the mouse back to the left side of the desk and ensure it is the same (or very close) to the location recorded in step 2.

Currently when I do this, it's way off by hundreds of pixels.

Is there a simple error in the code that I'm missing?

You seem to be presuming that screen-pixels-per-mouse-inch is constant, but that's not the case. Modern UI standards typically call for a fast mouse-inch to move the cursor more pixels than a slow mouse-inch.

You can try this yourself. Place your mouse against the right side of your keyboard (or laptop), move the mouse quickly rightward, now slowly move the mouse leftward until it abuts the keyboard again. The mouse is back where it started, but the screen cursor will not be.

You might be able to find an OS (or perhaps just a mouse driver) that lets you set custom mouse acceleration curves. In that case you could set a flat curve and it might work the way you want it to. (Or try disable mouse acceleration on google.)

[Jonathan Janisch]

Thanks Brian, I did not know that!