How to create keypress simulation program?

Question

Main Problem

I'm trying to create a widely applicable keypress simulation program, but I don't know how to use Java so I can only search for Q&As on the site, which is inefficient. So I want to open a new post and deal with these problems all in once:

I need the following functionality:

Press arbitrary keys (using key codes shown here) for example, some code like press[65] should press A.
Press arbitrary mouse keys (including mouse down, mouse up and mouse click)
Move mouse to a specific location
rotate the scrolling of the mouse
Screen shot

How to do these small functionalities? Thanks!

Some work I've done

One can use these as a reference:

JLink part:

Needs["JLink`"];
ReinstallJava[];
robotclass = JavaNew["java.awt.Robot"];
LoadJavaClass["java.awt.event.InputEvent"];
move[{x_, y_}] := robotclass@mouseMove[x, y];
mouseleftdown[] := robotclass@mousePress[InputEvent`BUTTON1UMASK];
mouseleftup[] := robotclass@mouseRelease[InputEvent`BUTTON1UMASK];
mouseleftclick[] := CompoundExpression[mouseleftdown[], mouseleftup[]];
mouseleftdoubleclick[] := Do[mouseleftclick[], 2];

Mathematica Part:

MousePosition[];
CurrentValue[{"MouseButtonTest",1}]

rotate scrolling and press arbitrary mouse keys (except left mouse button). but just willing to merge them together so that further readers can get a full image of how to manipulate such things with this post. — Wjx, Aug 25 '16 at 14:27
@Kuba up to now no Q&A include the content about scrolling or right key clicks. and unfortunately, simple alternation in the code apply for left nouse click will not work...... — Wjx, Aug 26 '16 at 13:45

score 5 · Accepted Answer · answered Aug 28 '16 at 10:12

A summary to all the stuffs:

Needs["JLink`"];
ReinstallJava[];

robotclass = JavaNew["java.awt.Robot"];
LoadJavaClass["java.awt.event.InputEvent"];
LoadJavaClass["java.awt.event.KeyEvent"];
LoadJavaClass["java.awt.Event"];

(*Action*)
mousemove[{x_, y_}] := robotclass@mouseMove[x, y];

leftmousedown[] := robotclass@mousePress[InputEvent`BUTTON1UMASK];
leftmouseup[] := robotclass@mouseRelease[InputEvent`BUTTON1UMASK];
rightmousedown[] := robotclass@mousePress[InputEvent`BUTTON3UMASK];
rightmouseup[] := robotclass@mouseRelease[InputEvent`BUTTON3UMASK];
midmousedown[] := robotclass@mousePress[InputEvent`BUTTON2UMASK];
midmouseup[] := robotclass@mouseRelease[InputEvent`BUTTON2UMASK];
leftmouseclick[t_: 1] := 
  Do[CompoundExpression[leftmousedown[], leftmouseup[]], t];
rightmouseclick[] := 
  CompoundExpression[rightmousedown[], rightmouseup[]];
midmouseclick[] := CompoundExpression[midmousedown[], midmouseup[]];

scroll[x_] := robotclass@mouseWheel@x;

keypress = (Function[k, 
      robotclass[keyPress[Symbol["KeyEvent`VKU" <> k]]]] /@ {##};
    Function[k, 
      robotclass[keyRelease[Symbol["KeyEvent`VKU" <> k]]]] /@ {##}) &;

(*Sense*)
getpixelcolor[{x_, y_}] := 
  StringCases[robotclass@getPixelColor[x, y]@toString[], 
   "=" ~~ d : DigitCharacter .. :> ToExpression@d];

screeninfo = 
  "FullScreenArea" /. 
   Flatten@SystemInformation["Devices", "ScreenInformation"];
screenshot[range : {{x0_, x1_}, {y0_, y1_}} : screeninfo] := 
  GUIScreenShot[range];

The name of those functions already explains everything, so no need for extra explanations~ This piece of code has already include all the key features in 按键精灵, a keypress or mouse move simulator used to automate repetitive works, e.g. play HearthStone. But with the ability of Mathematica, I think this piece of code can do much more! :)

how do I get keypress to emulate "f" – Conor Jul 22 '17 at 11:58 — Conor, Jul 22 '17 at 11:58

How to create keypress simulation program?

Main Problem

Some work I've done

Related

1 Answers1