B4J Library jAWTRobot - invoke keyboard and mouse events, etc...

This library started out as a simple wrapper for Oracle's java.awt.Robot package but it evolved out of hand into a library of general utilities. Some of these functions appear to duplicate functionality already present in B4J however this library is designed to be used in both UI apps and non-UI apps. Consequently, B4J functionality that requires the use of a JFX object (fx.showExternalDocument(), for example) is duplicated here but without the use of any objects only found in UI apps.

Things you can do with this library:
  • programmatically move the mouse and invoke system MouseButtonPressed events or KeyPressed events and much more
  • access the system clipboard
  • get a variety of general operating system, hardware and JVM info
  • get a variety of JVM, process and system performance info (memory usage, CPU load, etc...)
  • take screen shots of arbitrary rectangles on the system's screens
  • run arbitrary commands on the command line
  • redirect Standard Error and Standard Out to file
  • open external files or URLs
  • improved Exception-related stuff
  • improved Thread-related stuff
  • restart your app
The library is very heavily commented so you need only read them to see what is possible.

EDIT (8JUL2015): Added hostNameAndIPAddress() method. Library version remains the same, but it is a new file.
EDIT (30JUL2015): Updated to 1.1. See post #12 for details.
EDIT (25OCT2015): Updated to 1.3. See post #20 for details.
EDIT (26OCT2015): Updated to 1.35. See post #27 for details.
EDIT (22NOV2015): Updated to 1.37. See post #30 for details.
EDIT (30NOV2015): Updated to 1.4. See post #31 for details.
EDIT (30JAN2016): Updated to 1.42. See post #34 for details.
EDIT (31JAN2016): Updated to 1.45. See post #35 for details.
EDIT (31JAN2017): Updated to 1.50. See post #79 for details.
EDIT (02FEB2017): Updated to 1.51. See post #85 for details.
EDIT (06FEB2017): Updated to 1.52. Minor bug fix.
EDIT (04APR2017): Updated to 1.54. Bug fix for relaunchSelfX methods.
EDIT (28DEC2017): Updated to 1.55. Added ScreenCapture2. Read comments for more info.
EDIT (09JAN2019): Updated to 1.59. RobotPaste debugging build
EDIT (14JAN2019): Updated to 1.61. Another RobotPaste2 debugging build
 

Attachments

  • jAWTRobot.zip
    11.1 KB · Views: 612
  • jAWTRobot1.1.zip
    13.6 KB · Views: 377
  • jAWTRobot1.3.zip
    15.1 KB · Views: 321
  • jAWTRobot1.35.zip
    15.5 KB · Views: 351
  • jAWTRobot1.37.zip
    15.7 KB · Views: 332
  • jAWTRobot1.4.zip
    16.1 KB · Views: 379
  • jAWTRobot1.42.zip
    16.6 KB · Views: 292
  • jAWTRobot1.45.zip
    19.7 KB · Views: 510
  • jAWTRobot1.50.zip
    21.6 KB · Views: 338
  • jAWTRobot1.51.zip
    22.1 KB · Views: 300
  • jAWTRobot1.52.zip
    22.1 KB · Views: 379
  • jAWTRobot1.54.zip
    22.1 KB · Views: 512
  • jAWTRobot1.55.zip
    22.8 KB · Views: 576
  • jAWTRobot1.59.zip
    22.6 KB · Views: 228
  • jAWTRobot1.61.zip
    23.2 KB · Views: 808
Last edited:

inakigarm

Well-Known Member
Licensed User
Roycefer, great work. I would like you to add a way to check if the internet connection is present, for example IsRobotConnection true or false that would be really useful.
You can check intenet access trying to check www.google.com (or your own server) with okhttp library and jOkHttpUtils2
 

Roycefer

Well-Known Member
Licensed User
robo.SystemHostNameAndIPAddress() will give you the computer's name and local IP address. You can also get the local IP address with the jNetwork library. I imagine that if the computer isn't connected to a network, in both cases the local IP address will be blank, or maybe just the loopback address. I haven't tested this, but this is how you would detect a network connection.

If you have a network connection, checking if you're connected to the Internet is a matter of attempting to connect to a website (using the jOkHttpUtils2 library). I don't think there's a simple Java API for this that would be a good fit for this library. But it isn't too hard to write your own method using your own website of choice.
 

inakigarm

Well-Known Member
Licensed User
robo.SystemHostNameAndIPAddress() will give you the computer's name and local IP address. You can also get the local IP address with the jNetwork library. I imagine that if the computer isn't connected to a network, in both cases the local IP address will be blank, or maybe just the loopback address. I haven't tested this, but this is how you would detect a network connection.

Keep in mind that you could have a valid local Ip address (obtained by Dhcp for ex) but not an internet connection (lan enabled but wan side - connection to internet disabled). That's why you'd have to check the internet connection checking on a external server with public internet address
 

ThRuST

Well-Known Member
Licensed User
Is also Cut and Copy available in AWTRobot library? I stumbled upon a use for those as well as the great RobotPaste. It would be most useful.
 

Roycefer

Well-Known Member
Licensed User
Version 1.51:

Experimental support for Cut/Copy has been added as well as experimental support for the various relaunch methods on non-Windows systems.

Additionally, a lot of the methods that formerly returned nothing have been changed to return the Me Object so you can chain calls:
B4X:
Log(robo.RobotCut.RobotDelay(10).ClipboardStringGet)
'Cuts the currently selected text, retrieves it from the clipboard and logs it
'The Delay may be unnecessary
 
Last edited:

ThRuST

Well-Known Member
Licensed User
Roycefer, I have so far tried RobotCut on my PC laptop and it works well which is what I needed. Great implementation well done.
I'd like to credit as many as possible in a credits form with firstname 'nickname' Lastname. Perhaps you have a secret nickname :)
 

ThRuST

Well-Known Member
Licensed User
Funny, that's why you code like a virgin :D haha Just kidding :)
How about Roy 'Roycefer' Cefer o_O
 

Roycefer

Well-Known Member
Licensed User
Here is some sample code that demonstrates how to use some of the latest features in the jAWTRobot library:
B4X:
Sub Process_Globals
    Dim srvr As Server
    Dim borg As AWTRobot
    Dim errCounter As Int = 0
    Dim csu As CallSubUtils
End Sub

Sub AppStart()
    srvr.Initialize(blah blah 
    ..... blah blah

    srvr.Start
    csu.CallSubPlus(Me,"BlowUpThread",30*1000)
    borg.B4JCallingThreadNameSet("ProtectedMainThread")             '[1]
    If borg.B4JCallingThreadStartProtectedMessageLoop(Me, "borg") Then        '[2]
        Log("Message Loop ended pleasantly")
    Else   
        Log("Message Loop ended due to an Exception" & DateTime.Time(DateTime.Now))
    End If
    Log("End of thread")
End Sub

Sub borg_MessageLoopException(ThreadName As String, ExceptionMessage As String, StackTrace() As String) As Boolean    '[3]
    Log(ThreadName & " MessageLoop Crashed: " & ExceptionMessage & " stacktrace.length: " & StackTrace.Length)
    Dim tw As TextWriter
    tw.Initialize(File.OpenOutput(File.DirApp, ThreadName & "ErrLog.txt", True))
    tw.Write(DateTime.Date(DateTime.Now) & " " & DateTime.Time(DateTime.Now) & ": " & ExceptionMessage & CRLF)
    For Each ste As String In StackTrace
        tw.Write(TAB & ste & CRLF)
    Next
    tw.Flush
    tw.Close
    errCounter = errCounter + 1
    If errCounter>3 Then Return False   'we will end the message loop after 4 Exceptions
    Return True
End Sub

Sub BlowUpThread                                                    '[4]
    Log("About to blow up " & borg.B4JCallingThreadNameGet & "!!!")
    borg.JVMThrowException("¡¡Custom Exception!!")
End Sub

Explanation:
  1. Here, we are renaming the calling Thread. This is very handy when trying to track bugs or program behavior (especially with a tool like VisualVM). You can rename any Thread from within the Thread. It's very helpful to rename WebSocket Threads and include the userID of the user accessing the WebSocket.
  2. This replaces the normal StartMessageLoop call. Normally, an UncaughtException in the message loop would kill your Thread. But with a protected message loop, you can decide how to proceed when you get an UncaughtException. That's what is going on in [3].
  3. This event will be raised whenever we get an UncaughtException in our message loop. We can return True from this event sub if we wish the message loop to continue or return false to end the message loop. We can also call StopMessageLoop from anywhere in the Thread to end the message loop as we would normally.
  4. Here, we are throwing a custom Exception. This will raise the borg_MessageLoopException event in ProtectedMainThead.
You can start a protected message loop in any Thread. If you are going to do it in a WebSocket, it's best to do it in the WebSocket_Connected event.
 

stevel05

Expert
Licensed User
Here's an interesting issue that I hope you can help me solve.

Sending alt+page_up or alt+page_down acts as if you were pressing alt+numberpad_9 or alt+numberpad_3. I attach a small app to demonstrate the issue.

The page_up and page_down buttons work as expected, but the alt+page_up and alt+page_down buttons add a character.

If you physically enter alt+page_up or alt+page_down on a keyboard (assuming you have a separate page_up and page_down buttons) it does nothing, which is to be expected. If you physically press alt+numberpad_9 or alt+numberpad_3 it behaves the same as the Robot sending alt+page_up and alt+page_down buttons and adds a character.

I have searched google, but cannot find an explanation, just wondered if you had seen this before, or have an idea about where I should look to resolve it.

Thanks

Steve
 

Attachments

  • RobotTest.zip
    4.5 KB · Views: 277

Roycefer

Well-Known Member
Licensed User
My experiments and research indicate the following:
  • With Number Lock engaged, the number pad will type out numbers.
  • With Number Lock NOT engaged, the number pad will execute the alternative function of those number pad keys. 7 is "Home", 8 is "up", 9 is "page_up", etc...
  • On Windows, you can enter in special characters by pressing Alt and typing in the special character code with the number pad. This behavior ignores the Number Lock and will only work on number pad keys. For example, type Alt_168 (on the number pad) to get: ¿.
  • It looks like the AWTRobot "presses" the number pad page_up and page_down keys when asked to to "press" page_up/down, not the dedicated page_up/down keys (perhaps because the dedicated keys are not guaranteed to be there). Therefore, when the AWTRobot is asked to press "alt_page_down", it presses "alt_numberpad_3", which activates the Windows special character code for ♥.
  • Looking here: http://docs.oracle.com/javase/6/docs/api/constant-values.html#java.awt.event.KeyEvent.VK_PAGE_UP it seems that Java keycodes for page_up and numberpad_9 are different but they have the effect of "pressing" the same key. It's also interesting to note that the jNativeHookB4J library reports the same key codes for page_up and non-NumberLocked numberpad_9.
If you don't like this behavior, you can try consuming the event with the jNativeHookB4J library and then executing your own behavior. I haven't tested this, though and it doesn't seem like a desirable solution (it won't be able to distinguish between dedicated page_up and non-NumberLocked numberpad_9). What behavior are you trying to achieve for your program?
 

stevel05

Expert
Licensed User
Thanks for the reply. I am trying to send and alt+page_up and alt+page_down key combination (the dedicated keys) in response to a button click. I think you have confirmed my suspicions. I will look to see it there is an alternative route for these key combinations.
 

stevel05

Expert
Licensed User
This is what I have so far, it may be of use to someone. Using sendkeys with jScript in a batch seems to stop the additional character being displayed, and the keys can be caught within B4j, so it should work for B4j Apps. There are some programs I am testing that it doesn't work with, these could be coded at a lower level to the hardware page_up / page_down keys. Unless I can find a windows library that can send specific low level keycodes, I think this is a good as it's going to get.

I would think it may be possible to use the jScript library to achieve the same result, but this solution is not too slow and it's not going to be high on my priority list unless I can get it all working.
 

Attachments

  • RobotTest2.zip
    5 KB · Views: 262

Peter Simpson

Expert
Licensed User
Great library @Roycefer, but I have one small issue.
If I use AWTRobot.SystemMemoryPhysicalTotal or AWTRobot.SystemMemoryPhysicalFree both the results displays the same figures, I would have expected one figure to be more or less than the other figure.

The following logs are 100% incorrect, the 2 methods always reads identically.
PhysicalTotal = 2147483647
PhysicalFree = 2147483647

Thank you...
 

Roycefer

Well-Known Member
Licensed User
Those methods work correctly on my Windows 7 64-bit system, 32-bit system and RasPi2. They use the standard Java APIs. I'll need more info on your setup if I'm to do anything. I'm assuming you've already thoroughly proofread your code.
 

Peter Simpson

Expert
Licensed User
Those methods work correctly on my Windows 7 64-bit system, 32-bit system and RasPi2. They use the standard Java APIs. I'll need more info on your setup if I'm to do anything. I'm assuming you've already thoroughly proofread your code.

Thank you for that, it's working now even though all I did was nothing. I posted my question to you last night then I turned my laptop straight off. This morning I turned my laptop on and I read your reply to me (so thank you for that). I decided to reply with my code but I first ran the code just so that I could post the logs as well, to my complete surprise the logs read as follows :confused:

PhysicalTotal = 12711329792
PhysicalFree = 7109734400

Thank you for responding to my post @Roycefer, but I'm completely lost for words as all I've done is basically absolutely nothing :)
 
Top