B4J Tutorial [example] [B4X] Control the desktop with your phone

This example shows how simple it is to create network solutions with B4J, B4A and B4i.

In this case the network communication is based on UDP.

It implements a remote desktop mouse, where the Android device or iOS device control the desktop mouse.

The B4J app uses jAWTRobot library to control the mouse.
The commands are serialized with B4XSerializator.

Note that the B4J app doesn't show any UI except of a system tray icon.
It uses ControlsFX library to show notifications. Note that this requires adding a dummy form (with a UTILITY style to hide it from the task bar).

SS-2015-11-30_16.21.26.png


upload_2015-11-30_16-26-42.png
upload_2015-11-30_17-1-11.png


The nice thing about this example is that the code is very short and simple.

The desktop app listens on port 54323. See this post for information related to the firewall: https://www.b4x.com/android/forum/t...-read-the-logs-in-release-mode.51164/#content

An extended version that works over UDP, TCP or MQTT is available in post #6.
 

Attachments

  • RemoteDesktopControl_B4J.zip
    2.6 KB · Views: 2,124
  • RemoteDesktopControl_B4A.zip
    9.3 KB · Views: 2,123
  • RemoteDesktopControl_B4i.zip
    4.1 KB · Views: 1,375
Last edited:

Erel

B4X founder
Staff member
Licensed User
Longtime User
UDP is a bit faster. Though you won't be able to see any difference in a solution such as this one.

UDP is stateless. It makes it easier to work with for a task such as this one. The desktop app just listens for incoming data and the clients send data to the correct port. There is no connection or session involved.

The downside of UDP compared to TCP is that packets can be lost.
 

Erel

B4X founder
Staff member
Licensed User
Longtime User
upload_2015-12-2_13-18-37.png


An extended example that works with UDP ,TCP and MQTT is attached.

Unlike what I previously wrote, UDP behaves significantly better in this case.

This is a good example of using classes to easily switch between different network solutions.
 

Attachments

  • DesktopController_B4i.zip
    5.9 KB · Views: 1,047
  • DesktopController_B4J.zip
    4 KB · Views: 1,476
  • DesktopController_B4A.zip
    11.5 KB · Views: 1,469

micro

Well-Known Member
Licensed User
Longtime User
Hi Erel
a question, you think it's difficult send mouse movements together a rapid screenshot of the desktop?
Of course with appropriate scaling on the device.
It would be great to have a real remote desktop!
 

Roycefer

Well-Known Member
Licensed User
Longtime User
The jAWTRobot library can take screenshots. The problem is that taking and sending screenshots at a rate that simulates video can use up a bit of processing power and a lot of bandwidth. There are some schemes I've used whereby you only send small rectangles that have changed their appearance. But it's still not a trivial system to implement.
 

derez

Expert
Licensed User
Longtime User
I have a non-ui server that handles commands and sends screenshots, using the jAWTRobot
B4X:
Public Sub Handle(req As ServletRequest, resp As ServletResponse)
    robot.ScreenCurrentRectangleSetAsArbitrary(0,0,1280,720)
    img  = robot.ScreenCaptureAsByteArray
    Dim su As StringUtils
    resp.Write("<img width=1280 height=720 src='data:image/png;base64,")
    resp.Write(su.EncodeBase64(img))
    resp.Write("'/img>").write("</br>")
End Sub

But it is working with a messageloop. How can I combine the UI application above with the Non-ui ?
The receiving app (b4a) code is this:
B4X:
Sub JobDone (Job As HttpJob)
     If Job.Success = True Then
       Select Job.jobname
          ...           
        Case "screen"
            Dim st As String = Job.GetString
            Dim su As StringUtils
            Dim barr() As Byte
            barr = su.DecodeBase64(st)
            Dim InputStream1 As InputStream
            InputStream1.InitializeFromBytesArray(barr, 0, barr.Length)
            Dim bmp As Bitmap
            bmp.Initialize2(InputStream1)
            iv.SetBackgroundImage(bmp)
            iv.Visible = True
           
    End Select
Sending the screenshot every few seconds is enough for my needs.
In another way, how to send the picture (lets say by a timer) and how to get it in the receiver app.
 
Last edited:

derez

Expert
Licensed User
Longtime User
I managed to do it with TCP because it has a two way connection (while UDP requires the ip of the receiver which I don't have as a fix ip).
The server sends this with a timer :
B4X:
robot.ScreenCurrentRectangleSetAsArbitrary(0,0,1280,720)
img  = robot.ScreenCaptureAsByteArray
astream.Write(img)
The b4a use this in the TCPconnector:
B4X:
Sub astream_NewData (Buffer() As Byte)
    Dim InputStream1 As InputStream
    InputStream1.InitializeFromBytesArray(Buffer, 0, Buffer.Length)
    bmp.Initialize2(InputStream1)
End Sub
What is left to do is to send somehow the position of the mouse and display it on the image (or draw it and then send ?) because it is not included in the screenshot.
 

derez

Expert
Licensed User
Longtime User
I would like to know the position of the mouse in the b4j app.
I believe it should be retrieved somehow by the remotecommand object.
 

derez

Expert
Licensed User
Longtime User
The attached files work as a remote control of a PC by a b4a device, sending mouse movements and receiving the screenshot with possibility of zoom in and out.
I left only the TCP for communication.
It works with my nexus5, I have not scaled it yet to work with anything else.
The port and host ip are different from the original because this is what I use at home, so change them for your setup.
It is just a demo !
 

Attachments

  • RemoteController2_B4A.zip
    11.9 KB · Views: 1,273
  • RemoteController2_B4J.zip
    3.6 KB · Views: 1,203

tufanv

Expert
Licensed User
Longtime User
This example shows how simple it is to create network solutions with B4J, B4A and B4i.

In this case the network communication is based on UDP.

It implements a remote desktop mouse, where the Android device or iOS device control the desktop mouse.

The B4J app uses jAWTRobot library to control the mouse.
The commands are serialized with B4XSerializator.

Note that the B4J app doesn't show any UI except of a system tray icon.
It uses ControlsFX library to show notifications. Note that this requires adding a dummy form (with a UTILITY style to hide it from the task bar).

SS-2015-11-30_16.21.26.png


View attachment 39318 View attachment 39323

The nice thing about this example is that the code is very short and simple.

The desktop app listens on port 54323. See this post for information related to the firewall: https://www.b4x.com/android/forum/t...-read-the-logs-in-release-mode.51164/#content

An extended version that works over UDP, TCP or MQTT is available in post #6.

if we want to use this for mac , how does the firewall port allowing works ? Because in windows , it automaticly requests for authorization with a popup , if we want to use it with mac , how can we allow the firewall inside the app , or does it have to be manually added ?
 

Hypnos

Active Member
Licensed User
Longtime User
Anyone know what is this error?

B4X:
Picked up _JAVA_OPTIONS: -Djavax.net.ssl.trustStore=c:\windows\sun\java\deployment\trusted.cacerts
main._shownotification (java line: 163)
java.lang.NoSuchMethodError: javafx.stage.Window.impl_getWindows()Ljava/util/Iterator;
    at org.controlsfx.tools.Utils.getWindow(Utils.java:50)
    at org.controlsfx.control.Notifications$NotificationPopupHandler.show(Notifications.java:310)
    at org.controlsfx.control.Notifications.show(Notifications.java:269)
    at org.controlsfx.control.Notifications.showInformation(Notifications.java:244)
    at anywheresoftware.b4j.objects.ControlsUtils.ShowNotification3(ControlsUtils.java:138)
    at b4j.example.main._shownotification(main.java:163)
    at b4j.example.tcpconnector._showip(tcpconnector.java:87)
    at b4j.example.tcpconnector._initialize(tcpconnector.java:75)
    at b4j.example.main._appstart(main.java:95)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at anywheresoftware.b4a.BA.raiseEvent2(BA.java:91)
    at anywheresoftware.b4a.BA.raiseEvent(BA.java:78)
    at b4j.example.main.start(main.java:37)
    at javafx.graphics/com.sun.javafx.application.LauncherImpl.lambda$launchApplication1$9(LauncherImpl.java:846)
    at javafx.graphics/com.sun.javafx.application.PlatformImpl.lambda$runAndWait$12(PlatformImpl.java:455)
    at javafx.graphics/com.sun.javafx.application.PlatformImpl.lambda$runLater$10(PlatformImpl.java:428)
    at java.base/java.security.AccessController.doPrivileged(Native Method)
    at javafx.graphics/com.sun.javafx.application.PlatformImpl.lambda$runLater$11(PlatformImpl.java:427)
    at javafx.graphics/com.sun.glass.ui.InvokeLaterDispatcher$Future.run(InvokeLaterDispatcher.java:96)
    at javafx.graphics/com.sun.glass.ui.win.WinApplication._runLoop(Native Method)
    at javafx.graphics/com.sun.glass.ui.win.WinApplication.lambda$runLoop$3(WinApplication.java:174)
    at java.base/java.lang.Thread.run(Thread.java:834)
 

Ronce

Member
Licensed User
Anyone know what is this error?

B4X:
Picked up _JAVA_OPTIONS: -Djavax.net.ssl.trustStore=c:\windows\sun\java\deployment\trusted.cacerts
main._shownotification (java line: 163)
java.lang.NoSuchMethodError: javafx.stage.Window.impl_getWindows()Ljava/util/Iterator;
    at org.controlsfx.tools.Utils.getWindow(Utils.java:50)
    at org.controlsfx.control.Notifications$NotificationPopupHandler.show(Notifications.java:310)
    at org.controlsfx.control.Notifications.show(Notifications.java:269)
    at org.controlsfx.control.Notifications.showInformation(Notifications.java:244)
    at anywheresoftware.b4j.objects.ControlsUtils.ShowNotification3(ControlsUtils.java:138)
    at b4j.example.main._shownotification(main.java:163)
    at b4j.example.tcpconnector._showip(tcpconnector.java:87)
    at b4j.example.tcpconnector._initialize(tcpconnector.java:75)
    at b4j.example.main._appstart(main.java:95)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at anywheresoftware.b4a.BA.raiseEvent2(BA.java:91)
    at anywheresoftware.b4a.BA.raiseEvent(BA.java:78)
    at b4j.example.main.start(main.java:37)
    at javafx.graphics/com.sun.javafx.application.LauncherImpl.lambda$launchApplication1$9(LauncherImpl.java:846)
    at javafx.graphics/com.sun.javafx.application.PlatformImpl.lambda$runAndWait$12(PlatformImpl.java:455)
    at javafx.graphics/com.sun.javafx.application.PlatformImpl.lambda$runLater$10(PlatformImpl.java:428)
    at java.base/java.security.AccessController.doPrivileged(Native Method)
    at javafx.graphics/com.sun.javafx.application.PlatformImpl.lambda$runLater$11(PlatformImpl.java:427)
    at javafx.graphics/com.sun.glass.ui.InvokeLaterDispatcher$Future.run(InvokeLaterDispatcher.java:96)
    at javafx.graphics/com.sun.glass.ui.win.WinApplication._runLoop(Native Method)
    at javafx.graphics/com.sun.glass.ui.win.WinApplication.lambda$runLoop$3(WinApplication.java:174)
    at java.base/java.lang.Thread.run(Thread.java:834)

I don't know why that error happens...But to omit that error I remove the "Main.ShowNotification" code under public sub ShowIp and error gone then the program works :)

B4X:
Public Sub ShowIp
'    Main.ShowNotification("UDP connector",$"Waiting for UDP data.
'Make sure that the following port is open in the firewall (UDP): ${udpsock.Port}"$, False)
End Sub
 

zabayin

Member
Licensed User
I've changed this project to B4X but never success. the packet is well-received by B4J but in the handle command, it seems needs to be a little modification. I'm stuck in this.
B4X:
(ClassNotFoundException) java.lang.ClassNotFoundException: b4j.example.b4xmainpage$_remotecommand
I know this is an old thread But that thing should be here i think. Sorry Sir.
 
Top