Android Question Correct showing the screenshots of other screens

peacemaker

Expert
Licensed User
Longtime User
Happy NY, people !

Task is to properly scale the screenshots of various Android devices that got from remote devices and get the tap coordinate over them (for remote control).
I have
1) Screenshot that made by MediaProjection
2) And GetDeviceLayoutValues.Width and GetDeviceLayoutValues.Height of this device

1) Screenshots are different kind, if from different devices: if "old" device with mechanical\external system keys HOME, BACK, RECENT APPS - the screenshot has no these keys on the bottom.
If new device with the virtual system keys on the screen bottom - they are on the screenshot.
1.jpg2.jpg

2) GetDeviceLayoutValues.Height - does not correspond the real screen vertical size in pixels, seem, the notification area (optionally) and\or Activity.Title height (also optionally) make influence...

But if to see the screenshot and making taps and swipes over them - the dimension and proportions must be exacly correct.
Super sub from Erel
B4X:
Sub FillImageToView(bmp As Bitmap, ImageView As View)
    Dim bmpRatio As Float = bmp.Width / bmp.Height
    Dim viewRatio As Float = ImageView.Width / ImageView.Height
    If viewRatio > bmpRatio Then
        Dim NewHeight As Int = bmp.Width / viewRatio
        bmp = bmp.Crop(0, bmp.Height / 2 - NewHeight / 2, bmp.Width, NewHeight)
    Else if viewRatio < bmpRatio Then
        Dim NewWidth As Int = bmp.Height * viewRatio
        bmp = bmp.Crop(bmp.Width / 2 - NewWidth / 2, 0, NewWidth, bmp.Height)
    End If
    ImageView.SetBackgroundImage(bmp.Resize(ImageView.Width, ImageView.Height, True))
End Sub

helps not much.
Is there way to correct setup screenshots showing (on the fullscreen Activity, IncludeTitle=false) and tapping ?
 
Last edited:

peacemaker

Expert
Licensed User
Longtime User
Here those 2 screenshots shown on the same Android9 device with the virtual system keys.
1) Manually done screenshots are 1080 x 2340 px - fully correct size of the screen
2) But the screenshot done via MediaProjection was 1088 x 2131 px from the same Android9 device.
3) The fullscreen Activity that shows the fullscreen screenshot - anyway reserved the notif area at the top, extra space...

Seems, this bottom area of the virtual system keys is the trouble, and notif area at the top...
 

Attachments

  • Screenshot_20200101-121522.jpg
    Screenshot_20200101-121522.jpg
    241.6 KB · Views: 264
  • Screenshot_20200101-121537.jpg
    Screenshot_20200101-121537.jpg
    101.9 KB · Views: 259
Last edited:
Upvote 0

peacemaker

Expert
Licensed User
Longtime User
Thanks Erel, but is it possible to get the screenshot (on the source device) and show it (on another device) so that tap coordinates (on another device) are to be fully corresponding (of the source device) ?

As i can understand - the notif area, Activity.Title and bottom system buttons area (of latest Android) - are all optional on the screenshot ?
 
Upvote 0

peacemaker

Expert
Licensed User
Longtime User
:) yes, this is the question.
Remote control is at least:
1) Seeing the screen of the remote device (the screenshot)
2) And possibility to tap, long-tap, swipe over this screenshot (BTW, it's already works OK), but on another screen, with another size and scale.
3) And getting the tap coordinates that correspond to the remote display.
 
Upvote 0

peacemaker

Expert
Licensed User
Longtime User

Not fully. In your example, as i can understand, just one panel of drawing, all parameters are absolute.
But common remote control (as RDP in Windows, or Teamviewer in Android) - the full screen is visible, and should be tappable... by the exact screen coordinates: tap on my screen1 to simulate tap on remote screen2.
Size, scale, density of Screen1 and Screen2 are not same, in whole.
 
Last edited:
Upvote 0

LucaMs

Expert
Licensed User
Longtime User
Not fully. In your example, as i can understand, just one panel of drawing, all parameters are absolute.
But common remote control (as RDP in Windows, or Teamviewer in Android) - the full screen is visible, and should be tappable... by the exact screen coordinates: tap on my screen1 to simalate tap on remote screen2.
Size, scale, density of Screen1 and Screen2 are not same, in whole.
Sorry, it is not my example, Erel did it and, as you all know, he is always very approximate in his examples 🤣
(sooner or later Erel will beat me 🤣)

I meant to say that the ultimate goal I guess is something of that kind.

In a certain sense, you would like to "normalize", to manage the remote screen as if it were on your display (eventually displaying on your device a panel bigger than your display and swipe it).

Proportions, density, scales... Also Klaus could come into play here :)
 
Upvote 0

Sagenut

Expert
Licensed User
Longtime User
When you have the screenshot on your device, get the coordinates of the click and transform them in Percent.
Passing back this information to the remote device should permit to replicate the click in the relative position, transforming the Percent into coordinates with the remote device width and height.
 
Upvote 0

LucaMs

Expert
Licensed User
Longtime User
When you have the screenshot on your device, get the coordinates of the click and transform them in Percent.
Passing back this information to the remote device should permit to replicate the click in the relative position, transforming the Percent into coordinates with the remote device width and height.
Not so easy. Percent of? The displays may have different "ratio" w/h (thank you, display manufacturers; I look forward to television displays other than the current 16:9, also 13:10, 8:500, etc :mad: )

In addition, density also affects.
 
Upvote 0

Sagenut

Expert
Licensed User
Longtime User
Percent of the Imageview that contain the screenshot of the remote device.
Of course I don't know if @peacemaker is using an Imageview or something else.
If the Imageview is 400 x 800 (just an example), clicking the center at 200, 400 will produce 50%x and 50%y.
Sending this percents to a device with screen of 1080 x 1920 will results in a click at 540, 960...... so the screen center of the remote device.
 
Upvote 0

peacemaker

Expert
Licensed User
Longtime User
Yes ! if the screenshot is fully scaled to an ImageView with some proportional sides - It needs to send the tap coordinates in %, not in pixels. But before simulation - they must be recalculated into pixels of this local screen.
 
Upvote 0

LucaMs

Expert
Licensed User
Longtime User
Percent of the Imageview that contain the screenshot of the remote device.
Of course I don't know if @peacemaker is using an Imageview or something else.
If the Imageview is 400 x 800 (just an example), clicking the center at 200, 400 will produce 50%x and 50%y.
Sending this percents to a device with screen of 1080 x 1920 will results in a click at 540, 960...... so the screen center of the remote device.
1577973705147.png
 
Upvote 0

Sagenut

Expert
Licensed User
Longtime User
As @peacemaker wrote the imageview of the remote device must be scaled to keep the aspect ratio.
 
Upvote 0
Top