Android Question Image processing too slow...

freedom2000 · Nov 29, 2015

Hi all,

I wanted to do something like this : https://sites.google.com/site/todddanko/home/webcam_laser_ranger

I have thus added a small red laser under my phone and started to code an app.

Basically it works (or starts to work) but it is very very slow...

I am using the excellent @JordiCP camera NV21utils library to sample images, and this works quite fast.

The difficult task is to determine the dot position into the image.

The code is here :

Code:

Sub ComputeImage
    Processing = True
    Dim p() As Int
    Dim r As Reflector
    Dim h, s, v As Float
    Dim HSV(3) As Float = Array As Float(h, s, v)
    Dim found As Boolean = False
    Dim i, j As Int
        found = False
        Try
            For j = Ylaser -10 To Ylaser+10
                Log("ligne " & j)
                DoEvents
                '
             
                p = getPixels(myBitmap, 0, 0, myBitmap.Width/2, j ,  myBitmap.Width/2, 1)
                For i = 0 To myBitmap.Width/2-1
                    MyColor = GetARGB(p(i))
                    If (MyColor(1) > 250) Then
                        r.RunStaticMethod("android.graphics.Color", "RGBToHSV", Array As Object(MyColor(1), MyColor(2), MyColor(3), HSV), Array As String("java.lang.int", "java.lang.int", "java.lang.int", "[F"))
                        If (HSV(0) > 320) Then 'And (HSV(2) > 60)Then 
                            LbFps.Text = i & HSV(0) & " " & HSV(2)
                            LbFps.Color = p(i)
                            found = True
                            Log("found")
                        End If
                    End If
                 
                    If found = True Then Exit
                Next
                If found = True Then Exit
                'DoEvents
            Next
        Catch
            Log("error " & LastException.Message)
        End Try
        Processing = False
End Sub

It is based on getpixel function + "RGBToHSV"

Code:

Sub getPixels(bmp As Bitmap, offset As Int, stride As Int, x As Int, y As Int, width As Int, height As Int) As Int()
    Try
    Dim jo = bmp As JavaObject
    Dim pixels(width * height) As Int
    jo.RunMethod("getPixels", Array As Object(pixels, offset, width, x, y, width, height))
    Return pixels
    Catch
    Return Null
    End Try
End Sub

And this is slow...

I attach to this thread the whole project.
It compiles and runs but slowly.

Any idea to access in a fast way to a rectagle into an image and to perform some RGB comparisons on pixels values ?

Thanks

strat · Nov 30, 2015

Try to remove DoEvents line in the loop if it doesn't effect your code to work.

freedom2000 · Nov 30, 2015

Yes you are right, these DoEvents were there to allow UI refresh.
If not, Ui is completly frozen by the image processing stuff.

But with or without "DoEvents" the code is soooo slow !

inakigarm · Nov 30, 2015

I don't know if that can help but OpenCV getpixel should be quicker :

OpenCV getpixel: http://docs.opencv.org/2.4/doc/user_guide/ug_mat.html

Accessing OpenCV from B4A- inline java: https://www.b4x.com/android/forum/threads/inline-javacv-opencv.53963/#post-381504 (thanks to @DrewG )

(Accelerated Surface from @Informatix also has this method)

freedom2000 · Nov 30, 2015

Thank you for this answer --> seems very promissing

JordiCP · Dec 1, 2015

Hi

I have seen the link, very interesting. May be wrong, but if you are trying to find the brightest dots, you could try it directly with the Y values of the preview array (the highest, the lightest) so you will not need the lib nor the getpixel nor the rgb2hsv conversion, and it will be much much faster

It is explained a bit here

If this is not the case, I can change the output of the lib to give directly an rgb array

freedom2000 · Dec 1, 2015

JordiCP said:
Hi

I have seen the link, very interesting. May be wrong, but if you are trying to find the brightest dots, you could try it directly with the Y values of the preview array (the highest, the lightest) so you will not need the lib nor the getpixel nor the rgb2hsv conversion, and it will be much much faster

It is explained a bit here

If this is not the case, I can change the output of the lib to give directly an rgb array

Thank you for this very positive answer !

However looking at the pixel brightness will not work... When you are quite far (5m) from the dot, it is not very bright.
So the trick is to look at "redish area". The position of the dot can be searched into a rectangular box as it is moving along a line when the distance increases.

The best is thus to search into this predifined box for "hue" in the "red range".

I don't know if it would be easy for you to mod your lib.
But if yes it would be great !

I will keep the code fully opened !

JordiCP · Dec 1, 2015

Hi,

I have added a new method that converts the full YUV array to an array of ints in ARGB format and proper screen orientation

Similar to what was done with the previous method, this time you need to define an array of ints (taking into account the scale factor.) which will be passed to the method as a param.

Find attached the modified lib and the example with "test=1" (to test this new method. ). It will show no info in the screen, just will log the R,G,B components of the central screen pixel (but it converts all of them)

This will surely speed this part. But I don't know if the next steps (getting R,G,B components form the color and converting to HSV) will be the real bottlenecks

Don't hesitate to tell me if you need more conversions. These days I am modifying another library using the NDK so it will be easier for me to make changes as I have everything fresh in my head

freedom2000 · Dec 1, 2015

Wahou --> impressive

I will try it tonight and keep you informed

THANK YOU

freedom2000 · Dec 1, 2015

I have started to test.
I use full resolution image (to get better precision)

With your lib I can acquire the full buffer every 1,05s. With my Galaxy S5 and 1080 * 1920 pix

With "getpixels" I achieve... exactly the same result 1,05s

here is the code modified

Code:

Sub Camera1_Preview (PreviewPic() As Byte)
    'prevent queued events from previous camera settings, if any. Just in case
    If DateTime.Now > lastPreviewSaved + IntervalMs And (Processing = False) Then
   
        lastPreviewSaved = DateTime.Now

        'prevent queued events from previous camera settings, if any. Just in case
        If PreviewPic.Length<>(3*myBitmap.Width*myScale*myBitmap.Height*myScale/2) Then
            Log("Not processing")
            Return
        End If

        NV21toRGB.proceed( PreviewPic, myBitmap.Width*myScale, myBitmap.Height*myScale, myScale, myBitmap, camVertical , camEx.Front, camEx.DispRotation, myIndexEffect )
        myPreviewRGBArray = getPixels(myBitmap, 0, 0, 0, 0 ,  myBitmap.Width, myBitmap.Height)
        Log(" freq = " & (DateTime.Now - LastTime))
         'new method. Converts the PreviewPic() array to ARGB format in myPreviewRGBArray()
        'NV21toRGB.proceedArray(PreviewPic, myBitmap.Width*myScale, myBitmap.Height*myScale, myScale, myPreviewRGBArray, camVertical , camEx.Front, camEx.DispRotation, 0)
        'Dim centralColor As Int = myPreviewRGBArray( (myArrayWidth*myArrayHeight/2)+(myArrayWidth/2))
        'Dim R As Int = Bit.And(Bit.ShiftRight(centralColor,16),0xFF)
        'Dim G As Int = Bit.And(Bit.ShiftRight(centralColor, 8),0xFF)
        'Dim B As Int = Bit.And(centralColor,0xFF)
        'Log("CentralPoint RGB is (ARGB)="&R&","&G&","&B & " freq = " & (DateTime.Now - LastTime))
   
        LastTime = DateTime.Now
       
        'ComputeImage
    End If
End Sub

(Just have to uncomment your code and comment the getpixels part.)

In conclusion it is the rest of the code --> "RGBToHSV" which is very long...

BUT

After playing a while I have also found that I called the getpixels function "line per line" in ComputeImage function.
And I call it 20 times...
And whatever is the size of the cropped Image I want to get, the time is the same 1,05s (only depending on the preview size)

So I do spend more than 20s just calling GetPixels ... without any processing

--> I'd better call it only once and then process the buffer "line per line"

I will mod my code

Again thank you for your help

freedom2000 · Dec 1, 2015

Well... good news

Just calling once the getpixels function improves the performances by a 20 times ratio...

The remaining of the computeImage function is negligeable and I can acquire frames every 1s which should be enough for my application.

I will try to finish and I will post the code !

JordiCP · Dec 1, 2015

Good news, go for it!

Just to test (I would do it, just for curiosity

) if this second can be easilly reduced, I would suggest a try

If you are processing 21 lines each time (from Ylaser -10 To Ylaser+10), you could try to convert only those ones with getpixels (and see if is faster).
So, you could change your line

Code:

myPreviewRGBArray = getPixels(myBitmap, 0, 0, 0, 0 , myBitmap.Width, myBitmap.Height)

to

Code:

myPreviewRGBArray = getPixels(myBitmap, 0, 0, 0, Ylaser-10 , myBitmap.Width, 21)

You would have a 21 lines buffer with the area to be processed. Only changing the row index in "ComputeImage" accordingly would have the same results.

freedom2000 · Dec 2, 2015

Yes you are right --> this is the test I already did ! (And whatever is the size of the cropped Image I want to get, the time is the same 1,05s (only depending on the preview size)

It seems that the time is the same for a small cropping area or for the whole image.
Frankly speaking I don't understand why and I had been very disapointed by this result !
I will have to check again as I did it very fast yesterday evening ; so a double check will be better...

freedom2000 · Dec 2, 2015

Hi again,

Well ... I for sure did the test too fast yesterday evening...

Here is the code :

Code:

        NV21toRGB.proceed( PreviewPic, myBitmap.Width*myScale, myBitmap.Height*myScale, myScale, myBitmap, camVertical , camEx.Front, camEx.DispRotation, myIndexEffect )
        LastTime = DateTime.Now
        For i = 1 To 10
            myPreviewRGBArray = getPixels(myBitmap, 0, 0, 0, 0 ,  myBitmap.Width,myBitmap.Height)
        Next
        Log("time full frame = " & (DateTime.Now - LastTime))

     
        LastTime = DateTime.Now
        For i = 1 To 10
            myPreviewRGBArray = getPixels(myBitmap, 0, 0, 0, 0 ,  myBitmap.Width/2, 21)
        Next
        Log("time cropped frame = " & (DateTime.Now - LastTime))

And the result... much faster with a cropped area rather than with a full frame

Code:

** Activity (main) Resume **
0
preview size[Height=1080, IsInitialized=false, Width=1920
]
time full frame = 586
time cropped frame = 7
time full frame = 568
time cropped frame = 6
time full frame = 520
time cropped frame = 9
time full frame = 492
time cropped frame = 6
time full frame = 536
time cropped frame = 8
time full frame = 607
time cropped frame = 11
time full frame = 529
time cropped frame = 6
time full frame = 546
time cropped frame = 9
time full frame = 555
time cropped frame = 12
time full frame = 546
time cropped frame = 6
time full frame = 524
time cropped frame = 5
time full frame = 473
time cropped frame = 5
time full frame = 489
time cropped frame = 6
** Activity (main) Pause, UserClosed = false **
** Activity (main) Resume **

So @JordiCP you were totally right

Yesterday I measured the full loop time and not only the getpixels time...

Android Question Image processing too slow...

freedom2000

Well-Known Member

Attachments

strat

Active Member

freedom2000

Well-Known Member

inakigarm

Well-Known Member

freedom2000

Well-Known Member

JordiCP

Expert

freedom2000

Well-Known Member

JordiCP

Expert

Attachments

freedom2000

Well-Known Member

freedom2000

Well-Known Member

freedom2000

Well-Known Member

JordiCP

Expert

freedom2000

Well-Known Member

freedom2000

Well-Known Member

Similar Threads