Android Question [closed] Google vision textrecognizer: Frame imageformat NV21/YUY2 error

[KMatle]

I'm playing with the textrecognizer class of Google vision and I'm getting an error about the image format coming from Camera_Preview and

In Camera_Preview (note that there is another builder which isn't used) the error occurs on the last line

Dim SparseArrayTD As JavaObject = textdetector.RunMethod("detect", Array(frametext))

Imagedata is set with:

frameBuildertext.RunMethod("setImageData", Array(bbtext, cs.Width, cs.Height,  842094169))

Sub Camera1_Preview (data() As Byte)
    
    If DateTime.Now > LastPreview + IntervalBetweenPreviewsMs Then
        'Dim n As Long = DateTime.Now
        cvs.ClearRect(cvs.TargetRect)
        Dim frameBuilder,frameBuildertext As JavaObject
        Dim bb As JavaObject
        bb = bb.InitializeStatic("java.nio.ByteBuffer").RunMethod("wrap", Array(data))
        frameBuilder.InitializeNewInstance("com/google/android/gms/vision/Frame.Builder".Replace("/", "."), Null)
        frameBuildertext.InitializeNewInstance("com/google/android/gms/vision/Frame.Builder".Replace("/", "."), Null)
        Dim cs As CameraSize = camEx.GetPreviewSize
        frameBuilder.RunMethod("setImageData", Array(bb, cs.Width, cs.Height,  842094169))
                
        Dim bbtext As JavaObject
        bbtext = bbtext.InitializeStatic("java.nio.ByteBuffer").RunMethod("wrap", Array(data))
        frameBuildertext.RunMethod("setImageData", Array(bbtext, cs.Width, cs.Height,  842094169))
                
        Dim frame As JavaObject = frameBuilder.RunMethod("build", Null)
        Dim frametext As JavaObject = frameBuildertext.RunMethod("build", Null) 'works
        
        Dim SparseArray As JavaObject = detector.RunMethod("detect", Array(frame))
        Dim SparseArrayTD As JavaObject = textdetector.RunMethod("detect", Array(frametext)) 

...

This works and the detector is operational

Private Sub CreateTextDetector
    Dim ctxt As JavaObject
    ctxt.InitializeContext
    Dim builder As JavaObject
    builder.InitializeNewInstance("com.google.android.gms.vision.text/TextRecognizer.Builder".Replace("/", "."), Array(ctxt))
    Dim barcodeClass As String = "com.google.android.gms.vision.text.TextRecognizer".Replace("/", ".")
    Dim barcodeStatic As JavaObject
    barcodeStatic.InitializeStatic(barcodeClass)
    textdetector = builder.RunMethod("build", Null)
    Dim operational As Boolean = textdetector.RunMethod("isOperational", Null)
    Log("Is detector operational: " & operational)
    
End Sub

Error:

java.lang.reflect.InvocationTargetException
    at java.lang.reflect.Method.invoke(Native Method)
    at anywheresoftware.b4j.object.JavaObject.RunMethod(JavaObject.java:132)
    at java.lang.reflect.Method.invoke(Native Method)
    at anywheresoftware.b4a.shell.Shell.runMethod(Shell.java:732)
    at anywheresoftware.b4a.shell.Shell.raiseEventImpl(Shell.java:348)
    at anywheresoftware.b4a.shell.Shell.raiseEvent(Shell.java:255)
    at java.lang.reflect.Method.invoke(Native Method)
    at anywheresoftware.b4a.ShellBA.raiseEvent2(ShellBA.java:144)
    at anywheresoftware.b4a.keywords.Common.CallSub4(Common.java:1082)
    at anywheresoftware.b4a.keywords.Common.CallSubNew2(Common.java:1037)
    at de.matle.backupmaster.cameraexclass._camera_preview(cameraexclass.java:194)
    at java.lang.reflect.Method.invoke(Native Method)
    at anywheresoftware.b4a.shell.Shell.runMethod(Shell.java:732)
    at anywheresoftware.b4a.shell.Shell.raiseEventImpl(Shell.java:351)
    at anywheresoftware.b4a.shell.Shell.raiseEvent(Shell.java:255)
    at java.lang.reflect.Method.invoke(Native Method)
    at anywheresoftware.b4a.ShellBA.raiseEvent2(ShellBA.java:144)
    at anywheresoftware.b4a.BA.raiseEvent(BA.java:193)
    at anywheresoftware.b4a.objects.CameraW$2$1$1.onPreviewFrame(CameraW.java:147)
    at android.hardware.Camera$EventHandler.handleMessage(Camera.java:1315)
    at android.os.Handler.dispatchMessage(Handler.java:107)
    at android.os.Looper.loop(Looper.java:224)
    at android.app.ActivityThread.main(ActivityThread.java:7560)
    at java.lang.reflect.Method.invoke(Native Method)
    at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:539)
    at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:950)
Caused by: java.lang.IllegalArgumentException: only support ImageFormat.NV21 and ImageFormat.YUY2 for now
    at android.graphics.YuvImage.<init>(YuvImage.java:82)
    at com.google.android.gms.vision.text.TextRecognizer.detect(com.google.android.gms:play-services-vision@@20.1.3:22)
    ... 26 more

 

[JordiCP]

Just in case, have you tried with 17 (NV21) instead of 842094169 (YV12)?
https://developer.android.com/reference/android/graphics/ImageFormat.html

 

[Erel]

You can also use the CameraEx method that converts the preview data to a bitmap and call setBitmap instead of setImageData.