B4A Library Firebase ML-Vision (OnDevice Text-, Barcode-, Face-, Contour-Recognition and Imagelabeling)

DonManfred

Expert
Licensed User
This is a wrap for the Firebase ML-Vision live detection of
- Text
- Barcodes
- Faces
- Facontour
- and Imagelabeling.

New in V0.8
- Support for Cloudregocnition

FirebaseML
Author:
DonManfred
Version: 0.8
  • CameraSourcePreview
    • Functions:
      • BringToFront
      • DesignerCreateView (base As Panel, lw As Label, props As Map)
      • Initialize (EventName As String)
      • Invalidate
      • Invalidate2 (arg0 As android.graphics.Rect)
      • Invalidate3 (arg0 As Int, arg1 As Int, arg2 As Int, arg3 As Int)
      • IsInitialized As Boolean
      • release
      • RemoveView
      • RequestFocus As Boolean
      • SendToBack
      • SetBackgroundImage (arg0 As android.graphics.Bitmap) As BitmapDrawable
      • SetColorAnimated (arg0 As Int, arg1 As Int, arg2 As Int)
      • SetLayout (arg0 As Int, arg1 As Int, arg2 As Int, arg3 As Int)
      • SetLayoutAnimated (arg0 As Int, arg1 As Int, arg2 As Int, arg3 As Int, arg4 As Int)
      • SetVisibleAnimated (arg0 As Int, arg1 As Boolean)
      • start (cameraSource As com.google.firebase.samples.mlkit.common.CameraSource)
      • start2 (cameraSource As com.google.firebase.samples.mlkit.common.CameraSource, overlay As com.google.firebase.samples.mlkit.common.GraphicOverlay)
      • stop
    • Properties:
      • Background As android.graphics.drawable.Drawable
      • Color As Int [write only]
      • Enabled As Boolean
      • Height As Int
      • Left As Int
      • Padding As Int()
      • Parent As Object [read only]
      • Tag As Object
      • Top As Int
      • Visible As Boolean
      • Width As Int
  • FirebaseVisionImage
    • Events:
      • BarcodeDetected (success As Boolean, Barcodes As List)
      • FaceContourDetected (success As Boolean, contours As List)
      • FaceDetected (success As Boolean, faces As List)
      • ImageLabeled (success As Boolean, labels As List)
      • TextDetected (success As Boolean, textlist As List)
    • Fields:
      • BARCODE_DETECTION As String
      • CLASSIFICATION_FLOAT As String
      • CLASSIFICATION_QUANT As String
      • FACE_CONTOUR As String
      • FACE_DETECTION As String
      • IMAGE_LABEL_DETECTION As String
      • TEXT_DETECTION As String
    • Functions:
      • CreateMetadata (width As Int, height As Int, format As Int, rotation As Int) As com.google.firebase.ml.vision.common.FirebaseVisionImageMetadata
        480x360 is typically sufficient for image recognition
      • Initialize (EventName As String, camsrcpreview As com.google.firebase.samples.mlkit.common.CameraSourcePreview, overlay As com.google.firebase.samples.mlkit.common.GraphicOverlay, overlayBitmappathName As String, selectedModel As String)
      • onPause
      • onResume
      • start
      • stop
      • switchRecognizer (recognizer As String)
        One of FACE_DETECTION, FACE_CONTOUR, TEXT_DETECTION,
        BARCODE_DETECTION or IMAGE_LABEL_DETECTION
    • Properties:
      • CameraFacing As Int [read only]
      • Facing As Int [write only]
      • Img As com.google.firebase.ml.vision.common.FirebaseVisionImage [read only]
      • ImgFormatNV21 As Int [read only]
      • ImgFormatYV12 As Int [read only]
  • GraphicOverlay
    • Functions:
      • add (graphic As com.google.firebase.samples.mlkit.common.GraphicOverlay.Graphic)
        Adds a graphic to the overlay.
      • BringToFront
      • clear
        Removes all graphics from the overlay.
      • DesignerCreateView (base As Panel, lw As Label, props As Map)
      • Initialize (EventName As String)
      • Invalidate
      • Invalidate2 (arg0 As android.graphics.Rect)
      • Invalidate3 (arg0 As Int, arg1 As Int, arg2 As Int, arg3 As Int)
      • IsInitialized As Boolean
      • remove (graphic As com.google.firebase.samples.mlkit.common.GraphicOverlay.Graphic)
        Removes a graphic from the overlay.
      • RemoveView
      • RequestFocus As Boolean
      • SendToBack
      • SetBackgroundImage (arg0 As android.graphics.Bitmap) As BitmapDrawable
      • setCameraInfo (previewWidth As Int, previewHeight As Int, facing As Int)
        Sets the camera attributes for size and facing direction, which informs how to transform image
        coordinates later.
      • SetColorAnimated (arg0 As Int, arg1 As Int, arg2 As Int)
      • SetLayout (arg0 As Int, arg1 As Int, arg2 As Int, arg3 As Int)
      • SetLayoutAnimated (arg0 As Int, arg1 As Int, arg2 As Int, arg3 As Int, arg4 As Int)
      • SetVisibleAnimated (arg0 As Int, arg1 As Boolean)
    • Properties:
      • Background As android.graphics.drawable.Drawable
      • Color As Int [write only]
      • Enabled As Boolean
      • Height As Int
      • Left As Int
      • Padding As Int()
      • Parent As Object [read only]
      • Tag As Object
      • Top As Int
      • Visible As Boolean
      • Width As Int


Setup:
- Start the B4A SDKManager and update all recommended items. The Manager should download all Firebase-ML-Items...
- Follow the Firebaseintegration tutorial for the needed Manifestentries.
- Add this lines to your Mainmodule
B4X:
#AdditionalJar: com.google.android.gms:play-services-base
#AdditionalJar: com.google.android.gms:play-services-tasks
#AdditionalJar: com.google.firebase:firebase-ml-vision
#AdditionalJar: com.google.firebase:firebase-ml-vision-face-model
#AdditionalJar: com.google.firebase:firebase-ml-vision-image-label-model
#AdditionalJar: com.google.firebase:firebase-ml-common
#AdditionalJar: com.google.firebase:firebase-ml-model-interpreter
- Create a layout with a CameraSourcePreview and a GraphicOverlay

Example code
B4X:
#AdditionalJar: com.google.android.gms:play-services-base
#AdditionalJar: com.google.android.gms:play-services-tasks
#AdditionalJar: com.google.firebase:firebase-ml-vision
#AdditionalJar: com.google.firebase:firebase-ml-vision-face-model
#AdditionalJar: com.google.firebase:firebase-ml-vision-image-label-model
#AdditionalJar: com.google.firebase:firebase-ml-common
#AdditionalJar: com.google.firebase:firebase-ml-model-interpreter


Sub Process_Globals
    'These global variables will be declared once when the application starts.
    'These variables can be accessed from all modules.
    Private auth As FirebaseAuth
End Sub

Sub Globals
    'These global variables will be redeclared each time the activity is created.
    'These variables can only be accessed from this module.
    Dim detector As FirebaseVisionImage
    Private CameraSourcePreview1 As CameraSourcePreview
    Private GraphicOverlay1 As GraphicOverlay
    Private Button1 As Button
    Private btnStop As Button
    Private btnStart As Button
    Private spRegognizers As Spinner
    Private btnFacing As Button
    Private camFacing As Int = 0
End Sub

Sub Activity_Create(FirstTime As Boolean)
    'Do not forget to load the layout file created with the visual designer. For example:
    If FirstTime Then
        auth.Initialize("auth")
    End If
    Starter.rp.CheckAndRequest(Starter.rp.PERMISSION_CAMERA)
    Wait For Activity_PermissionResult (Permission As String, Result As Boolean)
    If Result = False Then
        ToastMessageShow("No permission for Camera!", True)
        'Return
    End If
    Starter.rp.CheckAndRequest(Starter.rp.PERMISSION_WRITE_EXTERNAL_STORAGE)
    Wait For Activity_PermissionResult (Permission As String, Result As Boolean)
    If Result = False Then
        ToastMessageShow("No permission for External Storage!", True)
        'Return
    End If

    Activity.LoadLayout("Layout1")
    spRegognizers.Clear
    spRegognizers.Add(detector.BARCODE_DETECTION)
    spRegognizers.Add(detector.TEXT_DETECTION)
    spRegognizers.Add(detector.IMAGE_LABEL_DETECTION)
    spRegognizers.Add(detector.FACE_DETECTION)
    spRegognizers.Add(detector.FACE_CONTOUR)
    detector.Initialize("MLVision",CameraSourcePreview1,GraphicOverlay1,File.Combine(File.DirAssets,"clown_nose.png"),detector.FACE_DETECTION)
    detector.Facing = 0

    If auth.CurrentUser.IsInitialized Then Auth_SignedIn(auth.CurrentUser)
End Sub
Sub MLVision_TextDetected(success As Boolean, texts As List)
    If texts.IsInitialized And texts.Size > 0 Then
        Log($"MLVision_TextDetected(${success}, ${texts.Size})"$)
        For i=0 To texts.Size-1
            Log(texts.Get(i))
        Next
    End If

End Sub
Sub MLVision_ImageLabeled(success As Boolean, labels As List)
    If success And labels.IsInitialized And labels.Size > 0 Then
        Log($"MLVision_ImageLabeled(${success}, ${labels.Size})"$)
        For i=0 To labels.Size-1
            Log(labels.Get(i))
        Next
    End If
End Sub
Sub MLVision_FaceDetected(success As Boolean, faces As List)
    If faces.IsInitialized And faces.Size > 0 Then
        Log($"MLVision_FaceDetected(${success}, ${faces.Size})"$)
        For i=0 To faces.Size-1
            Log(faces.Get(i))
        Next
    End If

End Sub
Sub MLVision_FaceContourDetected(success As Boolean, contours As List)
    If contours.IsInitialized And contours.Size > 0 Then
        Log($"MLVision_FaceContourDetected(${success}, ${contours.Size})"$)
        For i=0 To contours.Size-1
            'Log(contours.Get(i))
        Next
    End If
End Sub
Sub MLVision_BarcodeDetected(success As Boolean, Barcodes As List)
    If success And Barcodes.IsInitialized And Barcodes.Size > 0 Then
        Log($"MLVision_BarcodeDetected(${success}, ${Barcodes.Size})"$)
        For i=0 To Barcodes.Size-1
            Log(Barcodes.Get(i))
        Next
    End If
End Sub
Sub MLVision_DetectText(success As Boolean, info As String)
    Log($"MLVision_DetectText(${success}, ${info})"$)
End Sub
Sub ReadFile(Dir As String, FileName As String) As Byte()
    Dim out As OutputStream
    out.InitializeToBytesArray(100) 'size not really important
    File.Copy2(File.OpenInput(Dir, FileName), out)
    Return out.ToBytesArray
End Sub

Sub Auth_SignedIn (User As FirebaseUser)
    Log("SignedIn: " & User.DisplayName)
End Sub
Sub Activity_Resume
    detector.onResume

End Sub

Sub Activity_Pause (UserClosed As Boolean)
    detector.onPause
End Sub


Sub Button1_Click
    'Dim bytes() As Byte = ReadFile(File.DirAssets, "ilovecats.png")
End Sub

Sub btnStop_Click
    detector.stop
End Sub

Sub btnStart_Click
    detector.start
End Sub

Sub spRegognizers_ItemClick(Position As Int, Value As Object)
    Log($"spRegognizers_ItemClick(${Position}, ${Value})"$)
    detector.stop
    detector.switchRecognizer(Value)
    detector.start
End Sub

Sub btnFacing_Click
    detector.stop
    If camFacing = 0 Then
        detector.Facing = 1
        camFacing = 1
    Else
        detector.Facing = 0
        camFacing = 0
    End If
    detector.start
Known issues:
- I was not able to setup a detector for detecting an Image as of now. For now only the live detection is included.

Notes:
- The Included methods are ALL using "OnDevice-Recognition". For the ondevice-recognition a Free Firebase account should be OK.
- I can implement also the Cloud-Methods i guess.But you need to have a PAID plan to use them. As of now no Cloudmethods are implemented/supported.

Please create a new Thread in the Questionsforum for any Issue/Question you have.
 

Attachments

Last edited:

peacemaker

Expert
Licensed User
It is not an Issue/Question: i would like to request Cloud-Methods also. I hope my customer will agree for the project soon, where recognition of non-latin text is required, and donation doze for the wrapping will be available.
 

ppgirl

Member
Licensed User
Hello DonManfred,

Could you get me a compiled APK file . I build the project , but it will crash in internal when start (Android 7 or Android 9 mobile).


Logger connected to: PM1LHMA780202845
--------- beginning of crash
--------- beginning of system
--------- beginning of main
Copying updated assets files (2)
*** Service (starter) Create ***
** Service (starter) Start **
** Activity (main) Create, isFirst = true **
** Activity (main) Resume **
onAuthStateChanged: com.google.firebase.auth.internal.zzk@1b233a4
** Activity (main) Pause, UserClosed = false **
sending message to waiting queue (activity_permissionresult)
running waiting messages (1)
Using Barcode Detector Processor
Error occurred on line: 75 (Main)
java.lang.NullPointerException: Attempt to invoke virtual method 'java.lang.Object com.google.firebase.ml.common.internal.zzd.get(java.lang.Object)' on a null object reference
at com.google.firebase.ml.common.internal.zzm.zza(com.google.firebase:firebase-ml-common@@20.0.1:24)
at com.google.android.gms.internal.firebase_ml.zzqb.<init>(com.google.firebase:firebase-ml-vision@@22.0.0:7)
at com.google.firebase.ml.vision.barcode.FirebaseVisionBarcodeDetector.<init>(com.google.firebase:firebase-ml-vision@@22.0.0:16)
at com.google.firebase.ml.vision.barcode.FirebaseVisionBarcodeDetector.zza(com.google.firebase:firebase-ml-vision@@22.0.0:13)
at com.google.firebase.ml.vision.FirebaseVision.getVisionBarcodeDetector(com.google.firebase:firebase-ml-vision@@22.0.0:23)
at com.google.firebase.samples.mlkit.barcodescanning.BarcodeScanningProcessor.<init>(BarcodeScanningProcessor.java:52)
at de.donmanfred.FirebaseDetectorwrapper.createCameraSource(FirebaseDetectorwrapper.java:479)
at de.donmanfred.FirebaseDetectorwrapper.Initialize(FirebaseDetectorwrapper.java:93)
at com.b4a.testml.main$ResumableSub_Activity_Create.resume(main.java:537)
at anywheresoftware.b4a.shell.DebugResumableSub$DelegatableResumableSub.resumeAsUserSub(DebugResumableSub.java:48)
at java.lang.reflect.Method.invoke(Native Method)
at anywheresoftware.b4a.shell.Shell.runMethod(Shell.java:732)
at anywheresoftware.b4a.shell.Shell.raiseEventImpl(Shell.java:351)
at anywheresoftware.b4a.shell.Shell.raiseEvent(Shell.java:255)
at java.lang.reflect.Method.invoke(Native Method)
at anywheresoftware.b4a.ShellBA.raiseEvent2(ShellBA.java:144)
at anywheresoftware.b4a.BA.raiseEvent(BA.java:176)
at anywheresoftware.b4a.shell.DebugResumableSub$DelegatableResumableSub.resume(DebugResumableSub.java:43)
at anywheresoftware.b4a.BA.checkAndRunWaitForEvent(BA.java:250)
at anywheresoftware.b4a.ShellBA.raiseEvent2(ShellBA.java:137)
at anywheresoftware.b4a.BA$2.run(BA.java:370)
at anywheresoftware.b4a.BA.setActivityPaused(BA.java:442)
at com.b4a.testml.main$ResumeMessage.run(main.java:306)
at android.os.Handler.handleCallback(Handler.java:873)
at android.os.Handler.dispatchMessage(Handler.java:99)
at android.os.Looper.loop(Looper.java:193)
at android.app.ActivityThread.main(ActivityThread.java:6669)
at java.lang.reflect.Method.invoke(Native Method)
at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:493)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:858)
** Activity (main) Resume **
 

Attachments

Last edited:

roberto64

Active Member
Licensed User
Hi, to me this mistake and in the SDK it is all installed.
regards
B4X:
Compiling code.    Error
Error parsing manifest script:
Error parsing google-services.json:
Il file 'C:\Users\apple\Desktop\MLEx\google-services.json' non è stato trovato.
 

DonManfred

Expert
Licensed User

roberto64

Active Member
Licensed User
Hi, I send in execution but it gives me this error

B4X:
Maven artifact not found: com.google.android.gms/play-services-vision-image-label
 

maddy

Member
Licensed User
Hey manfred it popping up a bunch of error when i execute the following code my b4a version is 8.5 AND I HAD UPDATED ALL FIREBASE LIBRARRIES AS PER INSTRUCTIONS..
</CODE>
G:\FirebaseMLVisionEx\MLEx\Objects\bin\extra\res2\res\values\values.xml:84: error: Attribute "layout_dodgeInsetEdges" already defined with incompatible format.
G:\FirebaseMLVisionEx\MLEx\Objects\bin\extra\res8\res\values\values.xml:86: Original attribute defined here.
G:\FirebaseMLVisionEx\MLEx\Objects\bin\extra\res1\res\values\values.xml:36: error: Attribute "fontProviderFetchStrategy" already defined with incompatible format.
G:\FirebaseMLVisionEx\MLEx\Objects\bin\extra\res6\res\values\values.xml:42: Original attribute defined here.
G:\FirebaseMLVisionEx\MLEx\Objects\bin\extra\res1\res\values\values.xml:51: error: Attribute "fontProviderFetchTimeout" already defined with incompatible format.
G:\FirebaseMLVisionEx\MLEx\Objects\bin\extra\res6\res\values\values.xml:54: Original attribute defined here.
G:\FirebaseMLVisionEx\MLEx\Objects\bin\extra\res1\res\values\values.xml:61: error: Attribute "fontStyle" already defined with incompatible format.
G:\FirebaseMLVisionEx\MLEx\Objects\bin\extra\res6\res\values\values.xml:62:
</CODE>
 

peacemaker

Expert
Licensed User
@GanjaKyp, create a new topic in question forum and post there the ready to start test project, to be tried.
 

DonManfred

Expert
Licensed User
B4X:
AddApplicationText(
<meta-data
  android:name="com.google.firebase.ml.vision.DEPENDENCIES"
  android:value="barcode,,qrcode,,face,,ocr,,ica,,label" />
)
It will NOT work with the newest maven artifacts as they depend on firebase-installation-api which is - not yet - available in B4A.
 
Top