B4A Library OpenCV 3.x

OpenCV (Open Source Computer Vision Library) is a really huge project/framework actively developed, mainly written in C++ . It is released under a BSD license.
Read more here: http://opencv.org/
OpenCV versions: https://opencv.org/releases/


OpenCV library for B4A: wraps the official OpenCV 3.x release for Android (in fact not all, about 95% of it)
  • Feel free to test and use it. You can even
  • License: You can use it for your projects, but you are not allowed to distribute nor sell this library. Of course you can distribute apps that use it (remember that OpenCV itself has BSD license as stated before)
  • Supported hardware is: armeabi-v7a and arm64-v8
  • Versions
    • 1.04 (2020/05/17)
      • This version wraps OpenCV 3.4.1 Android release and fixes some bugs (mostly some instance methods and some not exposed classes) of the previous version.
      • One of the major additions is the DNNmodule and related classes.
      • Link to the B4A library files: HERE :)
    • 1.00 (2017/09/27)
      • First B4A library wrapper (OpenCv320forB4A V1.00) which replicates (95%) the official OpenCV 3.20 Java API for Android.
      • (removed link. Use 1.04)

  • Please note that my support will be limited to issues with the wrapper itself, not to help translating OpenCV code from other languages to B4A 🤷‍♂️


========================================================================

(There may be some inaccuracies in what I expose in this post, related to the OpenCV project, since I am relatively new to it and don't know all about its internals. If you find any, please let me know and I will correct it)

A bit of explanation
There exist 'official' OpenCV wrappers for different languages and platforms. Android is one of them.
The official OpenCV 3.20 for Android API includes a lot of classes, organized in modules. But it does not include "all" the original OpenCV modules (since there are other 'experimental', non-free, or platform specific modules which may be present in other platforms but not for Android). Also, there are build options to "tune" it...

I have played quite a lot with it this last year, with a huge project which I started with inline Java, and also translating examples or testing features. But what I have used is just a small percent of the classes and methods exposed. So, there may be some (let's hope not too many) things to fix.

How to learn OpenCVforB4A
If you have worked before with OpenCV, the learning curve will be easy.
If it was using Java with OpenCV for Android, then it will be immediate, since all the methods have exactly the same syntax (except for initializers, polimorphism, and some special cases where simply I did my best).
Anyhow, the ways that I can think of, are (will add links later, also suggestions as online tutorials,.. are welcome)
  • Attached examples.
  • B4A OpenCV Tutorials. I will write a couple of them with what I consider the most important building pieces (for instance the Mat class, which in B4A is OCVMat) and modules
  • Internet examples: there are A LOT of examples over there, written in C++, Java, Python, JavaCV. I would look for examples in the language that is easier to understand for you and then try to translate. Some tips about it (based on my experience)
  • OpenCV syntax has changed as new versions. So there is an 'old' syntax in which nearly everything started with "cv...". Since version 3.X, there was 'cleaner' organization (project was written in C++ instead of C), and there were major syntax changes.
  • JavaCV: Translating from JavaCV to OpenCV should be quite easy but not always direct. JavaCV uses a mix of the old OpenCV syntax with some of its own, and at the beginning it can be a bit confusing, but then it is also easy.
  • Python: there is a lot of material...

First steps. Prepare for some crashes...
  • In OpenCV nearly everything takes part in the native code.
  • When we call a Sub/method/algorithm, it performs some internal checks to see if all the input data is correct. This check is perfomed in the native side. If something is not correct (wrong OCVMat dimensions, some incoherent parameters,...) it throws an exception and crashes. If we are lucky, perhaps we see in the log some clues about the check that made it crash.
  • On the good side, it is very easy to achieve results with OpenCV (check the examples). The real difficult part, as with many other things, is to fine-tune it: OpenCV has a collection of really powerful 'primitive' objects and operations, and really complex algorithms that can do many things. But it is the user who has to glue all of them to achieve the desired results.


(from the previous Beta announcement)
  • IMPORTANT: you must take this into account:
  • OpenCV (the included binary modules) is a free(*) project, but subject to license terms as described here: http://opencv.org/
    • (*): There are some modules in the OpenCV project which are on-free, but here I am refering to the ones included in the library
  • My work: (the B4A library) is free to test and use, but you can for it :). I'll keep donators updated with "advanced" material and examples
  • If you are interested, please PM me with your mail address and I will send you a link with the library and some basic examples. (be patient if you don't receive it immediately, I'll do it as soon as possible).
  • There is no documentation. In short, the syntax is nearly-exactly the same as the OpenCV3.20 Java API, adding "OCV" prefix and only the minimum modifications to adapt to B4A, For reference (taking into account described syntax changes) you can look at http://docs.opencv.org/java/3.1.0/ (which is not the latest one, but the API is nearly the same).
  • It would be preferable if you have worked before with OpenCV and/or can translate examples from Java/C++ and/or simply are interested in it.
  • I recommend starting with the examples and try to understand what is done. Just experimenting can lead to crash after crash of the native libraries with nearly no useful information, and can be very discouraging.
  • I forgot, the included binaries are for ameabi-v7a and arm64-v8 devices
---------------------------------------------------------------------
Some screenshots taken from the examples
Canny operator - Features2D - Color space conversion
s1.png
2D-FFT
s2.png
Color Blob detection
s3.png
 

Attachments

  • JavaCameraView2.zip
    2.7 KB · Views: 1,679
  • CameraOpenCvTest7.zip
    8.9 KB · Views: 1,162
  • BlobDetector5.zip
    15.3 KB · Views: 1,090
  • FaceDetector8.zip
    21 KB · Views: 1,149
Last edited:

cesarcm

Member
Licensed User
Longtime User
Hi JordiCP,

How are you doing?

Please, one more question... yep! I am late :(


Private Sub List2Array3D(lst As List) As Object(,)
Dim rgb(3) As Double
Dim arr(lst.Size, 3) As Double

For i=0 To (lst.Size - 1)
rgb = lst.Get(i)
arr(i, 0) = rgb(0) : arr(i, 1) = rgb(1) : arr(i, 2) = rgb(2)
Next

return arr
End Sub


Dim mySignal As List
'PS: each list row contains an array regarding RGB colors

Dim outSignal As OCVMat
outSignal.Initialize2(mySignal.Size, 1, mType.CV_32FC3)

Dim arr(mySignal, 3) As Object = List2Array3D(mySignal)

outSignal.put(0, 0, arr) ???


'PS: how to put the array into OCVMat with PUT method???


Any idea?
Thanks.
Regards,
Cesar
 

cesarcm

Member
Licensed User
Longtime User
JCP and Erel,

Running your JordiCP's OpenCV code at "old" Activity model I was getting ~12fps (Motorola One Vision) ...

Now, using B4Xpages I am getting ~6-7fps :(

Any idea? Probably it is related to Erel's implementations of B4XPages ... or not?

Thanks.
Regards,
Cesar
 

cesarcm

Member
Licensed User
Longtime User
Erel,

Thanks but something is different ...

I have sent to JordiCP:


JCP,

I have tried -- using B4XPages -- both commands below:

mOpenCvCameraView.connectCamera2(pnlView, True) ' NEW

'mOpenCvCameraView.connectCamera(pnlView.Width,pnlView.Height)


No processing at all .. just to capture video frames => 9-10fps !!! (of course, after apply the algorithms it decreases to 5-6fps)

It means that using your original OpenCV library or the new one the FPS are the same ...

I will try using the same algorithm at the old Activuty project ...

Thanks.
Regards,
Cesar
 

cesarcm

Member
Licensed User
Longtime User
Erel,

I just have tested here again! Results below:

JCP,

Using your lib in "old standard" project: ACTIVITY

1) just grabbing video frame: 22-23fps

2) applying face detection: 10-11fps

3) full algorithm (from our project): 9-10fps

Weird ... it means that your LIB + our algorithms under B4XPages are 50% slower than the "old" Activity model ...

Thanks.
Regards,

Cesar
 

cesarcm

Member
Licensed User
Longtime User
Erel and JordiCP,

Running in RELEASE MODE we get 25fps !!! :D:p:);):cool:

Why so difference to DEBUG mode??? and why in debug mode there are almost 50% differences between Activity vs B4XPages projects ?

Thanks for all ;)
Kind regards,
Cesar
 

JordiCP

Expert
Licensed User
Longtime User
JavaCameraView uses a different thread in release mode, and the same UI thread in debug mode. I needed to do it this way to make it work in debug mode.

Besides, debug mode is optimized for a faster development/deployment cycle, where part of the code can be executed in the device and another part in the pc. That's why performance should ALWAYS be measured in release mode.

So, if everything is fast enough in release mode, you got it :)
 

Erel

B4X founder
Staff member
Licensed User
Longtime User
and why in debug mode there are almost 50% differences between Activity vs B4XPages projects ?
It has nothing to do with B4XPages.
As I've already wrote it two times, I will write again:
It has nothing to do with B4XPages.

Clean the project, remove breakpoints and it will run in the same speed. The debugger is very complex and has several execution pipelines that depend on all kinds of parameters.
Bottom line is to never test performance in debug mode.
 

cesarcm

Member
Licensed User
Longtime User
Erel,

Thanks a lot! YOU ROCK!

Please, accept my apologizes since you had to write 3x about B4XPages.

I just would like to understand what is the performance differences between Activity models under debug mode... It was making non sense when I was testing. Sorry about that.

Kind regards,
Cesar
 

PodSoft2016

Member
JavaCameraView uses a different thread in release mode, and the same UI thread in debug mode. I needed to do it this way to make it work in debug mode.

Besides, debug mode is optimized for a faster development/deployment cycle, where part of the code can be executed in the device and another part in the pc. That's why performance should ALWAYS be measured in release mode.

So, if everything is fast enough in release mode, you got it :)
Hello, i'm sorry sir for my question. can you help me about detecting face parts such as mouth, nose or ear in this library?
Thanks alot .
 

Hypnos

Active Member
Licensed User
Longtime User
Hello JordiCP,

I found one of your video in youtube and seems that you can use OpenCV in B4I, may I know whether your have the library for B4i ?
I want to use OpenCV on my IOS project and it is great if you can make this library for B4I.

Thanks!

 

DonManfred

Expert
Licensed User
Longtime User
may I know whether your have the library for B4i ?
as B4i is not using java (like B4A and B4J) it needs a wrapper in another language than java. Don´t know if JodiCP have written one.
I could not find a B4i Library with the forumsearch.
 

JordiCP

Expert
Licensed User
Longtime User
Hello JordiCP,

I found one of your video in youtube and seems that you can use OpenCV in B4I, may I know whether your have the library for B4i ?
I want to use OpenCV on my IOS project and it is great if you can make this library for B4I.

Thanks!

There's not Opencv for B4i yet.
The video was using B4i plus a small library made for that specific purpose, and all the Opencv processing work was made inside that lib.
 

biometrics

Active Member
Licensed User
Longtime User
Hi @JordiCP,

readNetFromCaffe is working fine for me with Caffe models but readNetFromTensorflow is failing to load Tensorflow models. I've read many Pyhton and C++ examples/tutorials online that use them and there are no mention of issues with it. Would you mind loading the attached models and see if there is an obvious error that can be fixed. Else I'll have to live with the Caffe models. Was hoping to use the Tensorflow models to speed up processing.

This works fine (Caffe model):
B4X:
libOpenCvNet = libOpenCvDnn.readNetFromCaffe(File.Combine(File.DirInternal, "deploy.prototxt"), File.Combine(File.DirInternal, "res10_300x300_ssd_iter_140000_fp16.caffemodel"))

But this fails (Tensorflow model):
B4X:
libOpenCvNet = libOpenCvDnn.readNetFromTensorflow(File.Combine(File.DirInternal, "opencv_face_detector.pbtxt"), File.Combine(File.DirInternal, "opencv_face_detector_uint8.pb"))

with this error:
B4X:
CvException [org.opencv.core.CvException: cv::Exception: OpenCV(3.4.1) /build/master_pack-android/opencv/modules/dnn/src/tensorflow/tf_io.cpp:42: error: (-2) FAILED: ReadProtoFromBinaryFile(param_file, param). Failed to parse GraphDef file: /data/user/0/isenzo.audiencemeasurementservice/files/opencv_face_detector.pbtxt in function void cv::dnn::ReadTFNetParamsFromBinaryFileOrDie(const char*, tensorflow::GraphDef*)

Edit: seems a 1.5MB attachment is too large. You can get those two files from here:

The article: https://learnopencv.com/face-detection-opencv-dlib-and-deep-learning-c-python/
The model files (note to view in Raw or else you get HTML when downloading the small file): https://github.com/spmallick/learnopencv/tree/master/FaceDetectionComparison/models

I have downloaded the Tensorflow models from several other tutorials thinking that perhaps this set is corrupt but they all do this.

Happy to send you a donation for your effort.
 

JordiCP

Expert
Licensed User
Longtime User
Can't help you much (now I don't have time to test it).

DNN adoption by OpenCV was 'quite recent' in version 3.4.1, and it is possible that some things didn't work properly. In fact not all combinations did work for me when I tested it: sometimes a module only worked when initted with the description file and others without it (if I remember correctly). If you read the Release Notes, starting with version 3.4.1 and then scroll upwards to see what was changed in the next versions, you'll see that there are lots of references to DNN improvements and bug fixes.

Anyhow, you could try:
  • Just in case there is a difference somewhere, test with these files (they are the exact ones they mention in the release notes): network and description.
  • Also, try to load the network using the 2nd syntax (i.e., without the description file): mDnn.readNetFromTensorflow1(model As String) As OCVNet

If none of them work for you I'll try to check it during the weekend. But if it does not work, I fear that there will not be much to do, and a higher OpenCV version will be needed.
 
Last edited:

biometrics

Active Member
Licensed User
Longtime User
Can't help you much (now I don't have time to test it).

DNN adoption by OpenCV was 'quite recent' in version 3.4.1, and it is possible that some things didn't work properly. In fact not all combinations did work for me when I tested it: sometimes a module only worked when initted with the description file and others without it (if I remember correctly). If you read the Release Notes, starting with version 3.4.1 and then scroll upwards to see what was changed in the next versions, you'll see that there are lots of references to DNN improvements and bug fixes.

Anyhow, you could try:
  • Just in case there is a difference somewhere, test with these files (they are the exact ones they mention in the release notes): network and description.
  • Also, try to load the network using the 2nd syntax (i.e., without the description file): mDnn.readNetFromTensorflow1(model As String) As OCVNet

If none of them work for you I'll try to check it during the weekend. But if it does not work, I fear that there will not be much to do, and a higher OpenCV version will be needed.
No luck, same problem. Thanks for explaining though, didn't realise this is still buggy stuff (tensorflow).

I did some benchmarks on only the face extraction for Caffe vs Haar Cascade on a 640x388 image with 15 faces and Haar Cascade was about 150ms and Caffe about 1000ms. This is on a Rockchip RK3328 with 2 GB RAM and Android 7.1 With Caffe I still have to loop through the array of detections and compare the confidence level which also takes time, I don't see a way to pass in the confidence level to reduce the array size (which is typical from 500-1,000).

So though Caffe is better at face detection for faces that are sideways, up/down and occluded, Haar Cascade still works well for frontal faces and is faster.

So I think I'll combine Haar Cascade for face detection and use Caffe for gender and age detection. I still have to match and track faces as they move too. So I need as much speed as I can get.

Would still like to benchmark the tensorflow model though. The two models are described as follows and I'm thinking that floating point would be slower...
  1. Floating point 16 version of the original caffe implementation ( 5.4 MB )
  2. 8 bit quantized version using Tensorflow ( 2.7 MB )
Another thing I noticed is that Caffe requires RGB (3 channel) images and Haar Cascade can work on B&W (1 channel). I guess that will also influence the speed. Don't know what tensorflows requirement is but the example was RGB.
 

roberto64

Active Member
Licensed User
Longtime User
Hi, aito, I'm using CamEx2 with opencv3 I don't understand why I can't see the rectangle in the camera,
Greetings
Sub OpenCamera (front As Boolean)
'Camera Version: 1.30
rp.CheckAndRequest(rp.PERMISSION_CAMERA)
Wait For B4XPage_PermissionResult (Permission As String, Result As Boolean)
If Result = False Then
ToastMessageShow("No permission!", True)
Return
End If
Wait For (cam.OpenCamera(front)) Complete (TaskIndex As Int)
If TaskIndex > 0 Then
MyTaskIndex = TaskIndex 'hold this index. It will be required in later calls.
Wait For(PrepareSurface) Complete (Success As Boolean)
If Success = True Then
t.Enabled = True
End If
End If
Log("Start success: " & Success)
If Success = False Then
ToastMessageShow("Failed to open camera", True)
End If
End Sub


Sub t_tick
Dim mbm As Bitmap
Dim myMat,imgmat As OCVMat
mbm = cam.GetPreviewBitmap(480,640)
' Log(cam.PreviewSize.Height& " " & cam.PreviewSize.Width)
mbm.InitializeMutable(480,640)
mUtils.bitmapToMat(mbm,imgmat,True)
myMat = imgmat.clone
Dim cs As OCVScalar
cs.Initialize3(0,0,255)
Dim w As Int = myMat.width
Dim h As Int = myMat.height
Dim start As OCVPoint
start.Set(Array As Double(w * 1 / 3, h * 1 / 3))
Dim finish As OCVPoint
finish.Set(Array As Double(w * 2 / 3, h * 2 / 3 ))
mImgProc.rectangle1(myMat,start, finish,cs,5)
End Sub
 
Top