B4A Library NV21utils Lib: camera preview to ARGB888

Hi all!

I am happy to release my first lib, which converts from default camera preview format to ARGB888. The processing method has been written in C and included in the java library (which does not anything more)

The library is included in the attached zip, with a modified version of the cameraEx class example. So, Erel's camera library will be needed to run the example.


What it does:

  • Takes the preview byte array and converts i to ARGB888, rotating and flipping if necessary. putting the result into a mutablebitmap previously generated by the user. So, no need to create and destroy bitmaps each time. (Important: the mutableBitmap must have been created with the proper final orientation and properly scaled dimensions). Also, the mutablebitmap is supposed to be ARGB888 format. If these requirements are not met, the routine will return and do nothing.
  • User can specify the subsampling (1,2,4 or 8, only these values). Depending on your needs, not always full resolution is needed and process time can be decreased this way.
  • It has been compiled for arm devices. Next release I will try to add other plattforms
But:
  • much faster options may exist over there. In case anybody knows of other simple approaches, please let me know, since I am also interested in it.

I don't have much time, but if someone finds it useful for processing, I will add a function which makes the conversion directly onto an array instead of a bitmap.


Please let me know if you decide to use it or find any issues with it.:)
 

Attachments

  • Camera_NV21_example_and_lib.zip
    29.9 KB · Views: 601
Click to expand...
S

sorex

Expert
Licensed User
Longtime User
just had a look at it (the running app that is).

these filters are quite speedy, what's the trick?
 
JordiCP

JordiCP

Expert
Licensed User
Longtime User
The whole project is based on Erel's cameraEx example, so all the stuff in the big panel and while-text-buttons is done by the same camera

The "demo" add-on is the small panel and the yellow-text-buttons. To do this I take advantage of subsampling (when scale>1) and in this case the "negate" effect is achieved negating lightness (replacing Y --> 255-Y ,which is not totally exact but does the trick) . The algorithm is "optimized" (up to where I can) and the way to do the conversion taken from here and other posts
 
wimpie3

wimpie3

Well-Known Member
Licensed User
Longtime User
This is a full crash log dump:

B4X: 
GC_CONCURRENT freed 4K, 6% free 41722K/44112K, paused 3ms+4ms, total 24ms
JNI WARNING: threadid=1 using JNI after critical get (GetIntField)
            in Ljcp/b4a/nv21util/NV21util;.proceed:([BIIILandroid/graphics/Bitmap;ZZII)V (GetIntField)
"main" prio=5 tid=1 NATIVE
  | group="main" sCount=0 dsCount=0 obj=0x415a7ca8 self=0x414e1408
  | sysTid=6956 nice=0 sched=0/0 cgrp=apps handle=1074393428
  | state=R schedstat=( 1408359102 350896791 4172 ) utm=113 stm=27 core=1
  #00  pc 0000132e  /system/lib/libcorkscrew.so (unwind_backtrace_thread+29)
  #01  pc 00060652  /system/lib/libdvm.so (dvmDumpNativeStack(DebugOutputTarget const*, int)+33)
  #02  pc 00054640  /system/lib/libdvm.so (dvmDumpThreadEx(DebugOutputTarget const*, Thread*, bool)+395)
  #03  pc 000546ae  /system/lib/libdvm.so (dvmDumpThread(Thread*, bool)+25)
  #04  pc 00038804  /system/lib/libdvm.so
  #05  pc 0003e5ae  /system/lib/libdvm.so
  #06  pc 0004e73e  /system/lib/libandroid_runtime.so
  #07  pc 000005cc  /system/lib/libjnigraphics.so (AndroidBitmap_lockPixels+7)
  #08  pc 00001b74  /data/app-lib/test.android-1/libjcp_000.so (Java_jcp_b4a_nv21util_NV21util_proceed+972)
  #09  pc 0001dbcc  /system/lib/libdvm.so (dvmPlatformInvoke+112)
  #10  pc 0004e122  /system/lib/libdvm.so (dvmCallJNIMethod(unsigned int const*, JValue*, Method const*, Thread*)+397)
  #11  pc 0004fb0e  /system/lib/libdvm.so (dvmResolveNativeMethod(unsigned int const*, JValue*, Method const*, Thread*)+181)
  #12  pc 00026fe0  /system/lib/libdvm.so
  #13  pc 0002dfa0  /system/lib/libdvm.so (dvmMterpStd(Thread*)+76)
  #14  pc 0002b638  /system/lib/libdvm.so (dvmInterpret(Thread*, Method const*, JValue*)+184)
  #15  pc 00060864  /system/lib/libdvm.so (dvmInvokeMethod(Object*, Method const*, ArrayObject*, ArrayObject*, ClassObject*, bool)+391)
  #16  pc 000687c6  /system/lib/libdvm.so
  #17  pc 00026fe0  /system/lib/libdvm.so
  #18  pc 0002dfa0  /system/lib/libdvm.so (dvmMterpStd(Thread*)+76)
  #19  pc 0002b638  /system/lib/libdvm.so (dvmInterpret(Thread*, Method const*, JValue*)+184)
  #20  pc 00060864  /system/lib/libdvm.so (dvmInvokeMethod(Object*, Method const*, ArrayObject*, ArrayObject*, ClassObject*, bool)+391)
  #21  pc 000687c6  /system/lib/libdvm.so
  #22  pc 00026fe0  /system/lib/libdvm.so
  #23  pc 0002dfa0  /system/lib/libdvm.so (dvmMterpStd(Thread*)+76)
  #24  pc 0002b638  /system/lib/libdvm.so (dvmInterpret(Thread*, Method const*, JValue*)+184)
  #25  pc 00060864  /system/lib/libdvm.so (dvmInvokeMethod(Object*, Method const*, ArrayObject*, ArrayObject*, ClassObject*, bool)+391)
  #26  pc 000687c6  /system/lib/libdvm.so
  #27  pc 00026fe0  /system/lib/libdvm.so
  #28  pc 0002dfa0  /system/lib/libdvm.so (dvmMterpStd(Thread*)+76)
  #29  pc 0002b638  /system/lib/libdvm.so (dvmInterpret(Thread*, Method const*, JValue*)+184)
  #30  pc 00060580  /system/lib/libdvm.so (dvmCallMethodV(Thread*, Method const*, Object*, bool, JValue*, std::__va_list)+335)
  #31  pc 00049d0a  /system/lib/libdvm.so
  at jcp.b4a.nv21util.NV21util.proceed(Native Method)
  at test.android.menuclass._window_closed(menuclass.java:3394)
  at java.lang.reflect.Method.invokeNative(Native Method)
  at java.lang.reflect.Method.invoke(Method.java:515)
  at anywheresoftware.b4a.BA.raiseEvent2(BA.java:174)
  at anywheresoftware.b4a.keywords.Common.CallSub4(Common.java:853)
  at anywheresoftware.b4a.keywords.Common.CallSubNew2(Common.java:810)
  at test.android.dialogpanel._returnvalue(dialogpanel.java:611)
  at test.android.dialogpanel._btnok_click(dialogpanel.java:403)
  at java.lang.reflect.Method.invokeNative(Native Method)
  at java.lang.reflect.Method.invoke(Method.java:515)
  at anywheresoftware.b4a.BA.raiseEvent2(BA.java:174)
  at anywheresoftware.b4a.BA.raiseEvent(BA.java:158)
  at anywheresoftware.b4a.objects.ViewWrapper$1.onClick(ViewWrapper.java:66)
  at android.view.View.performClick(View.java:4438)
  at android.view.View$PerformClick.run(View.java:18422)
  at android.os.Handler.handleCallback(Handler.java:733)
  at android.os.Handler.dispatchMessage(Handler.java:95)
  at android.os.Looper.loop(Looper.java:136)
  at android.app.ActivityThread.main(ActivityThread.java:5017)
  at java.lang.reflect.Method.invokeNative(Native Method)
  at java.lang.reflect.Method.invoke(Method.java:515)
  at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:779)
  at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:595)
  at dalvik.system.NativeStart.main(Native Method)

Critical "GetIntField"... does this mean an Int is expected, but something else is passed?
 
JordiCP

JordiCP

Expert
Licensed User
Longtime User
I'm gonna check it. But if it was due to this, I think my example wouldn't work neither

Sorry to insist, but just to be sure. The mutablebitmap is each time the same and is it attached to an ImageView?
 
wimpie3

wimpie3

Well-Known Member
Licensed User
Longtime User
Yes. I have just noticed that when I do this:
B4X: 
myBitmap.InitializeMutable(camEx.GetPreviewSize.Width+1, camEx.GetPreviewSize.Height+1)
instead of this:
B4X: 
myBitmap.InitializeMutable(camEx.GetPreviewSize.Width, camEx.GetPreviewSize.Height)
my app does not crash anymore, but the bitmap returns empty.
 
JordiCP

JordiCP

Expert
Licensed User
Longtime User
Yes, it returns empty because it is checked that width and height arguments are coherent with bitmap size. If not so, it returns and does nothing

I have seen that if a call to AndroidBitmap_lockpixels() fails, I wasn't protecting it and could result in an error.

Attached the modified library which, if this is the case, will report it in the unfiltered logs and return with no crash
 

Attachments

  • NV21util.jar
    63.5 KB · Views: 301
wimpie3

wimpie3

Well-Known Member
Licensed User
Longtime User
Are you sure this is a new version? I get the same crash!

B4X: 
JNI WARNING: threadid=1 using JNI after critical get (GetIntField)
            in Ljcp/b4a/nv21util/NV21util;.proceed:([BIIILandroid/graphics/Bitmap;ZZII)V (GetIntField)
"main" prio=5 tid=1 NATIVE
  | group="main" sCount=0 dsCount=0 obj=0x415a7ca8 self=0x414e1408
  | sysTid=20898 nice=0 sched=0/0 cgrp=apps handle=1074393428
  | state=R schedstat=( 1532801975 463587177 3335 ) utm=125 stm=28 core=1
 
JordiCP

JordiCP

Expert
Licensed User
Longtime User
I think so, the error must be before that line.

I attach a new one with also the XML, just in case. Now there is a log when the native routine is entered, so it must appear


It's quite hard to debug things this way, but it will be useful to see what is happening
 

Attachments

  • NV21.zip
    17.7 KB · Views: 283
wimpie3

wimpie3

Well-Known Member
Licensed User
Longtime User
Ok, I do get a new log line now with the bitmap size, immediately followed by the crash. I think more debug logging is needed. I'm happy to do further testing with you!

B4X: 
Entering native code
Bitmap is w=1024 h=768
JNI WARNING: threadid=1 using JNI after critical get (GetIntField)
            in Ljcp/b4a/nv21util/NV21util;.proceed:([BIIILandroid/graphics/Bitmap;ZZII)V (GetIntField)
"main" prio=5 tid=1 NATIVE
 
JordiCP

JordiCP

Expert
Licensed User
Longtime User
Now the procedure is full of logs. It should help to see where (and why) it fails
 

Attachments

  • NV21.zip
    18.5 KB · Views: 303
wimpie3

wimpie3

Well-Known Member
Licensed User
Longtime User
B4X: 
Entering native code
Bitmap is w=1024 h=768
Needed memory is 786432. gpixels buffer (at 0) size is 0
  ...reallocating memory
  ...Now, gpixels buffer (at 1989939208) is 786432
Call to GetPrimitiveArrayVritical...
    ....data buffer is at 1118809376
Call to AndroidBitmap_lockpixels...
JNI WARNING: threadid=1 using JNI after critical get (GetIntField)
            in Ljcp/b4a/nv21util/NV21util;.proceed:([BIIILandroid/graphics/Bitmap;ZZII)V (GetIntField)
"main" prio=5 tid=1 NATIVE
 
JordiCP

JordiCP

Expert
Licensed User
Longtime User
It crashes in the same call to lockPixels!! There is another log immediatelly after it which should appear if it didn't crash here.

I fear my skills are not enough to find it. But the example still works, don't know why:mad:

Just as a try, if you want to try this slightly different one...
 

Attachments

  • NV21.zip
    18.5 KB · Views: 289
wimpie3

wimpie3

Well-Known Member
Licensed User
Longtime User
Unfortunately, the problem persists. I REALLY hope a solution can be found...

B4X: 
Entering native code
Bitmap is w=1024 h=768
Needed memory is 786432. gpixels buffer (at 0) size is 0
  ...reallocating memory
  ...Now, gpixels buffer (at 1987010568) is 786432
Call to GetPrimitiveArrayVritical...
    ....data buffer is at 1136310224
Call to AndroidBitmap_lockpixels... dest_ptr is 0
JNI WARNING: threadid=1 using JNI after critical get (GetIntField)
            in Ljcp/b4a/nv21util/NV21util;.proceed:([BIIILandroid/graphics/Bitmap;ZZII)V (GetIntField)
"main" prio=5 tid=1 NATIVE
  | group="main" sCount=0 dsCount=0 obj=0x415a7ca8 self=0x414e1408


  | sysTid=2816 nice=0 sched=0/0 cgrp=apps handle=1074393428
  | state=R schedstat=( 5988116829 898342623 8000 ) utm=444 stm=154 core=2
  #00  pc 0000132e  /system/lib/libcorkscrew.so (unwind_backtrace_thread+29)
  #01  pc 00060652  /system/lib/libdvm.so (dvmDumpNativeStack(DebugOutputTarget const*, int)+33)
  #02  pc 00054640  /system/lib/libdvm.so (dvmDumpThreadEx(DebugOutputTarget const*, Thread*, bool)+395)
  #03  pc 000546ae  /system/lib/libdvm.so (dvmDumpThread(Thread*, bool)+25)
  #04  pc 00038804  /system/lib/libdvm.so
  #05  pc 0003e5ae  /system/lib/libdvm.so
  #06  pc 0004e73e  /system/lib/libandroid_runtime.so
  #07  pc 000005cc  /system/lib/libjnigraphics.so (AndroidBitmap_lockPixels+7)
  #08  pc 00001d0c  /data/app-lib/test.android-1/libjcp_000.so (Java_jcp_b4a_nv21util_NV21util_proceed+1320)
  #09  pc 0001dbcc  /system/lib/libdvm.so (dvmPlatformInvoke+112)
  #10  pc 0004e122  /system/lib/libdvm.so (dvmCallJNIMethod(unsigned int const*, JValue*, Method const*, Thread*)+397)
  #11  pc 0004fb0e  /system/lib/libdvm.so (dvmResolveNativeMethod(unsigned int const*, JValue*, Method const*, Thread*)+181)
  #12  pc 00026fe0  /system/lib/libdvm.so
  #13  pc 0002dfa0  /system/lib/libdvm.so (dvmMterpStd(Thread*)+76)
  #14  pc 0002b638  /system/lib/libdvm.so (dvmInterpret(Thread*, Method const*, JValue*)+184)
  #15  pc 00060864  /system/lib/libdvm.so (dvmInvokeMethod(Object*, Method const*, ArrayObject*, ArrayObject*, ClassObject*, bool)+391)
  #16  pc 000687c6  /system/lib/libdvm.so
  #17  pc 00026fe0  /system/lib/libdvm.so
  #18  pc 0002dfa0  /system/lib/libdvm.so (dvmMterpStd(Thread*)+76)
  #19  pc 0002b638  /system/lib/libdvm.so (dvmInterpret(Thread*, Method const*, JValue*)+184)
  #20  pc 00060864  /system/lib/libdvm.so (dvmInvokeMethod(Object*, Method const*, ArrayObject*, ArrayObject*, ClassObject*, bool)+391)
  #21  pc 000687c6  /system/lib/libdvm.so
  #22  pc 00026fe0  /system/lib/libdvm.so
  #23  pc 0002dfa0  /system/lib/libdvm.so (dvmMterpStd(Thread*)+76)
  #24  pc 0002b638  /system/lib/libdvm.so (dvmInterpret(Thread*, Method const*, JValue*)+184)
  #25  pc 00060864  /system/lib/libdvm.so (dvmInvokeMethod(Object*, Method const*, ArrayObject*, ArrayObject*, ClassObject*, bool)+391)
  #26  pc 000687c6  /system/lib/libdvm.so
  #27  pc 00026fe0  /system/lib/libdvm.so
  #28  pc 0002dfa0  /system/lib/libdvm.so (dvmMterpStd(Thread*)+76)
  #29  pc 0002b638  /system/lib/libdvm.so (dvmInterpret(Thread*, Method const*, JValue*)+184)
  #30  pc 00060580  /system/lib/libdvm.so (dvmCallMethodV(Thread*, Method const*, Object*, bool, JValue*, std::__va_list)+335)
  #31  pc 00049d0a  /system/lib/libdvm.so
  at jcp.b4a.nv21util.NV21util.proceed(Native Method)
  at test.android.menuclass._window_closed(menuclass.java:3410)
  at java.lang.reflect.Method.invokeNative(Native Method)
  at java.lang.reflect.Method.invoke(Method.java:515)
  at anywheresoftware.b4a.BA.raiseEvent2(BA.java:174)
  at anywheresoftware.b4a.keywords.Common.CallSub4(Common.java:853)
  at anywheresoftware.b4a.keywords.Common.CallSubNew2(Common.java:810)
  at test.android.dialogpanel._returnvalue(dialogpanel.java:611)
  at test.android.dialogpanel._btnok_click(dialogpanel.java:403)
  at java.lang.reflect.Method.invokeNative(Native Method)
  at java.lang.reflect.Method.invoke(Method.java:515)
  at anywheresoftware.b4a.BA.raiseEvent2(BA.java:174)
  at anywheresoftware.b4a.BA.raiseEvent(BA.java:158)
  at anywheresoftware.b4a.objects.ViewWrapper$1.onClick(ViewWrapper.java:66)
  at android.view.View.performClick(View.java:4438)
  at android.view.View$PerformClick.run(View.java:18422)
  at android.os.Handler.handleCallback(Handler.java:733)
  at android.os.Handler.dispatchMessage(Handler.java:95)
  at android.os.Looper.loop(Looper.java:136)
  at android.app.ActivityThread.main(ActivityThread.java:5017)
  at java.lang.reflect.Method.invokeNative(Native Method)
  at java.lang.reflect.Method.invoke(Method.java:515)
  at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:779)


  at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:595)
  at dalvik.system.NativeStart.main(Native Method)

VM aborting
Fatal signal 6 (SIGABRT) at 0x00000b00 (code=-6), thread 2816 (test.android)
Process com.twitter.android (pid 1871) has died.
 
JordiCP

JordiCP

Expert
Licensed User
Longtime User
Hi wimpie3,

It is such a strange thing. Did you say it happened just in one device and not others? Which Android version?

if you could isolate a small project where it happens and post it, I have different devices to test it

I am also interested in solving it, since I use this library and other similar ones in my projects, so would be good to know what happens

In the library there are two operations: converting from YUV to RGB and rotating writing the results onto the bitmap pixels. If problem persists I can do it differently, but first I would need to be sure
 
wimpie3

wimpie3

Well-Known Member
Licensed User
Longtime User
The problem exists on both of my mobiles, with different versions of Android, but unfortunately I cannot reproduce the bug for you. As soon as I take my camera module out and use that as a standalone program, everything works again. I'll try to see if I can come up with something.
 
JordiCP

JordiCP

Expert
Licensed User
Longtime User
The only thing coming to my mind is to check the following in your code:

* Call to freebuffers in Activity_Pause
* Just in case, be sure that the imageview to which the bitmap is attached is properly initialized and visible before the camera is initted
* Only to have a global instance of a NV21Utils variable
* Not to use other threads involving related variables.


Good luck!
 
freedom2000

freedom2000

Well-Known Member
Licensed User
Longtime User
JordiCP said:
What it does:
  • User can specify the subsampling (1,2,4 or 8, only these values). Depending on your needs, not always full resolution is needed and process time can be decreased this way.


I don't have much time, but if someone finds it useful for processing, I will add a function which makes the conversion directly onto an array instead of a bitmap.


Please let me know if you decide to use it or find any issues with it.:)
Click to expand...

Hi,

Thank you for sharing this lib.
I have run the example on my Nexus 5 Lollipop and it works really well.
I am interested in the "smallest" achievable preview as I don't need a HD image !

My wish it to use it to compute the ambiant color of a TV screen in order to pilot a Milight BUlb to reproduce the TV dominant color

more or less as in the Philips ambiant TV

I already now how to real time pilot a Milight, so should be easy to use your lib and pick the color of a few pixels ?
What would be your advice to do it ? Should I go with your proposal to have the Array option or do you know some already existing image processing methods to do it ?

Thanks again
 
You must log in or register to reply here.
Top