Android Example B4A_CartoonCamera : Real Time Image Processing using Renderscript Intrinsics

This demo app shows how to use RenderScript ScriptIntrinsics in real time image processing. As an example the camera preview is converted into a cartoon like image. The processed image can be saved by touching the screen.

From Android developer page:
"RenderScript is a framework for running computationally intensive tasks at high performance on Android . . . The RenderScript runtime parallelizes work across processors available on a device, such as multi-core CPUs and GPUs . . . RenderScript is especially useful for applications performing image processing, computational photography, or computer vision."

Renderscript was introduced in API 11 (Honeycomb), but to use it you need to learn an additional programming language, a kind of a C (C99) dialekt.

As from API 18 (JellyBean 4.3) Renderscript comes with a bunch of predefined scripts, called ScriptIntrinsics. These presets cover important functionality like image blur, color matrix transformations (e.g. greyscale, sepia) , convolution kernel processing (e.g. image sharpening, edge detection), lookup tables, color separation, level thresholding, image masking and blending and more. These Intrinsics are optimized for the devices hardware and are usually the fastest way to perform a specific operation.


To achieve the cartoon effect we do:

1. Convert the camera preview data to RGB using ScriptIntrinsicYuvToRGB
2. Slightly blur the RGB image : using ScriptIntrinsicBlur
3. Reduce the 16.7 m RGB-colors to a set of 27 palette colors : using ScriptIntrinsicLUT (LUT = lookup table) -> "palette image"
4. Convert RGB image to greyscale : using ScriptIntrinsicColorMatrix
5. Convolve greyimage with a kernel to find edges in image : using ScriptIntrinsicConvolve3x3
6. Threshold edges : using ScriptIntrinsicLUT -> "edge image"
7. Combine "palette image" with "edge image" : using ScriptIntrinsicBlend

RenderScript uses it´s own data types, called Elements. There are predefined Elements to store a single value (e.g. byte, float) or a 4-element vector (of byte, float...). Image processing often uses Element.U8_4 (unsigned 8 bit, 4 layers), which stores the color information of exact one pixel of an RGBA-Bitmap.

When creating a ScriptIntrinsic object, you have to pass the Element type the script will process.

The scripts work on "Allocations", where the raw and processed data is stored. These Allocations have to be created too. A typical use of Allocations to process an image looks like:

1. copy Bitmap-data into an in-Allocation (input)
2. pass the in-Allocation to the script
3. run the script while passing an out-Allocation (output)
4. copy out-Allocation to bitmap

When creating an Allocation, you have to pass the Allocation´s size (in 1,2 or 3 dimensions) and the Element type the Allocation stores. For conveniance (starting with API 18) an Allocation that holds image (pixel) data can be created directly from a Bitmap.

As many of the RenderScript (and other needed) classes are not exposed to B4A, the JavaObject library (2.05+) will be used. To compile the code you will also need the following libraries : camera (2.20+), Reflection (2.40+), Phone (2.26+); you also have to import the CameraExClass module (1.30+). Minimum API level is 18 (JellyBean 4.3). Apk was created with B4A 4.3.


Creating a RenderScript object is a computatively costly task, so create it once and use it as long as the process lives.

Sub Process_Globals
  Dim mRS As JavaObject  ' the RenderScript object
  Dim mScriptYuvToRGB, mScriptColorize, mScriptBlur, mScriptGrey, mScriptCon3x3, _
  mScriptBlend, mScriptThreshold As JavaObject  ' the ScriptIntrinsics
  Dim aYuv, aRGB, aBlur, aBlurColorized, aGreyBlur, aEdges  As JavaObject  ' the allocations to store data
End Sub

To keep things simple (and to keep the focus on the Renderscript stuff), a fixed (but common) camera preview size is used and there are no menues, buttons, sliders or other controllers to vary parameters and so on.

Sub Globals
  Private camEx As CameraExClass  ' camera
  Dim previewWidth As Int=640  ' for simplicity, use a camera preview size
  Dim previewHeight As Int=480  ' that works on most devices
  Dim yuvDataLength As Int = previewWidth*previewHeight*3/2  ' NV21 format : 12 bit per pixel
  Dim panelPreview,panelProcessed As Panel  ' panels & bitmap
  Dim bmpOut As Bitmap
  Dim frame As Int=0  ' count frames and sum up processing time  '
  Dim tSum As Long=0

  ' play with these parameters to tweak the result
  Dim blurRadius As Float = 3.0f  ' a float 0.0f < r <= 25.0f
  Dim edgeEnhanceFactor As Double = 3
  Dim edgesThreshold As Double=30  ' 0 to 255
End Sub

Checks API version, initialize bitmap and panels for output, init RenderScriptObjects

Sub Activity_Create(FirstTime As Boolean)
  ' check API version , must be >= 18
  Dim ph As Phone
  If ph.sdkVersion<18 Then
    Msgbox("Your API level is: " & ph.sdkVersion,"This app requires API >= 18 (JellyBean 4.3)")
    Activity.Finish
  End If

  bmpOut.InitializeMutable(previewWidth,previewHeight)  ' standard RGBA-bitmap, 4 bytes per pixel

  panelProcessed.Initialize("")
  Activity.AddView(panelProcessed,0,0,100%x,100%y)  ' the processed image
  panelPreview.Initialize("")
  Activity.AddView(panelPreview,0,0,20%x,20%y)  ' tiny camera preview in upper left corner

  If FirstTime Then initRenderScriptStuff  ' init only once

  ToastMessageShow("Touch display to save image",False)
End Sub

In this sub the RenderScript object, the Elements, Scripts and Allocations are created.

Sub initRenderScriptStuff

  ' get Activity context
  Dim jo As JavaObject
  jo.InitializeContext

  ' create the Renderscript object
  mRS = mRS.InitializeStatic("android.renderscript.RenderScript").RunMethodJO("create",Array(jo))

  ' renderscript elements
  Dim eU8,eU8_4 As JavaObject

  'Element.U8(rs) : unsigned 8 bit, holds 1 byte
  eU8 = eU8.InitializeStatic("android.renderscript.Element").RunMethodJO("U8",Array(mRS))

  'Element.U8_4(rs) : unsigned 8 bit 4 layers, holds 1 RGBA-pixel (4 bytes)
  eU8_4 = eU8_4.InitializeStatic("android.renderscript.Element").RunMethodJO("U8_4",Array(mRS))

  ' script to convert Yuv to Rgb
  mScriptYuvToRGB = mScriptYuvToRGB.InitializeStatic("android.renderscript.ScriptIntrinsicYuvToRGB").RunMethodJO("create",Array(mRS,eU8_4))

  ' script applying a lookup table for cartoon colorization, maps 16.7 m possible colors to 27 palette colors
  mScriptColorize = mScriptColorize.InitializeStatic("android.renderscript.ScriptIntrinsicLUT").RunMethodJO("create",Array(mRS,eU8_4))
  Dim value As Int
  For i=0 To 255
    value=Floor(i/85.001)*127  '  reduce 256 to 3 intensities : 0, 127, 254
    mScriptColorize.RunMethod("setRed",Array(i,value))
    mScriptColorize.RunMethod("setGreen",Array(i,value))
    mScriptColorize.RunMethod("setBlue",Array(i,value))
  Next

  ' script to convert RGB to greyscale
  mScriptGrey = mScriptGrey.InitializeStatic("android.renderscript.ScriptIntrinsicColorMatrix").RunMethodJO("create", Array As Object(mRS,eU8_4))
  mScriptGrey.RunMethod("setGreyscale",Null) ' use a build-in preset color matrix, RGB -> greyscale

  ' script to blur image
  mScriptBlur = mScriptBlur.InitializeStatic("android.renderscript.ScriptIntrinsicBlur").RunMethodJO("create",Array As Object(mRS,eU8_4))
  mScriptBlur.RunMethod("setRadius",Array(blurRadius))

  ' script to convolve image with a 3x3 kernel
  mScriptCon3x3 = mScriptCon3x3.InitializeStatic("android.renderscript.ScriptIntrinsicConvolve3x3").RunMethodJO("create",Array(mRS,eU8_4))
  ' set up 3x3 convolution kernel
  Dim coeff(9) As Float = Array As Float (1, 1, 1, 1,-8, 1, 1, 1, 1)
  For i=0 To 8
    coeff(i)=coeff(i)*edgeEnhanceFactor
  Next
  mScriptCon3x3.RunMethod("setCoefficients",Array(coeff)) ' transfer kernel to script

  ' script applying a lookup table to threshold edges, giving a black / white "edge mask"
  mScriptThreshold = mScriptThreshold.InitializeStatic("android.renderscript.ScriptIntrinsicLUT").RunMethodJO("create",Array(mRS,eU8_4))
  Dim value As Int
  For i=0 To 255
    value=0  ' no edge = black
    If i>edgesThreshold Then value=255  'edge = white
    mScriptThreshold.runMethod("setRed",Array(i,value))
    mScriptThreshold.RunMethod("setGreen",Array(i,value))
    mScriptThreshold.RunMethod("setBlue",Array(i,value))
  Next

  ' script to blend 2 images
  mScriptBlend = mScriptBlend.InitializeStatic("android.renderscript.ScriptIntrinsicBlend").RunMethodJO("create",Array(mRS,eU8_4))


  ' the Allocations:

  ' aYuv is 1-dimensional, it holds camera preview data (array of bytes with length yuvDataLength)
  aYuv = aYuv.initializeStatic("android.renderscript.Allocation").RunMethodJO("createSized",Array(mRS,eU8,yuvDataLength))

  ' all other allocations hold 4-byte pixel data and are 2-dimensional, they are created from an initialized (RGBA) bitmap
  aRGB = aRGB.InitializeStatic("android.renderscript.Allocation").RunMethodJO("createFromBitmap",Array(mRS,bmpOut))
  aBlur = aBlur.InitializeStatic("android.renderscript.Allocation").RunMethodJO("createFromBitmap",Array(mRS,bmpOut))
  aBlurColorized = aBlurColorized.InitializeStatic("android.renderscript.Allocation").RunMethodJO("createFromBitmap",Array(mRS,bmpOut))

  aGreyBlur = aGreyBlur.InitializeStatic("android.renderscript.Allocation").RunMethodJO("createFromBitmap",Array(mRS,bmpOut))
  aEdges = aEdges.InitializeStatic("android.renderscript.Allocation").RunMethodJO("createFromBitmap",Array(mRS,bmpOut))

End Sub

The image processing happens in the "camera loop":

' this Sub is triggered automatically by the os each time new Camera preview data is avaliable
' do not create renderscript objects here !
Sub Camera1_preview(yuvData() As Byte)
  Dim tStart, tStop As Long
  frame=frame+1
  tStart=DateTime.Now

  ' convert camera data (NV21 mode) to RGB
  aYuv.RunMethod("copyFrom",Array(yuvData))
  mScriptYuvToRGB.Runmethod("setInput",Array(aYuv))
  mScriptYuvToRGB.Runmethod("forEach",Array(aRGB))

  ' blur RGB image
  mScriptBlur.RunMethod("setInput",Array(aRGB))
  mScriptBlur.RunMethod("forEach",Array(aBlur))

  ' reduce 16.7 mil. colors to 27 cartoon colors, store the result in aBlurColorized
  mScriptColorize.RunMethod("forEach",Array(aBlur,aBlurColorized))

' convert blurred RGB image to greyscale
  mScriptGrey.RunMethod("forEach",Array(aBlur,aGreyBlur))

  ' detect edges ; edges are light, surfaces are dark
  mScriptCon3x3.RunMethod("setInput",Array(aGreyBlur))
  mScriptCon3x3.RunMethod("forEach",Array(aEdges))

  ' threshold edges -> detected edges are now white (255,255,255), other image parts are black (0,0,0)
  mScriptThreshold.RunMethod("forEach",Array(aEdges,aEdges))

  ' subtract the "edge image" from the "colorized image", clipping negative values to 0 (=black)
  mScriptBlend.RunMethod("forEachSubtract",Array(aEdges,aBlurColorized))

  ' result is in aBlurColorized, copy allocation to bitmap
  aBlurColorized.RunMethod("copyTo",Array(bmpOut))

  ' show bitmap
  panelProcessed.SetBackgroundImage(bmpOut)

  tStop=DateTime.Now
  tSum=tSum+tStop-tStart
  Log(tSum/frame)  ' log average processing time per frame in milliseconds
End Sub

In these subs the camera is set up and the preview ist started / stopped:

Sub Activity_Resume
  InitializeCamera
End Sub

Sub Camera1_Ready (Success As Boolean)
  If Success Then
    camEx.SetPreviewSize(previewWidth,previewHeight)
    camEx.SetContinuousAutoFocus
    camEx.CommitParameters
    camEx.StartPreview
  Else
    ToastMessageShow("Cannot open camera.", True)
  End If
End Sub

Private Sub InitializeCamera
  camEx.Initialize(panelPreview, False, Me, "Camera1") ' only back camera is supported
End Sub

Sub Activity_Pause (UserClosed As Boolean)
  camEx.Release
End Sub

When touching the screen the actual image is saved. You can access the image from the Galerie app.

' touch display to save image
Sub Activity_Touch (Action As Int, x As Float, y As Float) As Boolean
  Select Action
  Case Activity.ACTION_DOWN
    Dim filename As String
    Dim picnumber As Int=0

    ' generate an unique filename
    filename="B4A_CartoonCamera" & picnumber & ".jpg"
    Do While File.Exists(File.DirRootExternal,filename)=True
      picnumber=picnumber+1
      filename="B4A_CartoonCamera" & picnumber & ".jpg"
    Loop

    ' save image
    Dim out As OutputStream
    out = File.OpenOutput(File.DirRootExternal, filename, False)
    bmpOut.WriteToStream(out, 50, "JPEG")
    out.Close

    ' register saved image
    Dim i As Intent
    i.initialize("android.intent.action.MEDIA_SCANNER_SCAN_FILE", _
    "file://" & File.Combine(File.DirRootExternal, filename))
    Dim ph As Phone
    ph.SendBroadcastIntent(i)

    Log("saved as: " & filename)
    ToastMessageShow("Image saved as :  " & filename,False )

  Case Activity.ACTION_MOVE
  Case Activity.ACTION_UP
  End Select
  Return True
End Sub

Finally, don´t forget to uncomment this Sub in the CameraExClass module:

'Uncomment this sub if you need to handle the Preview event
Sub Camera_Preview (Data() As Byte)
  If SubExists(target, event & "_preview") Then
  CallSub2(target, event & "_preview", Data)
  End If
End Sub