Android Question OpenCV nMatToBitmap Assertion Failed

[PABLO2013]

cvexception [org.opencv.core.cvexception: opencv(3.4.1) error: (-215) src.dims == 2 && info.height == (uint32_t)src.rows && info.width == (uint32_t)src.cols in function void java_org_opencv_android_utils_nmattobitmap2(jnienv*, jclass, jlong, jobject, jboolean)

greetings I receive the following error when trying to use opencv with bitmaps from a camera, reviewing some comments I see that they are the characteristics of bitmap, the question would be how to know the characteristics of the bitmap and how I can change them to make them compatible with opencv, thanks

You will be able to resolve your error with following way.

First check whether your input image has only single channel. You can check it by running print img.shape. If the result is like (height, width, 3), then the image is not single channel. You can convert the image into a single channel one by:

img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

Then check whether the image type is not float. You can check it by running print img.dtype. If the result is related to float, you will need to change it also by:
img = img.astype('uint8')

And also the last thing, it is not actually an error in this case. But it can be an error in the future if you keep practicing this method of combining multiple flags. When you are using multiple flags, remember to combine then with not a plus sign, but with a | sign.

 ret3,th3 = cv2.threshold(blur,0,255,cv2.THRESH_BINARY | cv2.THRESH_OTSU)

 

[JordiCP]

Seems that the exception is raised because one of the expected conditions is not met. That is, the MAt must have 2 dimensions (openCV Mats can have an arbitrary number), and its number of rows and cols must be the same as the bitmap width and height.

Try logging the Mat's number of rows and cols, and also the bitmap's width and height before calling the matToBitmap function.

 

[PABLO2013]

greetings thanks jordiCP, reviewing the code there is a verification that I could not pass

    'prevent queued events from previous camera settings, if any. Just in case
    If PreviewPic.Length<>(3*myBitmap(0).Width*myBitmap(0).Height/2) Then
        Log("Preview pic length is: "&PreviewPic.Length)
        Log("Bitmap size is: w="&myBitmap(0).Width&" h="&myBitmap(0).Height)
        Log("Not processing")
       Return ---------------------------------------------
    End If

1. in my case I use standard sizes 640x480 --- for this part. ---(3*myBitmap(0).Width*myBitmap(0).Height/2)

2. as my frames are bmp I convert this DJIFPVWidget1.GetBitmapWH(640,480)

Sub BitmapToByte(Image As Bitmap) As Byte()
    ToastMessageShow("BitmapToByt",False)
Dim out As OutputStream
Dim bb() As Byte
out.InitializeToBytesArray(0)
Image.WriteToStream(out,100,"JPEG")
out.Close
bb=out.ToBytesArray
Log(bb.Length)
Return bb

End Sub

3. Camera1_Preview (BitmapToByte (DJIFPVWidget1.GetBitmapwh(640,480)))


4. PreviewPic.Length<>(3*myBitmap(0).Width*myBitmap(0).Height/2) this is true for all cases and gets me
..... In the end I can't get both parts to be compatible with each other and between opencv, originally this comes from the device's camera but I am injecting the frames from the decoding of another camera, As I say, how could I match both parts of this check, I don't know if it will be the density of my frames, the size, etc.

 

[JordiCP]

That comparison check only applies when the image comes from an Android camera preview event. In that case, the image format is YUV, and its length is 3*width*height/2 because of its internal format. In the examples, it is later converted to RGBA and/or grayscale.

If you already have an Android bitmap with known dimensions, you can directly convert it to a Mat and skip this comparison part, which does not apply in your case.

 

[PABLO2013]

greetings thanks jordi, this is the result, the frames are sent to the detector, and if there is a flow of frames towards it, but they look like this, thanks for your comments.

 

[JordiCP]

Difficult to guess what is happening here --> you are continuously getting frames from one source and want to process them with openCV in Android, right?. Is this source the same Android camera or is it another source? Please provide more info.

 

[PABLO2013]

you are continuously getting frames from one source and want to process them with openCV in Android, right?... yes.
Is this source the same Android camera or is it another source?.... Drone Camara

The source is the drone camera DJIFPVWidget1.GetBitmapWH(640,480) , make a code that Decodes the source to frames (bitmap, 720x1200), so far so good, but I try to enter the OCV BlobDetector example to which I remove all the part of the cameraExclass and some values I assumed them at 640x480, even the frames (bitmap, 720x1200) I converted them to 640x480, I don't know if that helps, I ask you because I don't know much about the subject and I don't know how to guide you, to find a solution, thanks


As data, use imageManipulations2 too and it recognizes circles, is ok .

 

[JordiCP]

The source is the drone camera DJIFPVWidget1.GetBitmapWH(640,480) , make a code that Decodes the source to frames (bitmap, 720x1200), so far so good, but I try to enter the OCV BlobDetector example to which I remove all the part of the cameraExclass and some values I assumed them at 640x480, even the frames (bitmap, 720x1200) I converted them to 640x480, I don't know if that helps, I ask you because I don't know much about the subject and I don't know how to guide you, to find a solution, thanks

If I understood correctly, you are getting a 640x480 bitmap from the drone camera. Then you are converting it to 720x1200 and later converting it back to 640x480?

Which is the size and format of the bitmap received by Android? Can this received bitmap be correctly seen in an Imageview? I'd like to help but can't guess what you are doing and where things take place (drone part or Android side) unless you provide more accurate info.

 

[PABLO2013]

This is the code, I tell you that I think the error occurs in this part

mYuvImage.put4(0,0,PreviewPic)
mImgProc.cvtColor(mYuvImage,mRgba,mImgProc.COLOR_YUV2RGBA_NV21,4)

and you told me that this is only for device cameras, but I don't know how to change it for my bmp frames

Well, I don't know how to pass my bmp to it, the color assignment is done directly by mBlobColorHsv.Initialize3(121,216,181), so that it is not necessary to choose it from the screen, my bmp (720 x 1200 , 160 , A888)

thanks

#Region Module Attributes
    #FullScreen: False
    #IncludeTitle: False
    #ApplicationLabel: OCV BlobDetector
    #VersionCode: 5
    #VersionName:
    #SupportedOrientations: landscape
    #CanInstallToExternalStorage: False
#End Region

'======================================================================================
' Version history
'    1.    -First version
'    2.    -Forces camera to work 480x640. In previous version it was assuming so, but crashed on devices where it did not happen (Samsung S6)
'        -(Good practices) All OCV objects needed in the main processing loop (camera preview) are initialized before and reused.
'    3   -Version 2 solution for camera size was not correct. Fixed
'======================================================================================

'======================================================================================
' BlobDetector example
'     Point with the camera to a large-enough colored object (better if it has big contrast with the rest of the image) and tap onto it
'     Based on its mean color, it will detect the biggest color blob with a similar color and draw a square on it
'======================================================================================

'Activity module
Sub Process_Globals

    Private frontCamera As Boolean = False
 
End Sub

Sub Globals
 
    Private Panel1 As Panel
    Private camEx As CameraExClass

    Dim DESIREDCAMERAWIDTH As Int = 640        'Set it to a resolution that your camera supports. 640x480 is more than enough to detect color blobs in most cases.
    Dim DESIREDCAMERAHEIGHT As Int =480


    Dim PreviewWidth As Int                ' Preview width and Height related to camera own orientation
    Dim PreviewHeight As Int
    Dim PreviewScreenWidth As Int        ' Prevew width and height in screen
    Dim PreviewScreenHeight As Int
 
 
    Dim myBitmap(1) As Bitmap
    Dim myIV(1) As ImageView

    'OCV
    Dim ocl As OCVOpenCVLoader
 
    Dim mCvt As OCVCvType
    Dim mUtils As OCVUtils
    Dim mImgProc As OCVImgproc
 
    'BLobDetectorActivity
    Private mIsColorSelected As Boolean=False
    Private mYuvImage As OCVMat
    Private mRgba As OCVMat
    Private mBlobColorRgba As OCVScalar
    Private mBlobColorHsv As OCVScalar
    Private mDetector As clBLobDetector
    Private mSpectrum As OCVMat
    Private SPECTRUM_SIZE As OCVSize
    Private CONTOUR_COLOR As OCVScalar
    Private ROTATED_CONTOUR_COLOR As OCVScalar
    Dim CameraBmp As DJIFPVWidget
    Private timer1 As Timer

End Sub

Sub Activity_Create(FirstTime As Boolean)
 
    Activity.LoadLayout("1")
 
    myIV(0).Initialize("")
    myIV(0).Gravity=Gravity.FILL

    Dim HI As Int =100%Y
    Dim WI As Int=HI*DESIREDCAMERAWIDTH/DESIREDCAMERAHEIGHT 'keep aspect ratio
    Panel1.SetLayout(0, 0,WI,HI)
    Activity.AddView(myIV(0),0,0,WI,HI)
    timer1.Initialize("timer1", 30)

     
End Sub


Sub Activity_Resume
'    InitializeCamera
End Sub

Private Sub InitializeCamera
'    camEx.Initialize(Panel1, frontCamera, Me, "Camera1")
'    frontCamera = camEx.Front
End Sub

Sub Activity_Pause (UserClosed As Boolean)
'    camEx.Release
End Sub




Sub Camera1_Ready (Success As Boolean)
    If Success Then
 
 
        PreviewWidth = 640            '640?
        PreviewHeight = 480
 
        'Create the bitmap in display orientatiom
        'If ((PreviewWidth>PreviewHeight) And (100%X>100%Y)) Or ((PreviewWidth<PreviewHeight) And (100%X<100%Y)) Then
        If True Then
            PreviewScreenHeight=PreviewHeight
            PreviewScreenWidth=PreviewWidth        
        Else
            PreviewScreenHeight=PreviewWidth
            PreviewScreenWidth=PreviewHeight    
        End If

        Log(camEx.DispRotation)
 
        For k=0 To myBitmap.Length-1
            myBitmap(k).InitializeMutable( PreviewScreenWidth, PreviewScreenHeight )                    
            myIV(k).Bitmap=myBitmap(k)
        Next
     
        ' Original Java code
        'mRgba = new Mat(height, width, CvType.CV_8UC4);
        'mDetector = new ColorBlobDetector();
        'mSpectrum = new Mat();
        'mBlobColorRgba = new Scalar(255);
        'mBlobColorHsv = new Scalar(255);
        'SPECTRUM_SIZE = new Size(200, 64);
        'CONTOUR_COLOR = new Scalar(255,0,0,255);
     
        mYuvImage.Initialize2(3*PreviewHeight/2,PreviewWidth,mCvt.CV_8UC1)
        mRgba.Initialize2(PreviewHeight,PreviewWidth,mCvt.CV_8UC4)
        mDetector.Initialize
        mBlobColorRgba.Set(Array As Double(255))
        mBlobColorHsv.Set(Array As Double(255))
        SPECTRUM_SIZE.Set(Array As Double(200,64))
        CONTOUR_COLOR.Set(Array As Double(255,0,255,0))
        ROTATED_CONTOUR_COLOR.Set(Array As Double(0,255,0,255))
 
    Else
        ToastMessageShow("Cannot open camera.", True)
    End If
End Sub


Sub  Button_Touch () ' I select de color directly..... mBlobColorHsv.Initialize3(121,216,181)
 
    Dim x As Int=0
    Dim y As Int=0
     
#if 0
//JAVA original code

       int cols = mRgba.cols();
        int rows = mRgba.rows();

        int xOffset = (mOpenCvCameraView.getWidth() - cols) / 2;
        int yOffset = (mOpenCvCameraView.getHeight() - rows) / 2;

        int x = (int)event.getX() - xOffset;
        int y = (int)event.getY() - yOffset;

        Log.i(TAG, "Touch image coordinates: (" + x + ", " + y + ")");

        if ((x < 0) || (y < 0) || (x > cols) || (y > rows)) return false;

        Rect touchedRect = new Rect();

        touchedRect.x = (x>4) ? x-4 : 0;
        touchedRect.y = (y>4) ? y-4 : 0;

        touchedRect.width = (x+4 < cols) ? x + 4 - touchedRect.x : cols - touchedRect.x;
        touchedRect.height = (y+4 < rows) ? y + 4 - touchedRect.y : rows - touchedRect.y;

        Mat touchedRegionRgba = mRgba.submat(touchedRect);

        Mat touchedRegionHsv = new Mat();
        Imgproc.cvtColor(touchedRegionRgba, touchedRegionHsv, Imgproc.COLOR_RGB2HSV_FULL);

        // Calculate average color of touched region
        mBlobColorHsv = Core.sumElems(touchedRegionHsv);
        int pointCount = touchedRect.width*touchedRect.height;
        for (int i = 0; i < mBlobColorHsv.val.length; i++)
            mBlobColorHsv.val[i] /= pointCount;

        mBlobColorRgba = converScalarHsv2Rgba(mBlobColorHsv);

        Log.i(TAG, "Touched rgba color: (" + mBlobColorRgba.val[0] + ", " + mBlobColorRgba.val[1] +
                ", " + mBlobColorRgba.val[2] + ", " + mBlobColorRgba.val[3] + ")");

        mDetector.setHsvColor(mBlobColorHsv);

        Imgproc.resize(mDetector.getSpectrum(), mSpectrum, SPECTRUM_SIZE);

        mIsColorSelected = true;

        touchedRegionRgba.release();
        touchedRegionHsv.release();

        return false; // don't need subsequent touch events
#End If

    Dim cols As Int = mRgba.cols()
    Dim rows As Int =  mRgba.rows()
 
    Dim radius As Int = 20
    Dim pointX As Int = X*PreviewWidth/myIV(0).Width
    Dim pointY As Int = Y*PreviewHeight/myIV(0).Height
 
    Dim touchedRegionRgba As OCVMat = mRgba.submat(pointY-radius,pointY+radius,pointX-radius,pointX+radius)'Y-10,Y+10,X-10,X+10)
    Dim touchedRegionHsv As OCVMat
    touchedRegionHsv.Initialize
    mImgProc.cvtColor(touchedRegionRgba,touchedRegionHsv,mImgProc.COLOR_RGB2HSV_FULL,3)
 
    Dim mCore As OCVCore
    mBlobColorHsv = mCore.sumElems(touchedRegionHsv)
    Dim pointCount As Int = 4*radius*radius
    For i=0 To mBlobColorHsv.val.Length-1
        mBlobColorHsv.val(i)=mBlobColorHsv.val(i)/pointCount
    Next
 
 
 
    mBlobColorHsv.Initialize3(121,216,181)
    mBlobColorRgba = convertScalarHsv2Rgba(mBlobColorHsv) ' I select de color directly
 
 
 
 
    mDetector.HsvColor=mBlobColorHsv
    mSpectrum=mSpectrum.zeros(1,1,mCvt.CV_8UC1)
     
    Dim mSpectrumTmp As OCVMat = mDetector.Spectrum
    mImgProc.resize1(mSpectrumTmp,mSpectrum,SPECTRUM_SIZE)
 
    mIsColorSelected=True
End Sub



Sub Timer11_Tick
PreviewBmp (BitmapToByte (CameraBmp.GetBitmapwh(640,480)))
End Sub


Sub BitmapToByte(Image As Bitmap) As Byte()
    ToastMessageShow("BitmapToByt",False)
Dim out As OutputStream
Dim bb() As Byte
out.InitializeToBytesArray(0)
Image.WriteToStream(out,100,"JPEG")
out.Close
bb=out.ToBytesArray
Log(bb.Length)
Return bb

End Sub

Sub PreviewBmp (PreviewPic() As Byte)

    'prevent queued events from previous camera settings, if any. Just in case
'    If PreviewPic.Length<>(3*myBitmap(0).Width*myBitmap(0).Height/2) Then
'        Log("Preview pic length is: "&PreviewPic.Length)
'        Log("Bitmap size is: w="&myBitmap(0).Width&" h="&myBitmap(0).Height)
'        Log("Not processing")
'        Return
'    End If

    'Dim now As Long = DateTime.Now


'.....................................................
    mYuvImage.put4(0,0,PreviewPic)        ' I thin my problem is in this part the error is here
    mImgProc.cvtColor(mYuvImage,mRgba,mImgProc.COLOR_YUV2RGBA_NV21,4)
'....................................................



    If mIsColorSelected=False Then Return
     
     
    mDetector.process(mRgba)
    Dim contours As List = mDetector.Contours
    'Log("Contours count: "&contours.Size)
 
    Dim colorLabel As OCVMat = mRgba.submat(4,68,4,68)
    colorLabel.setTo(mBlobColorRgba)
 
    Dim spectrumLabel As OCVMat= mRgba.submat(4,4+mSpectrum.rows(),70,70+mSpectrum.cols())
    mSpectrum.copyTo(spectrumLabel)
     
    If contours.Size>0 Then

        Dim mRect As OCVRect = mImgProc.boundingRect(contours.Get(0))
        Dim mList As List
        mList.Initialize2(Array(mRect))
        mImgProc.rectangle(mRgba,mRect.tl,mRect.br,CONTOUR_COLOR,1dip,8,0)
     
        Dim myContour0 As OCVMatOfPoint = contours.Get(0)    'OCVMatOfPoint2f in 3.20 version
     
        Dim mType As OCVCvType
        Dim myContour As OCVMatOfPoint2f    'OCVMatOfPoint2f in 3.20 version
        myContour.Initialize
        myContour0.convertTo2(myContour, mType.CV_32F)
     
        Dim myContour2 As OCVMatOfPoint2f        'must be 2f or approxPolyDP complains
        mImgProc.approxPolyDP(myContour,myContour2,0.1,True)
        Dim myContour3 As OCVMatOfPoint
        myContour3.Initialize3(myContour2.toArray)

        Log(myContour2.toArray.Length)
        Dim contour2List As List
        contour2List.Initialize2(Array As OCVMatOfPoint(myContour3))
        'contour2List.Add(myContour3)
        mImgProc.drawContours2(mRgba,contour2List,0,CONTOUR_COLOR)

    End If

    'Log("Ellapsed: "& (DateTime.Now-now))
 
    mUtils.matToBitmap(mRgba,myBitmap(0),False)
    myIV(0).invalidate    'Refresh the view

End Sub

Private Sub convertScalarHsv2Rgba(hsvColor As OCVScalar) As OCVScalar
 
    Dim pointMatRgba As OCVMat
    pointMatRgba.Initialize
    Dim pointMatHsv As OCVMat
    pointMatHsv=pointMatHsv.zeros(1,1,mCvt.CV_8UC3)
    pointMatHsv.setTo(hsvColor)
    mImgProc.cvtColor(pointMatHsv,pointMatRgba, mImgProc.COLOR_HLS2RGB_FULL,4)
    Dim retScalar As OCVScalar
 
    Dim d() As Double=pointMatRgba.get5(0,0)
    retScalar.Set(d)


Return retScalar
 
End Sub