B4J Question image.writetostream using disk - how to avoid it ?

[Magma]

Hi there...

well i am using this fabulous snippet [B4X][XUI] Image to JPEG byte array with resize/quality options but... i ve met something that don't want to have, the image.writetostream is using hdd/ssd when want to compress quality of jpeg... is it possible to avoid... ?

If you have a timer and some/many XUIImageToJPEGByteArray ..and go a "walk" at appdata\local\temp you will see a lot of imageioxxxxxxxxxxxxxxxxxxxxxxxx.tmp files creating/deleting for doing the job... this reading/writting to disk creates a delay to that i want to do...

    Dim out As OutputStream
    out.InitializeToBytesArray(0)
    image.WriteToStream(out, quality,  "JPEG")  'That seems creating .tmp file to do the job....
    out.Close

is it possible using something different to do my job faster / on the fly... ?

ps: i am sure... that ImageWriter/imageio do the same in java...

 

[Magma]

...well after a lot of searching...

Found this...

ImageIO.setUseCache(false);

And yes it has some difference at speed...

Well I put that at first line of OliverA Snippet inline Java (image to bytes)

Made some tests... at same resolution image 1920x1080, same time capturing...

1) OliverA Snippet without setUseCache(false); ... had about 6-7 FPS
2) with setUseCache(false);.... get about 9-10 FPS
3) with OpenCV speed ...about 11-12 FPS (the FASTEST until now)

but generally imageio capturing or writing or making any work... using disk - but with this option not using... don't know what problems would make if not having cache... but the speed has difference... maybe will be good in next versions of b4xbitmap, image generally have the option somway to disable setusecache... (wish)

 

[kimstudio]

@Magma Nice finding!

I searched the Java Docs to understand the reasoning behind:

public static void setUseCache(boolean useCache)
Sets a flag indicating whether a disk-based cache file should be used when creating ImageInputStreams and ImageOutputStreams.
When reading from a standard InputStream, it may be necessary to save previously read information in a cache since the underlying stream does not allow data to be re-read. Similarly, when writing to a standard OutputStream, a cache may be used to allow a previously written value to be changed before flushing it to the final destination.
The cache may reside in main memory or on disk. Setting this flag to false disallows the use of disk for future streams, which may be advantageous when working with small images, as the overhead of creating and destroying files is removed.
On startup, the value is set to true.

 

[Magma]

I am wondering how possible is to use opencv for capturing screen direct.... to mat bytes ?? If it is so fast will increase a lot the FPS

 

[OliverA]

I am wondering how possible is to use opencv for capturing screen direct

Ok, how are you currently capturing the screen?

1) OliverA Snippet without setUseCache(false); ... had about 6-7 FPS
2) with setUseCache(false);.... get about 9-10 FPS
3) with OpenCV speed ...about 11-12 FPS (the FASTEST until now)

Is this just for the image-to-bytes function or screen capture plus image-to-bytes? For just the image-to-bytes conversion, that seems slow unless we are dealing with some huge images. What is the size of the image you are converting?

 

[Magma]

...Well.. for testing purposes I am capture whole screen 1920x1080 for example...

at my app is 1/5 at full screen capture... 4 times .. at region i need... the real size of image could be 4-5MB uncompressed.. or more...

The way of capturing is a JNI/JNA... about 2x times (at least) faster from robot (not using imageio)...

i am adding it here to share it:

#if java
import java.awt.Rectangle;
import java.awt.image.BufferedImage;
import java.awt.image.ColorModel;
import java.awt.image.DataBuffer;
import java.awt.image.DataBufferInt;
import java.awt.image.DataBufferUShort;
import java.awt.image.DirectColorModel;
import java.awt.image.Raster;
import java.awt.image.WritableRaster;

import com.sun.jna.Native;
import com.sun.jna.win32.W32APIOptions;
import com.sun.jna.platform.win32.WinDef;
import com.sun.jna.platform.win32.WinNT;
import com.sun.jna.platform.win32.WinGDI;

public class JNAScreenShot {

    public BufferedImage getScreenshot(Rectangle bounds) {
        WinDef.HDC windowDC = GDI.GetDC(USER.GetDesktopWindow());
        WinDef.HBITMAP outputBitmap =
                GDI.CreateCompatibleBitmap(windowDC,
                bounds.width, bounds.height);
        try {
            WinDef.HDC blitDC = GDI.CreateCompatibleDC(windowDC);
            try {
                WinNT.HANDLE oldBitmap =
                        GDI.SelectObject(blitDC, outputBitmap);
                try {
                    GDI.BitBlt(blitDC,
                            0, 0, bounds.width, bounds.height,
                            windowDC,
                            bounds.x, bounds.y,
                            GDI32.SRCCOPY);
                } finally {
                    GDI.SelectObject(blitDC, oldBitmap);
                }
                WinGDI.BITMAPINFO bi = new WinGDI.BITMAPINFO(40);
                bi.bmiHeader.biSize = 40;
                boolean ok =
                        GDI.GetDIBits(blitDC, outputBitmap, 0, bounds.height,
                        (byte[]) null, bi, WinGDI.DIB_RGB_COLORS);
                if (ok) {
                    WinGDI.BITMAPINFOHEADER bih = bi.bmiHeader;
                    bih.biHeight = -Math.abs(bih.biHeight);
                    bi.bmiHeader.biCompression = 0;
                    return bufferedImageFromBitmap(blitDC, outputBitmap, bi);
                } else {
                    return null;
                }
            } finally {
                GDI.DeleteObject(blitDC);
            }
        } finally {
            GDI.DeleteObject(outputBitmap);
        }
    }

    private BufferedImage bufferedImageFromBitmap(WinDef.HDC blitDC,
            WinDef.HBITMAP outputBitmap,
            WinGDI.BITMAPINFO bi) {
        WinGDI.BITMAPINFOHEADER bih = bi.bmiHeader;
        int height = Math.abs(bih.biHeight);
        final ColorModel cm;
        final DataBuffer buffer;
        final WritableRaster raster;
        int strideBits =
                (bih.biWidth * bih.biBitCount);
        int strideBytesAligned =
                (((strideBits - 1) | 0x1F) + 1) >> 3;
        final int strideElementsAligned;
        switch (bih.biBitCount) {
            case 16:
                strideElementsAligned = strideBytesAligned / 2;
                cm = new DirectColorModel(16, 0x7C00, 0x3E0, 0x1F);
                buffer =
                        new DataBufferUShort(strideElementsAligned * height);
                raster =
                        Raster.createPackedRaster(buffer,
                        bih.biWidth, height,
                        strideElementsAligned,
                        ((DirectColorModel) cm).getMasks(),
                        null);
                break;
            case 32:
                strideElementsAligned = strideBytesAligned / 4;
                cm = new DirectColorModel(32, 0xFF0000, 0xFF00, 0xFF);
                buffer =
                        new DataBufferInt(strideElementsAligned * height);
                raster =
                        Raster.createPackedRaster(buffer,
                        bih.biWidth, height,
                        strideElementsAligned,
                        ((DirectColorModel) cm).getMasks(),
                        null);
                break;
            default:
                throw new IllegalArgumentException("Unsupported bit count: " + bih.biBitCount);
        }
        final boolean ok;
        switch (buffer.getDataType()) {
            case DataBuffer.TYPE_INT: {
                int[] pixels = ((DataBufferInt) buffer).getData();
                ok = GDI.GetDIBits(blitDC, outputBitmap, 0, raster.getHeight(), pixels, bi, 0);
            }
            break;
            case DataBuffer.TYPE_USHORT: {
                short[] pixels = ((DataBufferUShort) buffer).getData();
                ok = GDI.GetDIBits(blitDC, outputBitmap, 0, raster.getHeight(), pixels, bi, 0);
            }
            break;
            default:
                throw new AssertionError("Unexpected buffer element type: " + buffer.getDataType());
        }
        if (ok) {
            return new BufferedImage(cm, raster, false, null);
        } else {
            return null;
        }
    }
    private final User32 USER = User32.INSTANCE;
    private final GDI32 GDI = GDI32.INSTANCE;
}

interface GDI32 extends com.sun.jna.platform.win32.GDI32,
        com.sun.jna.platform.win32.WinGDI,
        com.sun.jna.platform.win32.WinDef {

    GDI32 INSTANCE =
            (GDI32) Native.loadLibrary(GDI32.class);

    boolean BitBlt(HDC hdcDest, int nXDest, int nYDest,
            int nWidth, int nHeight, HDC hdcSrc,
            int nXSrc, int nYSrc, int dwRop);

    HDC GetDC(HWND hWnd);

    boolean GetDIBits(HDC dc, HBITMAP bmp, int startScan, int scanLines,
            byte[] pixels, BITMAPINFO bi, int usage);

    boolean GetDIBits(HDC dc, HBITMAP bmp, int startScan, int scanLines,
            short[] pixels, BITMAPINFO bi, int usage);

    boolean GetDIBits(HDC dc, HBITMAP bmp, int startScan, int scanLines,
            int[] pixels, BITMAPINFO bi, int usage);
    int SRCCOPY = 0xCC0020;
}

interface User32 extends com.sun.jna.platform.win32.User32 {

    User32 INSTANCE = (User32) Native.loadLibrary(User32.class, W32APIOptions.UNICODE_OPTIONS);

    com.sun.jna.platform.win32.WinDef.HWND GetDesktopWindow();
}
#End If

call it like this:

    fxutils.initializestatic("javafx.embed.swing.SwingFXUtils")
    bufimg.InitializeStatic("java.awt.image.BufferedImage")

    Dim Rect1 As JavaObject
    Rect1.InitializeNewInstance("java.awt.Rectangle",Array(0,0,1920,1080))   'here you can set the rectangle of window you need... to capture.. for test purposes leaving it full

    Dim Jo As JavaObject = Me
           
    Jo.InitializeNewInstance("b4j.example.main.JNAScreenShot", Array(Null))

    bufimg=Jo.RunMethod("getScreenshot",Array(Rect1))

'Ofcourse here... i can use your snippet.. directly bufimg to bytes... but I was thinking if there is someway faster... routine...

    Dim bbi As Image=fxutils.runmethod("toFXImage",Array(bufimg,Null))

Next command was your XUIImageToJPEGByteArray snippet...

 

[Magma]

Ok....

BufferImage converting to JPEG Bytes using OpenCV

#if Java

import java.awt.Rectangle;
import java.awt.image.BufferedImage;
import java.awt.image.ColorModel;
import java.awt.image.DataBuffer;
import java.awt.image.DataBufferInt;
import java.awt.image.DataBufferUShort;
import java.awt.image.DirectColorModel;
import java.awt.image.Raster;
import java.awt.image.WritableRaster;

import com.sun.jna.Native;
import com.sun.jna.win32.W32APIOptions;
import com.sun.jna.platform.win32.WinDef;
import com.sun.jna.platform.win32.WinNT;
import com.sun.jna.platform.win32.WinGDI;


import java.awt.AlphaComposite;
import java.awt.image.BufferedImage;
import java.awt.image.DataBufferByte;
import java.awt.Graphics2D;

import javafx.scene.image.Image;
import javafx.embed.swing.SwingFXUtils;

import org.opencv.core.CvType;
import org.opencv.core.Mat;
import org.opencv.core.MatOfByte;
import org.opencv.core.MatOfInt;
import org.opencv.imgcodecs.Imgcodecs;
import org.opencv.imgproc.Imgproc;

// Converts image to JPEG a byte array of the resulting JPEG. Ability to resize and adjust JPEG quality.
// Negative width and height values = %, such that -50 = 50% and -200 = 200%
// Positive width and height values = pixel dimensions
// If one value (either width or height) are 0, then the other value is proportionally
//  calculated from the first.
// If both width and height are 0 or -100, no resizing takes place
// If quality = -1, use Java's default quality



public static byte[] BufImgToJPEGByte(BufferedImage srcImage, int width, int height, int qualityPercent) {


   if ((qualityPercent < -1) || (qualityPercent > 100)) {
      throw new IllegalArgumentException("Quality out of bounds!");
    }
 
   //float quality = qualityPercent / 100f;
   //OpenCV just needs an int
   int quality = qualityPercent;
 
   //OpenCV uses int's
   int oldWidth = (int) srcImage.getWidth(); //aImage.getWidth();
   int oldHeight = (int) srcImage.getHeight(); //aImage.getHeight();
   if (oldWidth == 0 || oldHeight == 0) {
       throw new IllegalArgumentException("Source image with 0 width and/or height!");
   }
 
   boolean resize = true;
   if ((width == 0 && height == 0) || (width == -100 && height == -100)) resize = false;

   BufferedImage destImage;
   Graphics2D g;
   //Need to see if we alread have the correct BufferedImage type, if not, convert
   if (srcImage.getType() != BufferedImage.TYPE_3BYTE_BGR) {
      //http://stackoverflow.com/questions/21740729/converting-bufferedimage-to-mat-opencv-in-java
       destImage = new BufferedImage(srcImage.getWidth(), srcImage.getHeight(), BufferedImage.TYPE_3BYTE_BGR);
       g = destImage.createGraphics();
       g.setComposite(AlphaComposite.Src);
       g.drawImage(srcImage, 0, 0, null);
       g.dispose();
   } else {
      destImage = srcImage;
   }
 
   //Create OpenCV Mat, retrieve data from BufferedImage and assign to Mat
   Mat srcMat = new Mat(destImage.getHeight(), destImage.getWidth(), CvType.CV_8UC3);
   byte[] pixels = ((DataBufferByte) destImage.getRaster().getDataBuffer()).getData();
   srcMat.put(0, 0, pixels);
   Mat destMat;
 
   if (resize) {
      int newWidth = width;
      int newHeight = height;
      // Calculate new dimensions
      if (newWidth < 0) newWidth = -1 * oldWidth * newWidth / 100;
      if (newHeight < 0) newHeight = -1 * oldHeight * newHeight / 100;
      if (newWidth == 0) newWidth = oldWidth * newHeight / oldHeight;
      if (newHeight == 0) newHeight = oldHeight * newWidth / oldWidth;

      int interpolation;
      if ((newHeight * newWidth) > (oldHeight * oldWidth)) {
         //upscaling
         interpolation = Imgproc.INTER_CUBIC;
      } else {
         //downscaling
         interpolation = Imgproc.INTER_AREA;
      }
      destMat = new Mat(newHeight, newWidth, srcMat.type());
      Imgproc.resize(srcMat, destMat, destMat.size(), 0, 0, interpolation);
   } else {
      destMat = srcMat;
   }
 
   //Convert mat to JPG byte array of given quality
   MatOfByte mob=new MatOfByte();                        //Destination Mat byte array
   int[] intParams = new int[2];                        //Create parameters for imencode
   intParams[0] = Imgcodecs.IMWRITE_JPEG_QUALITY;  //IMWRITE_JPEG_QUALITY
   intParams[1] = (int) quality;
   MatOfInt params = new MatOfInt(intParams);
   Imgcodecs.imencode(".jpg", destMat, mob, params);        //.jpg
 
   return mob.toArray();                                //Convert Mat byte array to regular byte array
}

#End If

Sub BufImgToJPEGByteArray(ar As JavaObject, width As Int, height As Int, quality As Int) As Byte()
    Dim jo As JavaObject = Me
    Return jo.RunMethod("BufImgToJPEGByte", Array As Object (ar, width, height, quality))
End Sub

use like this:

    Dim Rect1 As JavaObject
    Rect1.InitializeNewInstance("java.awt.Rectangle",Array(0,0,1920,1080))  'setting region

            Jo.InitializeNewInstance("b4j.example.main.JNAScreenShot", Array(Null)) 'my screenshot routine... new instance...

            bufimg=Jo.RunMethod("getScreenshot",Array(Rect1)) 'my screenshot routine... grab
          
            Dim bbi2() As Byte
            bbi2=BufImgToJPEGByteArray(bufimg, 0,0,75)  'return jpeg bytes...

get +1 FPS extra speed.. ofcourse when region is smaller... much more FPS.... i think the next level will be delta diff capturing... but is very difficult to achieved...

 

[OliverA]

BufferImage converting to JPEG Bytes using OpenCV

I would let Java take care of the resizing. I know you don't use it (you just use the quality setting). Why let Java do it? For now, a BufferedImage of TYPE_3BYTE_BGR is created, no matter what. Java can do that and resize the image in one (for us) step. My limited testing has shown that this is the most efficient means of doing it. Also, through testing, I've noticed that when you shrink the image, the process really speeds up (makes sense, less data to manipulate).

On the other hand, I don't really like the Windows Bitmap to Java BufferedImage (either SHORT, or INT) to Java BufferedImage (BYTE) to OpenCV mat. That is a lot of data shoveling. I'm currently researching Bitmap to OpenCV mapping or just doing it with Windows GDI (if none of the code is to be ever used on another platform).

 

[Magma]

I would let Java take care of the resizing. I know you don't use it (you just use the quality setting). Why let Java do it? For now, a BufferedImage of TYPE_3BYTE_BGR is created, no matter what. Java can do that and resize the image in one (for us) step. My limited testing has shown that this is the most efficient means of doing it. Also, through testing, I've noticed that when you shrink the image, the process really speeds up (makes sense, less data to manipulate).

On the other hand, I don't really like the Windows Bitmap to Java BufferedImage (either SHORT, or INT) to Java BufferedImage (BYTE) to OpenCV mat. That is a lot of data shoveling. I'm currently researching Bitmap to OpenCV mapping or just doing it with Windows GDI (if none of the code is to be ever used on another platform).

you mean...

not to pass at b4x variables of javaobject... just call once java ... to capture at bufferedimage and convert it to bytes?

not using
bufimg=Jo.RunMethod
bbi2=BufImgToJPEGByteArray
...
but do it with one call to java ?

 

[OliverA]

you mean...

not to pass at b4x variables of javaobject... just call once java ... to capture at bufferedimage and convert it to bytes?

not using
bufimg=Jo.RunMethod
bbi2=BufImgToJPEGByteArray
...
but do it with one call to java ?

Yes, sort of. I've got some ideas... Need some time... And we need to give Java another chance. I think the Java code to convert to JPG may be sped up. Need to do some testing...