B4J Question JPEG - Append String Blocs Together to Form Valid Image

mmieher

Active Member
Licensed User
Longtime User
This is a continuation of this --> https://www.b4x.com/android/forum/t...e-file-cluelessness.145987/page-2#post-925674

I very much appreciate @drgottjr posting the B4a code. I have been attempting to duplicate it in my B4j Exporterator.

The B4a app appears to simply StringBuilder append the blocs of binary blobs fetched from the map buried in the Json string? However, when I try this I get "half" an image, and now with strange star-dots (attached).

The math regarding the "total size" and the "offset" does not work. The B4a app does not appear to care about either of these things.

I am missing something simple.

Here is a bunch of info...

B4X:
Waiting for debugger to connect...
Program started.
AppStart
SUB  ProcessFile
SUB  ParseRecord:  inJson.Lenth = 87570
top level = **********************************************
Dim parser As JSONParser
parser.Initialize(<text>)
Dim root As Map = parser.NextObject
Dim Blob_Segment_No As String = root.Get("Blob_Segment_No")
Dim Blob_Image_ID As String = root.Get("Blob_Image_ID")
Dim Blob_Person_ID As String = root.Get("Blob_Person_ID")
Dim Blob_Total_Size As String = root.Get("Blob_Total_Size")
Dim Blob_Seg_Offset As String = root.Get("Blob_Seg_Offset")
Dim Blob_Format As String = root.Get("Blob_Format")
Dim Blob_Segment As Map = root.Get("Blob_Segment")
Dim $binary As String = Blob_Segment.Get("$binary")
Dim $type As String = Blob_Segment.Get("$type")
------------------------------------------------------------------
Blob_Segment_No = 0
Blob_Total_Size = 108533
Blob_Seg_Offset = 0
*** new image ***
binary from map length = 87384
NEW sb.Length = 87384
rCount = 1
------------------------------------------------------------------
Blob_Segment_No = 1
Blob_Total_Size = 108533
Blob_Seg_Offset = 65536
*** append to image ***
binary from map length = 57332
NEW sb.Length = 144716
rCount = 2
SaveImage:  sb.Length = 144716
*** wrote bytes ***

B4X:
Private Sub ProcessFile
    Log("SUB  ProcessFile")    
    
    '    show top-level json
    Dim TR As TextReader
    TR.Initialize(File.OpenInput(File.DirAssets,"myexport.json"))
    Dim rLine As Object = TR.ReadLine
    Dim result As String = LogB4jCode(rLine)
    LogError("top level = **********************************************" & CRLF)
    Log(result)
    
    Do While rLine<> Null
        LogError("------------------------------------------------------------------")

        ProcessRecord(rLine)
        
        rCount = rCount + 1
        Log("rCount = " & rCount)
        If rCount = 2 Then
            SaveImage
            Exit
        End If
        
        rLine = TR.ReadLine
        
    Loop
    
End Sub

Private Sub ProcessRecord(inJson As String)
    'Log("ProcessRecord: inJson.Length = " & inJson.Length)
    
    Dim Parser As JSONParser
    Parser.Initialize(inJson)
    
    Dim jRoot As Map = Parser.NextObject
    
    Dim Blob_Segment_No As String = jRoot.Get("Blob_Segment_No")
    Log("Blob_Segment_No = " & Blob_Segment_No)
    
    'Dim Blob_Image_ID As String = jRoot.Get("Blob_Image_ID")
    'Log("Blob_Image_ID = " & Blob_Image_ID)
    'Dim Blob_Person_ID As String = jRoot.Get("Blob_Person_ID")
    'Log("Blob_Person_ID = " & Blob_Person_ID)
    Dim Blob_Total_Size As String = jRoot.Get("Blob_Total_Size")
    Log("Blob_Total_Size = " & Blob_Total_Size)
    Dim Blob_Seg_Offset As String = jRoot.Get("Blob_Seg_Offset")
    Log("Blob_Seg_Offset = " & Blob_Seg_Offset)
    'Dim Blob_Format As String = jRoot.Get("Blob_Format")
    'Log("Blob_Format = " & Blob_Format)
    Dim Blob_Segment As Map = jRoot.Get("Blob_Segment")
    'Log("Blob_Segment.Size = " & Blob_Segment.Size)
    ''Dim binary As String = Blob_Segment.Get("$binary")
    'Log("binary 30 = " & binary.SubString2(0, 30))
    'Dim jType As String = Blob_Segment.Get("$type")
    'Log("jType = " & jType)

    If Blob_Segment_No = 0 Then
        LogColor("*** new image ***", 0xFFD8FF00 )

        If sb.IsInitialized Then
            SaveImage
        End If

        sb.Initialize
        sb.Append(GetBinaryData(Blob_Segment))
        
    Else
        LogColor("*** append to image ***", 0xFFD8FF00)
        
        Dim str As String = GetBinaryData(Blob_Segment)
        sb.Append(str)
            
    End If

    Log("NEW sb.Length = " & sb.Length)
    
End Sub

Private Sub SaveImage
    Log("SaveImage:  sb.Length = " & sb.Length)
    
    If sb.Length = 0 Then
        LogError("zero length imgstr")
        Return
    End If

    '    write out new image
    Dim su As StringUtils
    Dim by() As Byte
    by = su.DecodeBase64(sb.ToString)
    File.WriteBytes(DIR,"test.jpeg", by)
    
    LogColor("*** wrote bytes ***",  0xFFFF00AA)
    
End Sub


Private Sub GetBinaryData(inMap As Map) As String
    'Log("SUB  GetBinaryData:  inMap.Size = " & inMap.size)
    
    Dim rStr As String
    rStr = inMap.Get("$binary")
    
    Log("binary from map length = " & rStr.Length)
    
    Return rStr
    
End Sub
 

Attachments

  • test.jpeg
    test.jpeg
    64 KB · Views: 55
Solution
uses original "json" file and reads in entries line by line. my b4x way to join arrays is just about the same as solutions using java, so i'm leaving mine in. android may have an "arrayconcat" method using guava, but we coding for b4j here.

drgottjr

Expert
Licensed User
Longtime User
ok, here's story. there is an error/exception in my b4a solution which does not register in the log. i wasn't aware of it until porting the b4a version to b4a (attached).
although the processing logic is correct, the conversion from base64 string ---> bytes has to be done at an earlier place in the processing loop. the attached example addresses that. there are no long any missing parts of the image.

the concatenation of the base64 strings in each json object is what's incorrect. each base64 string needs to be decoded to bytes first, and the bytes "concatenated". those bytes will produce a full image. the joining of the base64 strings first and then decoding to bytes is wrong. it has to be the other way around.

if the example attached is run as is, the difference will be visible to anyone who has followed the thread from b4a to here. after the first image has been displayed, click the "click me" button to see the next image.

for the moment i am using my modified json file that the op posted in the earlier thread. i will revert to the unmodified version and an updated example after a short nap.

also, i don't think we have a routine to add elements to an array, so i did it in a roundabout way. i hope to clean that up. i would normally use inline java, which does have a way to deal with array modification and such, but i wanted to post something that showed the images in full.
 

Attachments

  • mmieher2.zip
    120.4 KB · Views: 46
Upvote 0

drgottjr

Expert
Licensed User
Longtime User
uses original "json" file and reads in entries line by line. my b4x way to join arrays is just about the same as solutions using java, so i'm leaving mine in. android may have an "arrayconcat" method using guava, but we coding for b4j here.
 

Attachments

  • mmieher3.zip
    237.8 KB · Views: 47
Last edited:
Upvote 0
Solution

mmieher

Active Member
Licensed User
Longtime User
ok, here's story. there is an error/exception in my b4a solution which does not register in the log. i wasn't aware of it until porting the b4a version to b4a (attached).
although the processing logic is correct, the conversion from base64 string ---> bytes has to be done at an earlier place in the processing loop. the attached example addresses that. there are no long any missing parts of the image.

the concatenation of the base64 strings in each json object is what's incorrect. each base64 string needs to be decoded to bytes first, and the bytes "concatenated". those bytes will produce a full image. the joining of the base64 strings first and then decoding to bytes is wrong. it has to be the other way around.

if the example attached is run as is, the difference will be visible to anyone who has followed the thread from b4a to here. after the first image has been displayed, click the "click me" button to see the next image.

for the moment i am using my modified json file that the op posted in the earlier thread. i will revert to the unmodified version and an updated example after a short nap.

also, i don't think we have a routine to add elements to an array, so i did it in a roundabout way. i hope to clean that up. i would normally use inline java, which does have a way to deal with array modification and such, but i wanted to post something that showed the images in full.
You are a heavy dude, Dr. Gotti. Thank you so much.

I would need to study to understand, but it even looks cool visually:
B4X:
  For i = 0 To l.Size - 1
        Dim b() As Byte = l.Get(i)
        Dim x As Int
        For x = 0 To b.Length - 1
            image(x + offset) = b(x)    
        Next
        offset = offset + b.Length
    Next
 
Upvote 0

drgottjr

Expert
Licensed User
Longtime User
i was surprised to see that a number of online java solutions to combining arrays used the exact same steps
i used in b4x. arrays in java are fixed. there are some specialized libraries for android (eg, guava) that
address this, but i thought seeing what is involved might be helpful. life could have been a little simpler had i
kept track of the total bytes needed for each image chunk as i went along instead of having to determine how
many bytes i was going to need when i was ready to build an image from the bytes in all the chunks. it would
have saved a couple steps at the end.
you should be able to recreate your lost images from whatever file it is that you found. i didn't add any save as
routine, but once you have the bytes, you can handle that step. carry on.
 
Upvote 0

mmieher

Active Member
Licensed User
Longtime User
I have it all working.

Not as pretty as yours, but this really only needs to run without error one time. Adobe.Everything will take over from there.

I never realized that a List could be used in this manner. Not sure why it never occurred to me.

Since your B4a version, I have been puzzled by the list called "collection". How to make? I tried Notepad+ but the world hated me for it. Did not understand the need to code it in the first place.

I think something is not right in this comment:
B4X:
    '{"Blob_Person_ID":"zflas","Blob_Image_ID":"0","Blob_Segment_No":"0","Blob_Format":"\u0003","Blob_Segment":{"$binary":
    '"$type": "00"},"Blob_Seg_Offset":"0","Blob_Total_Size":"108533"}]}

I get stuck easily, so my first action is to go through or around whatever I need to learn later, and get to the target. Will Chit Chat any images that are #superterrific.

Thanks again.
 
Upvote 0

drgottjr

Expert
Licensed User
Longtime User
in the assets folder for the b4a version (at least), you should find images.json and orgimages.json. the
latter is your original file. the former is how i turned a text file of disparate json strings into 1 json string
containing a list of json strings i could loop through with our usual json parser.

i happen to have done it with notepad++, but it could have been done programatically. but when i thought
a little about your file, i realized that it was possible to process it on a line by line basis without altering
the main program logic: the concatenation of chunks of base64 strings into a jpg image. so i reverted to
reading the file line by line.

the little commented snippet that you posted and called "not right" is the result of my modification. the
"]}" at the end are not in your file.

so, first, a list can hold anything, even another list. you normally don't want to mix what's in the list, as it
makes processing a little tricky, although - technically - a list of objects is probably legal, and you can leave
it to whoever comes after you to figure out how to process the list. anyway, whereas
a stringbuilder was applicable for concatenating base64 string chunks, it wouldn't work chunks of bytes.
so i switched to a list. you can't use an array to hold the chunks because java arrays can't be extended.
a list is limited only by memory. once all the chunks for a given image have been added to a list, you can
get back to an array, which is what is needed to create a bitmap. so you can think of a list as an
ever expanding array. you just need to get back to array for certain operations. in fact, java itself has a
built-in list.toArray() method to do just that. i think i looked at the methods our list supports; i don't recall
seeing a toArray() method.

as to what "Collection" is all about, your file is essentially 3 lines of json strings (i've truncated the strings to fit across the page instead of wrapping around):

{"Blob_Person_ID":"zflas",...,"Blob_Segment_No":"0","BlobSegment":{"$binary": "/9j/4AAQSkZJRgABAAEAlgCWAAD// ...","$type": "00"},...}
{"Blob_Person_ID":"zflas",...,"Blob_Segment_No":"1","Blob_Segment":{"$binary": "cAe3+e1Zq71ZjbX ...", "$type": "00"},...}
{"Blob_Person_ID":"zflas",...,"Blob_Segment_No":"0","Blob_Segment":{"$binary": "/9j/4AAQSkZJRgABAAEAlgCWAAD ...", "$type": "00"},...}

to turn these 3 json strings in 1 big json string, you add some scaffolding:

{"Collection": [
{"Blob_Person_ID":"zflas",...,"Blob_Segment_No":"0","BlobSegment":{"$binary": "/9j/4AAQSkZJRgABAAEAlgCWAAD// ...","$type": "00"},...}
{"Blob_Person_ID":"zflas",...,"Blob_Segment_No":"1","Blob_Segment":{"$binary": "cAe3+e1Zq71ZjbX ...", "$type": "00"},...}
{"Blob_Person_ID":"zflas",...,"Blob_Segment_No":"0","Blob_Segment":{"$binary": "/9j/4AAQSkZJRgABAAEAlgCWAAD ...", "$type": "00"},...}
]}

if you look at the 1st and 3rd entries, you see the $binary values begin with "/9j". this is the base64 signature for a jpg file. if you look at the 2nd entry's $binary value, there is no such signature. it's the missing chunk belonging to the image started in entry 1. it needs to be added to entry 1. there is no law that says there might not be more chunks. your file sample only presented 2 chunks for the first image. you can see the "Blog_Segment_No" counting from 0. you add the chunks until you hit another segment 0 (new image)
 
Last edited:
Upvote 0

TILogistic

Expert
Licensed User
Longtime User
I'm sorry:
I just saw this post and how I'm working with json and images.

Note:
As an example take as reference Blob_Total_Size.
See logs.

My contribution B4X:
Use of StringUtils, B4XBytesBuilder and Others
1.gif
 

Attachments

  • mmieher-oparra.zip
    120.4 KB · Views: 48
Upvote 0

mmieher

Active Member
Licensed User
Longtime User
I plowed though the 1869 images. Some good stuff.

No Jon Bon Jovi, but this was all worth it.

You all are the best ever.
I ❤️ NERDS!
 
Upvote 0
Top