Android Question [B4X] Way to compress/uncompress a string... ?

[Magma]

Hi to all there Again

I have big databases/tables with tracking points.... so i ve thinked for mass-viewing of 1000+ records will be better making from server side a big string (???) record (Generated by server app-probably b4j) that having inside of it a map transformed to string... and compressed too (need to know the way-with example)...

I think that will help because there is no need in this situation more than a record and no need anything to search...

I am thinking too.. if it is better not be a record.. but a file... your opinion ?

1) need to know how i can compress/uncompress string or map ?
2) need to know if it is better to use file or sql record... will be better ?

data:
001 abcd 41.231231 25.65622
002 dfere 42.231221 25.52212
003 dgere 42.231221 25.52212
....
999 agcd 41.221231 25.62622
1000 aycd 41.121231 25.62622

thanks!!

 

[Erel]

A database made of 1000 records is a very small database. Why not embed it in your app with a SQLite database?

 

[emexes]

I agree, 1000 records is nothing.

If you store the longitude and latitude as numbers (preferably Ints scaled by 1000000) then they will reduce down to 4 bytes each = 50% "compression"

If it is a track, ie, points generally going in a direction with speed changing slowly, rather than a list of POIs that are all over the place, then you can happily reduce this down to around one byte per second, for the purpose of logging and archiving.

 

[Magma]

Sorry for... late answering running for family too..

1000 records ---> will be one query...

actually there are different points (tracking) of different vehicles... so i want to make a service (form server side) that every second willl see the different position (if there is difference) and will make one big string/text (record) that will include a map?... or make better a different file have in a map (with the points/strings/tracking not a geo-map)... so need this one file (will have 1000 different vehicles points) to compress it as better i can...

The file could be downloaded from clients / from a jetty sever

Sure from long number if go to int number is a solution... but can i compress it more perhaps with zlib or somehow ?

 

[DonManfred]

1) need to know how i can compress/uncompress string or map ?

Sub Process_Globals
    'These global variables will be declared once when the application starts.
    'These variables can be accessed from all modules.
    Public GZip As GZipStrings
End Sub

Sub Globals
    'These global variables will be redeclared each time the activity is created.
    'These variables can only be accessed from this module.
    Private su As StringUtils
End Sub

Sub Activity_Create(FirstTime As Boolean)
    'Do not forget to load the layout file created with the visual designer. For example:
    'Activity.LoadLayout("Layout1")
   
    Dim org As String = $"Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet.
Duis autem vel eum iriure dolor in hendrerit in vulputate velit esse molestie consequat, vel illum dolore eu feugiat nulla facilisis at vero eros et accumsan et iusto odio dignissim qui blandit praesent luptatum zzril delenit augue duis dolore te feugait nulla facilisi. Lorem ipsum dolor sit amet, consectetuer adipiscing elit, sed diam nonummy nibh euismod tincidunt ut laoreet dolore magna aliquam erat volutpat.
Ut wisi enim ad minim veniam, quis nostrud exerci tation ullamcorper suscipit lobortis nisl ut aliquip ex ea commodo consequat. Duis autem vel eum iriure dolor in hendrerit in vulputate velit esse molestie consequat, vel illum dolore eu feugiat nulla facilisis at vero eros et accumsan et iusto odio dignissim qui blandit praesent luptatum zzril delenit augue duis dolore te feugait nulla facilisi.
Nam liber tempor cum soluta nobis eleifend option congue nihil imperdiet doming id quod mazim placerat facer possim assum. Lorem ipsum dolor sit amet, consectetuer adipiscing elit, sed diam nonummy nibh euismod tincidunt ut laoreet dolore magna aliquam erat volutpat. Ut wisi enim ad minim veniam, quis nostrud exerci tation ullamcorper suscipit lobortis nisl ut aliquip ex ea commodo consequat.
Duis autem vel eum iriure dolor in hendrerit in vulputate velit esse molestie consequat, vel illum dolore eu feugiat nulla facilisis.
At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, At accusam aliquyam diam diam dolore dolores duo eirmod eos erat, et nonumy sed tempor et et invidunt justo labore Stet clita ea et gubergren, kasd magna no rebum. sanctus sea sed takimata ut vero voluptua. est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat.
Consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus.
Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet.
Duis autem vel eum iriure dolor in hendrerit in vulputate velit esse molestie consequat, vel illum dolore eu feugiat nulla facilisis at vero eros et accumsan et iusto odio dignissim qui blandit praesent luptatum zzril delenit augue duis dolore te feugait nulla facilisi. Lorem ipsum dolor sit amet, consectetuer adipiscing elit, sed diam nonummy nibh euismod tincidunt ut laoreet dolore magna aliquam erat volutpat.
Ut wisi enim ad minim veniam, quis nostrud exerci tation ullamcorper suscipit lobortis nisl ut aliquip ex ea commodo consequat. Duis autem vel eum iriure dolor in hendrerit in vulputate velit esse molestie consequat, vel illum dolore eu feugiat nulla facilisis at vero eros et accumsan et iusto odio dignissim qui blandit praesent luptatum zzril delenit augue duis dolore te feugait nulla facilisi.
Nam liber tempor cum soluta nobis eleifend option congue nihil imperdiet doming id quod mazim placerat facer possim assum. Lorem ipsum dolor sit amet, consectetuer adipiscing elit, sed diam nonummy nibh euismod tincidunt ut laoreet dolore magna aliquam erat volutpat. Ut wisi enim ad minim veniam, quis nostrud exerci tation ullamcorper suscipit lobortis nisl ut aliquip ex ea commodo consequat.
Duis autem vel eum iriure dolor in hendrerit in vulputate velit esse molestie consequat, vel illum dolore eu feugiat nulla facilisis."$

    Log($"OrgLength: ${org.Length}"$) ' 6163 Bytes
    '
    ' Compression
    '    
    Dim compressed() As Byte = GZip.compress(org)
    Log($"CompressedBytesLength: ${compressed.Length}"$) ' 665 Bytes
    Dim base64 As String = su.EncodeBase64(compressed)
    Log($"CompressedBytes converted to base64 Length: ${base64.Length}"$) ' 888Bytes

    Log($"CompressedBytes converted to base64: ${base64}"$) ' 888Bytes

    '
    ' Decompression
    '
    Dim decompressedbytes() As Byte = su.DecodeBase64(base64)
    Log($"decompressedbytesLength: ${decompressedbytes.Length}"$)
    Dim bc As ByteConverter
    Dim uncompressed As String = bc.StringFromBytes(decompressedbytes,"UTF8")
    Log($"uncompressedLength: ${uncompressed.Length}"$) ' 6163 Bytes
   
    Log($"Decompressed String = ${uncompressed}"$)
   
End Sub

2) need to know if it is better to use file or sql record... will be better ?

Based on your requirement i would suggest to use a SQLite Database to store the Data. You probably can better filter the Data from the Database.

 

[OliverA]

Please note, if you are using a web server as a back end to retrieve information, then, with most modern web servers, the text data is compressed out of the box (ok, it takes negotiation between the client and the server, but most do support compression). If you compress the data, then you are just doubling up the work already being done.

 

[OliverA]

so i ve thinked for mass-viewing of 1000+ records will be better making from server side a big string

1000 records ---> will be one query...

Not every client that accesses this service should be doing this. Only a "master" viewer. Therefore you should be looking at most clients just pulling up information on data in their immediate surroundings. If you insist/plan on having every device that accesses your service pull all the data on the 1000+ objects, then you may want to look into Google's Firebase DB to push the data down to your devices instead of querying an SQL server constantly.

 

[Magma]

@OliverA Jetty compressing data... ? for server side... and every browser/httpjob-okhttputils2 uncompress ???

 

[Magma]

@DonManfred Already using Mariadb sql server... but i need to take 1000 records/time as 1... not like sql query from every client - because will be the same for every client!

As I said... the part of 1000 record query will be taken at server side one time ! every 1-2 or 5 seconds - i ll decide that... So will export the records to a string/map/something compress it someway and put it at file... then the clients will ask for it (download) when they want the same thing

Thanks for your answer... that is what wanted !

 

[OliverA]

@OliverA Jetty compressing data... ? for server side... and every browser/httpjob-okhttputils2 uncompress ???

Compression has been part of HTTP for awhile. See https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept-Encoding. You can always do a request with HttpJob and then check the response headers to see what was negotiated.