Android Question Random Access file data

[RickV]

I need some assistance in making the following code work as a random access random record size read and write chunck or fixed width read and write if you will.

Spent a couple of hours reading many posts in the forum, with many coffees LOL, but not sure why I am getting a syntax error on the if and end end if statements.......... any ideas ?

Basically I am writing a zmodem crash recovery protocol to transfer image file data over a tcp-ip socket. Cannot use ftp must be done with this type of implementation as b4x data packets arrive out of sequence in the astreams new data event. (widely documented)

If anyone can see issues, please let me know the correct way.
Thanks

    Dim aaa As String
    Dim RAF As RandomAccessFile
    Dim b() As Byte
    Dim x() As Byte
            
    TXRec = Rec ' record number
    RAF.Initialize(File.DirInternalCache, fName, True)
    RAF.CurrentPosition = Rec
    If RAF.Size - (RAF.CurrentPosition + 4096) => 4096 Then '   <------ this line throws syntax error in logs window
            x = RAF.ReadBytes(b, 0, 4096, TXRec)
            TXRec = TXRec + 4096
            aaa = BytesToString(x, 0, x.Length, CharSet)
            SendIt(18550, Pad(fName,20,0) & aaa) 'this handles the encryption, checksums etc and sends the data
        End If  '   <------ this line throws syntax error in logs window

        If RAF.Size - (RAF.CurrentPosition + 2048) => 2048 Then
            x = RAF.ReadBytes(b, 0, 2048, TXRec)
            TXRec = TXRec + 2048
            aaa = BytesToString(x,0,x.Length,CharSet)
            SendIt(18550, Pad(fName,20,0) & aaa)
        End If
'        '''''''  
'        '''''''  
        If RAF.Size - (RAF.CurrentPosition + 1) => 1 Then
            x = RAF.ReadBytes(b, 0, 1, TXRec)
            TXRec = TXRec + 1
            aaa = BytesToString(x,0,x.Length,CharSet)
            SendIt(18550, Pad(fName,20,0) & aaa)
        End If
        If RAF.Size - (RAF.CurrentPosition + 1) > RAF.Size  Then
            SendIt(18551, Pad(fName,20,0))
        End If
    'End If
    RAF.Close

 

[RickV]

Here is a modified version of the code, the above error has since disappeared however, I reall don't think I am using the readbytes bit correctly as I am getting an index out of bounds error on the first read.

Dim aaa As String
    Dim RAF As RandomAccessFile
    Dim b() As Byte
    Dim x() As Byte
            
    TXRec = Rec
    RAF.Initialize(File.DirInternalCache, fName, True)
    RAF.CurrentPosition = Rec
    
    If RAF.Size - (RAF.CurrentPosition + 4096) > 4095 Then
        RAF.ReadBytes(b, 1, 4096,  TXRec)
            TXRec = TXRec + 4096
            aaa = BytesToString(x, 0, x.Length, CharSet)
            SendIt(18550, Pad(fName,20,0) & aaa)
        End If
    '....
    '.....
    RAF.Close

 

[Erel]

I don't understand the code.
There are many simpler ways to build a custom binary datastore.
The correct solution depends on the use case. Do you want to store millions of records? What is the expected data size?

KeyValueStore is probably a good option.

 

[teddybear]

Basically I am writing a zmodem crash recovery protocol to transfer image file data over a tcp-ip socket. Cannot use ftp must be done with this type of implementation as b4x data packets arrive out of sequence in the astreams new data event. (widely documented)

1. I don't understand why you said this b4x data packets arrive out of sequence in the astreams new data event.

Here is a modified version of the code, the above error has since disappeared however, I reall don't think I am using the readbytes bit correctly as I am getting an index out of bounds error on the first read.

Dim aaa As String
    Dim RAF As RandomAccessFile
    Dim b() As Byte
    Dim x() As Byte
   
    TXRec = Rec
    RAF.Initialize(File.DirInternalCache, fName, True)
    RAF.CurrentPosition = Rec
 
    If RAF.Size - (RAF.CurrentPosition + 4096) > 4095 Then
        RAF.ReadBytes(b, 1, 4096,  TXRec)
            TXRec = TXRec + 4096
            aaa = BytesToString(x, 0, x.Length, CharSet)
            SendIt(18550, Pad(fName,20,0) & aaa)
        End If
    '....
    '.....
    RAF.Closes

2.When doing ReadBytes, you did not allocate any space to the byte array b and x
The correct way：

Dim b(4096),x(...) as byte

 

[RickV]

1. I don't understand why you said this b4x data packets arrive out of sequence in the astreams new data event.

2.When doing ReadBytes, you did not allocate any space to the byte array b and x
The correct way：

Dim b(4096),x(...) as byte

Thank you i will try that ☺

 

[RickV]

I don't understand the code.
There are many simpler ways to build a custom binary datastore.
The correct solution depends on the use case. Do you want to store millions of records? What is the expected data size?

KeyValueStore is probably a good option.

Thanks I will read up on the keystore topic.

the code should flow like this....
1 User pics image then transfer starts
2 the file is open as binary random access no specific record length
3 Lets assume the file to be transferred is 100k, the file is opened, the recuested record is 1 as its the initial start of the transfer. The file size is grater 4096 bytes so the first record is pulled at 4096 bytes and transmitted. Current record request is increased to 4096 (actual bytes sent).
3 The next request is record is 4097, the code re-executes and checks if the file size is > 4096 + 4096 and pulls the next 4096 bytes otherwise, checks file size is > 4096 + 2048 and so on so that we pull and send the next file chunk in specific block sizes.

the data store will probably wind up being a GB at full capacity. the images will be evidence images and stored in SQL datbase according to job number as primary key.

 

[teddybear]

It looks like you want to transfer image files using Zmodem protocol, why don't you use FTP to do that?
Server side you can use b4x ftp server, and client side you can use Net library.

 

[RickV]

It looks like you want to transfer image files using Zmodem protocol, why don't you use FTP to do that?
Server side you can use b4x ftp server, and client side you can use Net library.

yes thats exactly what I am doing but dont have ftp server and some Equpment is hard coded PLC and PIC using zmodem.

 

[RickV]

Am I using the RAF and the set current byte and readbytes correctly ?

ok so I set the byte arrays width to 4096 to 1 progressivily down (not shown here)

I feed into this sub containing the below code to get the record at TXRec. TXRec is sent back by the server app after receiving the first data chunk at what ever width. In the case below after the initial run set to 1, the sever responds with 4097 indicating that is the next byte of the file it wants.

It succeeds with the first 4096 bytes but fails on the second 4096 bytes. The server receives 4096, sends back "I want 4097" but instead somehow gets bytes 1-4096 again even after correctly requesting 4097, 8143(2x4096) but never increments through the file getting the next block.

The file is 2.5MB so there is plenty of data available.

Any ideas what I have missed? I think I have been looking at this for too long LOL Coffee break now ☺

sub zmTX(fName as string, TXRec as long)

    RAF.Initialize(File.DirInternal, fName, True)
    RAF.CurrentPosition = TXRec
    If (RAF.Currentposition + 4096) <= RAF.Size Then
        Dim x(4096) As Byte
        RAF.ReadBytes(x, 0, 4096,  TXRec)
        TXRec = TXRec + 4096
        aaa = BytesToString(x, 0, x.Length, CharSet)
        'sendit handles the headers crc etc and TX
        SendIt(18550, Pad(fName,20,0) & ERec(aaa.Length) & aaa)
        RAF.Close
        TotalBytesSent = TotalBytesSent + 4096
        Return
    End If
    '=========================================
    ' Code above repeats but the values below
    '=========================================
    '2048
    '1024
    '...
    '...
    '8
    '4
    '2
    '1
    '0 respond done
End Sub

 

[teddybear]

How do you call sub zmTX? I guess you use a loop to read the file, 4k bytes each time, then package them into zmodem packet and send it.
I have 2 questions for the sub
1.how to pass TXRec to the sub, I guess it is the send back "I want 4097"?
2.using (RAF.Currentposition + 4096) <= RAF.Size is for testing end of the file?
it also is not good way to open the file in the sub.
you'd better post a small project.

 

[drgottjr]

i think i'm pretty much on the same page as @teddybear.
i've been following this thread (and one similar to it). i'm having
some difficulty understanding it, and i wanted to comment if it's ok.

no one said you have to use async. asynch behavior aside, tcp does
not guarantee sending or receiving packets in "order". it has nothing to
do with b4x. b4x uses the same underlying network and file transfer
protocols as implemented in java and native code used by
commercial servers. it is the same code, not some b4x protocol.
the tcp protocol handles such issues. frankly, it's hard to imagine the
entire system working as well as it does if everybody was following his
own schemes. even if async packets arrive out of sequence (and you
must, for some reason, use async) , you could use the same method
used by tcp to keep track of things.

if someone were to implement zmodem, once an 8-bit serial protocol is
handed over to tcp, the packet which makes up a particular zmodem
packet is wrapped in a tcp packet. tcp basically ignores what you may
have done; its protocol insures error correction and delivery and, if
necessary, piecing together packets that arrive "out of order". it has no idea
what you had to go through to create a packet of zmodem data. tcp is
at work on both sides of the connection. tcp on your side tells the other side
what's coming. tcp on the other side verifies or asks for retransmission. this
occurs more than once for each packet.

reading bytes from a file is usually not done the way you are proposing.
readBytes() in java and C returns the number of bytes read. you choose a
convenient chunk size and stick with it until there are no more bytes to read.
the system tells you when you've reached the end. if you need to pad the
chunk, it would be for the last chunk read.

then there is the matter of storing images in a database, but that's another matter.

 

[RickV]

The sub zmTX is called after the user selects a file / image (a file). After contentchooser has a success event, I copy the file to dirinternal as temp.xxx. At this point I initiate the transfer

...
...
if success then
    zmTX("temp.xxx",1)
end if

the zmodem tranfer sub then open the file gets the record of "chunck size" that fits than calls sendit.

sub sendit(ControlCode as long, TheDataToSend as string)
    'not code but so you see whats going on.......
    dim OutString as string
    OutString = Header(FF+02) & length of TheDataToSend (xxyyzz)
    'xxyyzz is a three byte integer I developed in 90's for 
    'radio telemetry systems int's were too small and longs 
    'too large for the processor at the time.
    OutString = OutString & CRC(TheDataToSend)
    OutString = OutString & CRC(OutString)
end sub

ok, so at this point, we have selected the file to transmit to the server app, initiated the zmodem transfer.

Now, the server app pulls the data according to "Control Code" .and process the packet according to its type. In this case 18550 us zmodem packet in.

I check the packet crc if ok them pull the data chunck and checks its crc. All good take the data chunk at width x and append it to the filename sent along with the data chunck and place it at the record 1 (first byte on new file). Now that the file size is (assuming large file transfer and chunk size 4096) the new size is 4096, the server sends a packet back with the filename and the next byte that it expects which is 4097.

The yser aoo now sees this packet (new data arived event) and processes the packet data Control Code. It sees 18550 so is zmodem packet file data request. We call zmTX(filename, 4097) and the next round happens.

Ive had this working in VB6 enterprise for years and still works on telemtery today and software updates for remote locations. Would be nice to get it working in b4x as may people may benifit from this, maybe not.

Now there was a comment about the tcpip packets and async serializor. I had read a post of Erels last year about the async streams , the serializer etc. In that post / tutorial whatever it was, Erel had mentioned that there is NO garentee to the order that the packets would arrive and the expected data would be the full packet size.

In other words, we would have to collect all the data arrived, construck the string of data that the port receives, port being either tcp/uip serial whatever, the conduit. Then I process the datalength byte if the header exists. If the data is not long enough to match the packet size we store the input string as a variable and append to it as the next data packet is arives. Recheck the data length bytes and if => then continue to actually process the data packet.

I am using the character set ISO??59 I think for memory to allow this way of comms to work with a "string" over byte array.

☺☺☺ Yes there is probably a better way but just trust what I been using for years.

 

[RickV]

Sorry for the typos.... Anyhow, just wanted to mention that zmodem, even though an older protocol, well so is FTP, but it is a reliable protocol. XModem and YModem and even Kermit are very old protocols used way back in the 80s, I remember the old 300 baud transfer rates.... yeah I know, we used to get arroused at 4800 then the 9600 baud modems came out.

Regardless, the radio telemetries usually run on 1200 baud HF and UHF based 4800.

Its mentioned the old 8 bit... NIC were once 8 bit, then 16 bit and 32 bit busses but at the end of the day the modern general NIC is still only 100 MBit. These also have a cpnnection protocol crc etc which TCP sits on top of.

It is also probably important to mention here, telcos well in Australia anyway, wont allow incomming connnections to devices over the telco networks so everything has to be initiated from the device end. Now what most of the younger generation dont realise, FTP, SMTP, HTTP is from what I see anyway, all labelled as "the internet" and where any inter-network comms is required it is straight away the answer. Its not always the go-to solution. Sometimes we need to use the "internet" as the preferred comms tunnel but what we poke down the tube can be a variety of coms protocols, like even just plain text, no checksums "Hello world".

We are now in a techno world of PIC, PLC, Arduimo etc. other chips are atached to give these programable chips other devices, a bluetooth radio, a NIC even optics and WiFi.

So maybe zmodem isnt as dead as some may think? It is always good to have tools laying around from yesteryear.

 

[RickV]

How do you call sub zmTX? I guess you use a loop to read the file, 4k bytes each time, then package them into zmodem packet and send it.
I have 2 questions for the sub
1.how to pass TXRec to the sub, I guess it is the send back "I want 4097"?
2.using (RAF.Currentposition + 4096) <= RAF.Size is for testing end of the file?
it also is not good way to open the file in the sub.
you'd better post a small project.

Full handshake between client and server. see below.

 

[RickV]

Forgot to answer TeddyBear q2. No to check that the specific chunk size would fit before reading past the end of a file causing an error. Its more of select the largest block that fits.

ok so further to the post........

3 byte integer ---- BS I hear you say hahhahaha

Here is the code for a 3 byte integer not so long LOL
Things we had to do to make stuff work back in the day !!!

Values in and out are 0 - 16,581,375 = 255x255x255
erec encodes number to 3 byte string
drec decodes number 3 byte string to long

Maybe this code could help others in other unusual projects.

Rec(aaa As String) As Long
    
    If aaa.Length <>3 Then Return 0

    Dim MSB As Long
    Dim MmSB As Long
    Dim LSB As Long

    MSB = Asc(aaa.CharAt(0)) * 65536
    MmSB = Asc(aaa.CharAt(1)) ' * 256)
    MmSB = MmSB * 256
    LSB = Asc(aaa.CharAt(2))
    Return  (MSB + MmSB + LSB)
End Sub

Sub ERec(nRec As Long) As String

    Dim xx As Int
    Dim yy As Int
    Dim zz As Int
        
    xx = nRec / 65536 'msb
    yy = (nRec - (xx * 65536)) / 256
    zz = nRec Mod 256  'lsb
    
    Return Chr(xx) & Chr(yy) & Chr(zz)
End Sub

 

[RickV]

It looks like you want to transfer image files using Zmodem protocol, why don't you use FTP to do that?
Server side you can use b4x ftp server, and client side you can use Net library.

@teddybear - https://www.b4x.com/android/forum/t...d-with-socket-and-asyncstreams.74320/#content
I have just downloaded this and had a quick peek. I see that there needs to be a range of ports to be allowed through the router. I am a little confused in its operation. Lets assume I set the FTP listen port to 21, wouldn't the server part handle the concurrent and subsequent ports as the web server only requires one public port?

 

[teddybear]

if success then
zmTX("temp.xxx",1)
end if

1. Why is it 1 instead of 0 when calling zmTX("temp.xxx",1)？

2. Acording to your sub zmTX in post#9 , you only send the first chunk of the file, where or when you will send the next chunk of the file?

3.Regarding the FTP protocol, you can see https://en.wikipedia.org/wiki/File_Transfer_Protocol

 

[RickV]

1. Why is it 1 instead of 0 when calling zmTX("temp.xxx",1)？

2. Acording to your sub zmTX in post#9 , you only send the first chunk of the file, where or when you will send the next chunk of the file?

3.Regarding the FTP protocol, you can see https://en.wikipedia.org/wiki/File_Transfer_Protocol

1. Why is it 1 instead of 0 when calling zmTX("temp.xxx",1)？

A: Thats the first byte of the file we are sending. I have spotted the error with first byte should be zero however the process should still function just one byte in front.


2. Acording to your sub zmTX in post#9 , you only send the first chunk of the file, where or when you will send the next chunk of the file?

A: The first chunk is sent when user selects the file to send which initiates the handshake process with the first zmodem packet. The server side is a dumb terminal so to speak, you tell it what you want and sends a response.
(I hope the formatting stays below)

Device side        Server side
send req / data        send response
============================================
zMTX 1(1-1096)        zmRX1 - req 4097
zMTX 4097(4097-8142)    zmRX4097 - req 8143

u thera?        yep
reservoir level #5    48%
pump state #13        On - 1200 L/s

Ive also taken a peek at the FTP Server concept you suggested, I have added the code but getting what appears to be permission issues on the directory the code points to. Will make a new thread on that one.

 

[MicroDrie]

I agree with the writers who do not fully understand what exactly you want. The advantage of a standard is that there are many standards. Now I have the idea that you want to use the originally serial ZMODEM protocol over TCP/IP with a device that works according to the ZMODEM protocol. In that case you will have to fully adhere to the ZMODEM protocol, including matters such things like coordination, error detection, re-transmission and handshake. Here you can find the scraymer / Zmodem-in-Java implementation on Github.

 

[RickV]

Yes zmodem-ish protocol. I think everyone has the zmdem part in focus and not the actual problem

Lets look at this

dim b() as byte
b = File.ReadBytes(File.DirInternal,fName)

This gets the whole file in an array of bytes - a very long string of bytes.
Lets get rid of zmodem out of our heads for a moment, Just look at the file I / O
Rather than get all the bytes in one chunk and move along the string of bytes with say mid(b, x, 4096) I want to open as a random access file and pull a record or string of bytes xo many characters wide,., poke it into my data packet and send it.

I already have all the other stuff, headers, checksums, failsafes etc. I just need to get the file data x bytes wide.

Back in VB6 days it was a case of
Dim Chunk1 as string *4096
Dim Chunk2 as string * 2048

b4x I have discovered
dim Chunk1(4096) as byte
dim Chunk2(2048) as byte

inVB6 we used SEEK to get to a current byte position with the file where we would GET a record, a RANDOM set of sequential bytes x bytes wide.
GET #1 , , Chunk1
GET #1 , , Chunk2

What the b4x equivalent to SEEK ?
How do I move the current position within the file to byte x ?
This is the predominate question. Not the protocol.