B4R Question Is there a max on AsyncStream prefix mode object array?

[miker2069]

I have question regarding using AsyncStream in Prefix mode. If I have an array of objects say 200 elements, that I serialize to a byte array and then send to a B4J app, on the B4J app I seem to hit array bound index issues when my program enumerates the deserialized object array in the B4J app. At 144 I hit arraybound errors. if I send less than that, then all is well.

I'm thinking it's on the B4R side. I'm using the B4RSerializator .ConvertArrayToBytes2 sub to serialize the array. My buffer array used in arg 2 I believe is big enough (for testing I set it to an array of 2400 bytes). I also increased my stack size to 8000. Is there a max on the number of elements that can be serialized and sent from the B4R side?

 

[Erel]

Try to convert the returned array of bytes to objects with ConvertBytesToArray, without sending anything. My guess that it will work.

The bottleneck is probably in the network buffer. You can split the large array of bytes with ByteConverter.Substring.

 

[miker2069]

It looks like I can deserialize on the device side so I suspect you're right it's a limit in the network buffer perhaps? I can split it up with multiple astream.write calls.

On a side note and somewhat related, it would be nice to have a version of B4RSerializator.ConvertArrayToBytes2 that converts "up to some number of elements in the object array. So if I have an object array with a max size of 200, but I'm only using the first 100 elements it would be more efficient to only convert what I need and not the non-used elements. I will put it in as a wish.

Thanks!

 

[miker2069]

Looks like the maxbuffer is set to 500 bytes looking at the AsyncStreams::InitializePrefix in AsyncStreams.cpp. My byte stream is 1200, so yeah that would be a problem . I suspect I could cheat and bump this up on my end and recompile (obviously would break when the library is updated by subsequent updates)...

    void AsyncStreams::InitializePrefix (B4RStream* Stream, bool BigEndian, SubVoidArray NewDataSub, SubVoidVoid ErrorSub) {
        Initialize(Stream, NewDataSub, ErrorSub);
        prefixMode = true;
        bigEndian = BigEndian;
        MaxBufferSize = 500;
        WaitForMoreDataDelay = 300;
    }

 

[Erel]

I could cheat and bump this up on my end and recompile

No need. It is exposed as a field.

However MaxBufferSize is not used when you write data, only for reading.

Go with splitting. It is very simple in B4R side. In B4J side you should create a large array and copy the new data to this array with ByteConverter.ArrayCopy.

 

[miker2069]

Okay I am splitting my byte payload on the B4R side, with something like this:

    Dim idx As Int
    Dim payload_len As Double
    Dim count As Int
    Dim max_chunksize As Double = 400
    Dim left_to_send As Double = byte_buffer.Length

    idx = 0
    Do Until idx >=byte_buffer.Length 'byte_buffer is the originally converted obj array
        count = count + 1
        payload_len = Min(max_chunksize,left_to_send)
        Astream.Write2(byte_buffer,idx,payload_len)
        server.Socket.Stream.Flush
        idx = idx + payload_len
        left_to_send = left_to_send - payload_len
        Log("Sent Payload chunk  ",count, " payload chunk length bytes: ",payload_len, "  remaining: ", left_to_send)
    Loop

On my B4J side I am receiving it like so:

    Dim idx As Int = 0
    Dim payload(payload_len) As Byte 'payload_len was received earlier..assume it's 1200 bytes

    idx = 0
    Do Until idx >=payload_len
        Wait For astreams_NewData (Buffer() As Byte)
        UpdateStatus("Received payload data...Buffer Length:" & Buffer.Length)
        bc.ArrayCopy(Buffer,0,payload,idx,Buffer.Length)
        idx = idx + Buffer.Length
    Loop

    Dim obj_buffer2() As Object = ser.ConvertBytesToArray(payload)
    Dim num_codes As Int = obj_buffer2(0) 'number of codes is always first element

    'TxtNumCodes.Text = NumberFormat(num_codes,0,0)
    UpdateStatus("num_codes:  " & num_codes)
    'TxtIRCodeStr.Text = FlatArray(obj_buffer,num_codes,",")
    ret_ircode.IRCode = FlatNumberArray(obj_buffer2,num_codes,",")
    ret_ircode.num_codes = num_codes

    Return ret_ircode

The number of elements in the obj_buffer2 is 200, yet enumerating through I hit the error in exact same spot as before this technique. When enumerating it, it stops at element 143 with an array index error. Very strange, since I seem to successfully get the byte stream chunks and combine them. Odd that I'm still hitting an array issue on the B4J side.

 

[miker2069]

For the time being I've worked around this particular issue, with the following alternate code:

    Log("Obj Buffer length: ",Main.code_array.Length)
    'Dim byte_buffer() As Byte = ser.ConvertArrayToBytes(Main.code_array)
   
    'copy 100 elements at a time and send
    Dim i As Int = 0
    Dim c_idx As Int
    Dim count As Int
   
    'send status and number of objects
    Dim status_buffer() As Byte = ser.ConvertArrayToBytes(Array As Object("SUCCESS",Main.code_array.Length))
    Astream.Write(status_buffer)
    server.Socket.Stream.Flush
   
    c_idx = 0
    'copy 100 elements at a time
    Do Until c_idx >= Main.code_array.length
        ClearTmpCodeArray
        count = count + 1
        Log("Write obj set ", count)
        For i = 0 To tmp_obj_buf.Length - 1
            Main.bc.ObjectSet( Main.code_array(c_idx),tmp_obj_buf(i))
            c_idx = c_idx + 1
        Next
        'now send this chunk
        Dim byte_buffer() As Byte = ser.ConvertArrayToBytes2(tmp_obj_buf,Main.gbl_code_str) 'local_buffer)
        Log("byte buffer length: ", byte_buffer.Length)
        Astream.Write(byte_buffer)
        server.Socket.Stream.Flush
   
    Loop
   
    CallSubPlus("CloseConnection", 100, 0)
    Return

Instead of splitting the byte stream, I'm splitting the original object array into 100 element units (I'm copying 100 elements into a tmp obj array and serialize and send that). On the other B4J end I'm just aseembling the obj arrays into a bigger array and this works.

It's still strange why the original workaround produced the same array out of bounds error. I wonder if I'm still going beyond the network send buffer limit (I tried the original workaround with only sending 100 bytes at a time and still had an issue). Strange.