B4J Question (Solved) OData query, reading large json files *out of memory*

[Yafuhenk]

I want to create the next generation of Easy4Sales in B4X (currently VB.Net) Easy4Sales is a tool for sales managers that creates reports and overviews.
A database holds all the data. Data is imported via SAP Odata queries. The downloaded Json files can be huge!

A download procedure based on Httpjob gives an out of memory error when I import one fiscal year. I read on the forum that in such cases I have to send several OData requests instead of one, but that is not what I want, since in VB.Net is works perfect. Can someone give me some hints how I can convert the following vb.net snippet to B4X?

Thanks in advance

Dim buffer(4096) As Byte
Dim request As WebRequest
request = WebRequest.Create(odataquerysales)
request.Credentials = New NetworkCredential("username", "password")
request.Timeout = 300000
Dim response As WebResponse = request.GetResponse()
Dim bytesRead As Integer
Dim totalBytesRead As Long
Dim bytesToDownload As Long = response.ContentLength
Using dataStream As Stream = response.GetResponseStream()
    Using fs As New FileStream("c:\myfile.json", FileMode.Create)
    Do
        bytesRead = dataStream.Read(buffer, 0, buffer.Length)
        totalBytesRead += bytesRead
        If bytesRead > 0 Then
            fs.Write(buffer, 0, bytesRead)
        End If
    Loop While bytesRead > 0
    End Using
End Using

 

[Erel]

Please post the full error message and the relevant B4X code.

What is the size of the downloaded json string?

 

[Erel]

This code doesn't parse the string. It simply writes it to a file. My guess is that you are calling Job.GetString for no reason. Copy the file with File.Copy2 and Job.GetInputStream

 

[Yafuhenk]

Thanks Erel, your guess was right.

The following code works fine. I downloaded a file of 260 mb

Sub ImportData(tablename As String, fromdate As String, todate As String)
    'In order to make this work add "HU2_ACCEPTALL" to Build Configurations (CTRL+B)
    Dim job As HttpJob
    Dim odata As String
    job.Initialize("", Me)
    job.Username = "username"
    job.Password = "password"
    job.GetRequest.Timeout = 300000
    Select Case tablename
        Case "Sales"
            odata = settings.Get("odata_import_sales")
            odata = odata.Replace("INSERT_FROM_DATE_HERE", fromdate)
            odata = odata.Replace("INSERT_TO_DATE_HERE", todate)
            Log(odata)
    End Select
    Log("Request download " & tablename & ".")
    job.Download(odata)
    Wait For (job) JobDone(job As HttpJob)
    If job.Success = True Then
        File.Copy2(job.GetInputStream, File.OpenOutput(xui.DefaultFolder, tablename & "_" & fromdate & "_ " & todate &  ".json", False))
        Log(tablename & " downloaded.")
    Else
        Log("Something went wrong during download.")
    End If
End Sub

 

[Erel]

The following code works fine.

It doesn't. You aren't closing the output stream.

 If job.Success = True Then
         Dim out As OutputStream = File.OpenOutput(xui.DefaultFolder, tablename & "_" & fromdate & "_ " & todate &  ".json"
        File.Copy2(job.GetInputStream, out, False))
        out.close
        Log(tablename & " downloaded.")
    Else

 

[Yafuhenk]

The file was not closed indeed. Now it works. Thanks