iOS Question read long text file

[Uniko Sistemi srl]

Good morning everyone.
I have a problem that only occurs with b4i.
I have a txt file that contains 3500 lines, in which I should search for an item that can be present multiple times. so I should find a method to search for all the desired items within this file. I tried readlist (but the file is too big and becomes slow)..can anyone give me some advice? A thousand thanks
unfortunately I have to use txt (customer need) and I can't use sql

 

[emexes]

I have a txt file that contains 3500 line ... but the file is too big and becomes slow

How many characters long is a typical line?

3500 lines x say 100 characters/line x 2 bytes/character = 700000 bytes = 0.7 MB = way less than a typical image file

Presumably ReadList (done once before searching) is lightning fast.

What do you call slow? How long does a typical search of those 3500 lines take?

Perhaps show us the search code, that presumably loops through the 3500 lines in the list?

 

[emexes]

I have a problem that only occurs with b4i.

This is interesting, though.

Do you have the same search code working in B4A or B4J but at an acceptable speed?

 

[Uniko Sistemi srl]

How many characters long is a typical line?

3500 lines x say 100 characters/line x 2 bytes/character = 700000 bytes = 0.7 MB = way less than a typical image file

Presumably ReadList (done once before searching) is lightning fast.

What do you call slow? How long does a typical search of those 3500 lines take?

Perhaps show us the search code, that presumably loops through the 3500 lines in the list?

60/65 char for line

for example, simply project :


[CODE lang="b4x" title="example"]
#if b4a
Dim fileindice As List = File.ReadList(File.DirInternal,"indicec.txt")

#else
Dim fileindice As List = File.ReadList(File.DirDocuments,"indicec.txt")
#End If
Dim nome As String
Dim nome2 As String
'listapdf = fileindice

For i = 0 To fileindice.Size-1
nome = fileindice.Get(i)
If nome.Contains(globe.tesseraperserver) And globe.tesseraperserver <> "" Then
globe.contindice = globe.contindice+1
Else
nome2=nome2
End If
Next [/CODE]
in this simple cycle, on Android it is very fast, on iOS it takes up to 6/7 seconds

I'm doing a simple thing, I'm looking for a certain number within the list that I created by reading the file.
what could I have done wrong?

 

[emexes]

example

[CODE lang="b4x" title="example"]
For i = 0 To fileindice.Size-1
nome = fileindice.Get(i)
If nome.Contains(globe.tesseraperserver) And globe.tesseraperserver <> "" Then
globe.contindice = globe.contindice+1
Else
nome2=nome2
End If
Next [/CODE]

Well, maybe I'm not seeing something, but the first two changes I'd make are:

1/ hoist the empty-string check out of the loop (should be ok, as long as you're not later relying on nome being the last list item)
2/ remove the nome2 = nome2 (unless this is a typo, maybe it's meant to be nome2 = nome)

[CODE lang="b4x" title="example"]
If globe.tesseraperserver <> "" Then
For i = 0 To fileindice.Size - 1
nome = fileindice.Get(i)
If nome.Contains(globe.tesseraperserver) Then
globe.contindice = globe.contindice + 1
End If
Next
End If [/CODE]

 

[emexes]

and then I'd check that the delay is indeed in this section of your looks-ok-to-me code, by adding:
[CODE lang="b4x" title="example" highlight="1,10,11"]
Dim StartTime As Long = DateTime.Now
If globe.tesseraperserver <> "" Then
For i = 0 To fileindice.Size - 1
nome = fileindice.Get(i)
If nome.Contains(globe.tesseraperserver) Then
globe.contindice = globe.contindice + 1
End If
Next
End If
Dim EndTime As Long = DateTime.Now
Log("Search took " & (EndTime - StartTime) & " milliseconds")
[/CODE]

 

[Uniko Sistemi srl]

and then I'd check that the delay is indeed in this section of your looks-ok-to-me code, by adding:
[CODE lang="b4x" title="example" highlight="1,10,11"]
Dim StartTime As Long = DateTime.Now
If globe.tesseraperserver <> "" Then
For i = 0 To fileindice.Size - 1
nome = fileindice.Get(i)
If nome.Contains(globe.tesseraperserver) Then
globe.contindice = globe.contindice + 1
End If
Next
End If
Dim EndTime As Long = DateTime.Now
Log("Search took " & (EndTime - StartTime) & " milliseconds")
[/CODE]

Result is : Search took 20150 milliseconds

unfortunately it is an unacceptable speed. who knows why on Android it's so fast instead.

 

[emexes]

Result is : Search took 20150 milliseconds

Yikes.

it is an unacceptable speed

for searching 3500 lines of 60..65 characters, I agree

next tack:
[CODE lang="b4x" title="example" highlight="5"]
Dim StartTime As Long = DateTime.Now
If globe.tesseraperserver <> "" Then
For i = 0 To fileindice.Size - 1
nome = fileindice.Get(i)
If i < 10 Then Log(fileindice.size & Tab & i & Tab & (DateTime.Now - StartTime) & Tab & "[" & nome & "]")
If nome.Contains(globe.tesseraperserver) Then
globe.contindice = globe.contindice + 1
End If
Next
End If
Dim EndTime As Long = DateTime.Now
Log("Search took " & (EndTime - StartTime) & " milliseconds")
[/CODE]

 

[Uniko Sistemi srl]

Yikes.


for searching 3500 lines of 60..65 characters, I agree

next tack:
[CODE lang="b4x" title="example" highlight="5"]
Dim StartTime As Long = DateTime.Now
If globe.tesseraperserver <> "" Then
For i = 0 To fileindice.Size - 1
nome = fileindice.Get(i)
If i < 10 Then Log(fileindice.size & Tab & i & Tab & (DateTime.Now - StartTime) & Tab & "[" & nome & "]")
If nome.Contains(globe.tesseraperserver) Then
globe.contindice = globe.contindice + 1
End If
Next
End If
Dim EndTime As Long = DateTime.Now
Log("Search took " & (EndTime - StartTime) & " milliseconds")
[/CODE]

results : Search took 40422 milliseconds

it got worse :-(

 

[emexes]

the only potential difference I can imagine is that the string handling is different between B4A and B4I

perhaps it is creating and allocating new memory each loop at the nome = fileindice.Get(i) line

and then maybe it is even slower in debug mode

so perhaps try:

[CODE lang="b4x" title="example" highlight="4-7"]
Dim StartTime As Long = DateTime.Now
If globe.tesseraperserver <> "" Then
For i = 0 To fileindice.Size - 1
''' nome = fileindice.Get(i)
''' If i < 10 Then Log(fileindice.size & Tab & i & Tab & (DateTime.Now - StartTime) & Tab & "[" & nome & "]")
''' If nome.Contains(globe.tesseraperserver) Then
If fileindice.Get(i).As(String).Contains(globe.tesseraperserver) Then
globe.contindice = globe.contindice + 1
End If
Next
End If
Dim EndTime As Long = DateTime.Now
Log("Search took " & (EndTime - StartTime) & " milliseconds")
[/CODE]

 

[Uniko Sistemi srl]

the only potential difference I can imagine is that the string handling is different between B4A and B4I

perhaps it is creating and allocating new memory each loop at the nome = fileindice.Get(i) line

and then maybe it is even slower in debug mode

so perhaps try:

[CODE lang="b4x" title="example" highlight="4-7"]
Dim StartTime As Long = DateTime.Now
If globe.tesseraperserver <> "" Then
For i = 0 To fileindice.Size - 1
''' nome = fileindice.Get(i)
''' If i < 10 Then Log(fileindice.size & Tab & i & Tab & (DateTime.Now - StartTime) & Tab & "[" & nome & "]")
''' If nome.Contains(globe.tesseraperserver) Then
If fileindice.Get(i).As(String).Contains(globe.tesseraperserver) Then
globe.contindice = globe.contindice + 1
End If
Next
End If
Dim EndTime As Long = DateTime.Now
Log("Search took " & (EndTime - StartTime) & " milliseconds")
[/CODE]

I understand what you mean, but unfortunately it responds like this: Search took 20110 milliseconds

 

[emexes]

it got worse :-(

? sorry

but yes, because of the new extra ten Log lines, probably (although an extra 2 seconds per Log line does seem extraordinary)

can you post the ten Log lines?

 

[Uniko Sistemi srl]

? Scusa

ma sì, a causa delle nuove dieci righe di log extra, probabilmente (anche se 2 secondi in più per riga di log sembrano straordinari)

puoi pubblicare le dieci righe di registro?

3398 0 17 [1904302_cp_181023_310124_02_010_MM_020_Sconto Benvenuto.txt]
3398 1 27 [4008239_cp_181023_310124_02_010_MM_020_Sconto Benvenuto.txt]
3398 2 42 [4602000_cp_181023_310124_02_010_MM_020_Sconto Benvenuto.txt]
3398 3 55 [4602001_cp_181023_310124_02_010_MM_020_Sconto Benvenuto.txt]
3398 4 67 [4602002_cp_181023_310124_02_010_MM_020_Sconto Benvenuto.txt]
3398 5 79 [4602003_cp_181023_310124_02_010_MM_020_Sconto Benvenuto.txt]
3398 6 90 [4602004_cp_181023_310124_02_010_MM_020_Sconto Benvenuto.txt]
3398 7 103 [4602005_cp_181023_310124_02_010_MM_020_Sconto Benvenuto.txt]
3398 8 115 [4602006_cp_191023_190124_02_010_MM_020_Sconto di Benvenuto.txt]
3398 9 127 [4602007_cp_181023_310124_02_010_MM_020_Sconto Benvenuto.txt]

 

[emexes]

Amazing. ?

Dumb question, but better to ask than assume:

globe.tesseraperserver is just a global string variable?

(ie not a Sub / function call)

 

[emexes]

And is it as slow in Release mode as it is in Debug mode?

You might need to comment out the Log calls so that your app runs properly in Release mode.

 

[Uniko Sistemi srl]

Amazing. ?

Dumb question, but better to ask than assume:

globe.tesseraperserver is just a global string variable?

(ie not a Sub / function call)

yes, exactly, it is a global variable that is needed for the entire project. the speed is identical even in the release version unfortunately

 

[emexes]

If it was me, I would write a little test program that only reads the file and then does a few searches.

Yes, that is clutching at straws.

But the better you can pin the problem down, to fewer lines, the quicker Erel can tell us what the heck is happening.

It might be that the little test program works much faster.

Is there any background processing going on in your big program? Like CPU time being chewed up receiving data via HTTP or FTP or BLE communications?

 

[emexes]

Is your program updating any status information on screen during the search, eg a live count of how many matches found, or how many records searched?

It might be that Android only redraws the screen pixels once per frame ie 60 times a second, and that iOS might redraw the screen pixels each time you update the status value, even if the value is the same as before.

Again, still clutching at straws ? if anybody else has better ideas, then feel free to join in ?

 

[emexes]

3398 0 17 [1904302_cp_181023_310124_02_010_MM_020_Sconto Benvenuto.txt]
3398 1 27 [4008239_cp_181023_310124_02_010_MM_020_Sconto Benvenuto.txt]
3398 2 42 [4602000_cp_181023_310124_02_010_MM_020_Sconto Benvenuto.txt]
3398 3 55 [4602001_cp_181023_310124_02_010_MM_020_Sconto Benvenuto.txt]
3398 4 67 [4602002_cp_181023_310124_02_010_MM_020_Sconto Benvenuto.txt]
3398 5 79 [4602003_cp_181023_310124_02_010_MM_020_Sconto Benvenuto.txt]
3398 6 90 [4602004_cp_181023_310124_02_010_MM_020_Sconto Benvenuto.txt]
3398 7 103 [4602005_cp_181023_310124_02_010_MM_020_Sconto Benvenuto.txt]
3398 8 115 [4602006_cp_191023_190124_02_010_MM_020_Sconto di Benvenuto.txt]
3398 9 127 [4602007_cp_181023_310124_02_010_MM_020_Sconto Benvenuto.txt]

hang on...

at worst, adding those extra ten log lines should have taken an extra 140 milliseconds, not an extra 20000 milliseconds

wtf?

what happens if we make it twenty extra log lines, ie i < 20 instead of i < 10 ?

does that add yet another 20000 milliseconds ie about 60000 milliseconds (60 seconds) in total?

 

[Uniko Sistemi srl]

Is your program updating any status information on screen during the search, eg a live count of how many matches found, or how many records searched?

It might be that Android only redraws the screen pixels once per frame ie 60 times a second, and that iOS might redraw the screen pixels each time you update the status value, even if the value is the same as before.

Again, still clutching at straws ? if anybody else has better ideas, then feel free to join in ?

So I tried to evaluate it again in release mode with your latest suggestions and damn, the problem does NOT arise!! amazing! why did it cause me so many problems while debugging?? I am tired ?