Android Question How to solve "android.os.NetworkOnMainThreadException" error

[DanteS]

I am trying, since a few weeks, to develop a B4A application to consult the real time price of stocks. The best results have been obtained using the TestjSoup project published by Erel. I modified the code under the btnLoadFromURL button to connect the Yahoo page first and Zacks page finally, and the process runs very well. I added an edittextbox (tx_tkt) to enter the ticket code of the stock to be consulted and three labels to show the three values generally desired: price, day change and % of change.
When I click the btnLoadFromURL button, the program sends the url and the ticket stock code storaged in the tx_tkt Edittextbox as a parameter and the page responds sending all the text contained in it. With a Function (ExtractValues), created by me some years ago in vb.net, converted to B4A , I scrape the text and fill the three response labels.
I have had two main problems: the Yahoo page detects that I am not a regular user and sends a correct response but with old data and the second one: when I create an executable version, using release insted of debug, I receive the error: android.os.NetworkOnMainThreadException, and the page did not connect. For the first problem I decided to run away, because I supposed that I have nothing to do, to solve it. Better, I went to Zacks page. For the second one, I would like to hear solutions and sugestions from you.

The code used to call the page is

Private Sub btnLoadFromURL_Click
    Dim x As String
    Dim n As Int

    url = "https://www.zacks.com/stock/quote/" & tx_tkt.Text.ToUpperCase
    n=0
   
    Try
        x=js.connect(url)
        n=x.Length
    Catch
        tx_texto.Text=LastException
        x=""
    End Try
       
    n=x.IndexOf("<header>")                        'beguining of the price block of text                          
    If n>0 Then
        x=x.SubString(n)
        n=x.IndexOf("change")                    'end of the price block of text
        If n>0 Then
            x=x.SubString2(0,n+200)
            tx_texto.Text=x
            ExtractValues(tx_texto.text)
        End If
    End If
End Sub

 

[DonManfred]

tx_tkt.Text.ToUpperCase

what is the value here?
Mean: what is the url you are calling?

I can´t see how a NetworkOnMainthread exception can happen here. You code is INCOMPLETE

Have you tried to use okhttputils to download the page and work with the result? It will not raise any such Exception

Does zocks.com have an API? If yes then use the Api.

 

[drgottjr]

network i/o involves, among others, the possibility of delay in transmission. in other words, a situation over which
you have no control. such delays are anathema to android and may cause the so-called ANR failure because the
main thread (your app) is frozen waiting for the operation to finish. as a result, network i/o runs on a separate thread.
if it's successful, you get a result when one is available. if it fails, its failure is independent of the app's running (unless
you make assumptions in your app regarding the success of the operation, ie, assuming it was successful when it wasn't).
this allows your app to do other things, if applicable, until the network call has finished. now, i realize it may not be
your intention to do any other things while you wait for a result, but that's not the way android sees things.

network libraries made available to us have their network i/o run on a separate thread. that's why libraries such as
okhttputils2 work. otherwise, they too would crash like your app. if you're going to carry out network i/o yourself,
you need to run those ops on a separate thread.

in real life many network routines can run on the main thread without incident. but out of an abundance of caution,
android does not permit it unless overridden. if you override android's default behavior, you run the risk of
failure, not only due to timing issues, but as a result of programming errors (network protocols are demanding and
must be followed strictly). it's one thing to write a bad application for personal use; it's another to expect google to
distribute such an app on play.

you don't show how you make your call to zacks, but if you were using okhttputils2 - as urged by @DonManfred - you
wouldn't see the error you see now.

 

[DanteS]

Thanks DonManfred.

This are my answers to your questions.

1. The url is https://www.zacks.com/stock/quote/
The page needs that you paste at the end of the url the ticket code of the stock (MA for Master Card, AAPL for Apple, TSLA for Tesla, etc.)
This ticket code is stored by the user in the edittext box tx_tkt. .ToUpperCase only convert the code to UpperCase.

2. The code shown is the code where the error appears. The rest of the code (the ExtractValues function) enter in escene after the response of the page, so it don´t participates in the error.

3. No, I am not used okhttputils in this app.

4. No, Zacks.com don´t use API

Please, take into account that the error appears only in compiled mode. When I use it in debug mode it runs OK.

 

[DanteS]

Thanks drgotjr

Yes, I show the connection code:
x=js.connect(url)
I suppose that js is a method of the jSoup library. Erel uses in his project the libraries jSoup, B4XPages, and Code

Please, take into account that the error appears only in compiled mode. When I use it in debug mode it runs OK.

 

[Star-Dust]

I have had two main problems: the Yahoo page detects that I am not a regular user and sends a correct response but with old data

Most likely the problem is that you have not set UserAgent and therefore it recognizes that it is not a browser. Try this string in the UseraAgent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:56.0) Gecko/20100101 Firefox/56.0

For the rest, as they advised you, use okHttp (or okhttputils) and you will solve the problem in release mode

 

[DonManfred]

Please, take into account that the error appears only in compiled mode.

What kind of Object is js which is used to Connect?

It it always better if you create a small project showing the problem and Upload it.

 

[DanteS]

What kind of Object is js which is used to Connect?

Hi Don
I suppose that js is a method of the jSoup library, used in his project by Erel
I used it exactly as he used it in the code bihind the button to call an url

It it always better if you create a small project showing the problem and Upload it.

Yes, I am going to take it into account.

In this case I suppose that as the error appears in the button code, it was enough to show this part.

 

[DanteS]

Try this string in the UseraAgent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:56.0) Gecko/20100101 Firefox/56.0

Hi Star-Dust
Thanks for you answer
Do you mean that I need to declare and configure an UserAgent in some part of the project? If affirmative, can you show me where and how to insert the string in it?

For the rest, as they advised you, use okHttp (or okhttputils) and you will solve the problem in release mode

Yes, I am clear now that it is necesary to use this libraries. But is it enough only to include the library in the project or I need to use too a method of this instead of the js.connet(url)? that I am using now

 

[Star-Dust]

Hi Star-Dust
Thanks for you answer
Do you mean that I need to declare and configure an UserAgent in some part of the project? If affirmative, can you show me where and how to insert the string in it?

Yes, I am clear now that it is necesary to use this libraries. But is it enough only to include the library in the project or I need to use too a method of this instead of the js.connet(url)? that I am using now

You can set UserAgent with okHttpUtils.
You will no longer need to download the html code with js.connect because you will do it with okHttutils.

If you need to parse the html code you can also use the Erel library

JTidy library - Convert HTML pages to XML

This library wraps JTidy open source project. It is supported by B4A and B4J. It allows you to convert a HTML page to XHTML page. XHTML can be parsed with a XML parser. This approach is better than trying to parse HTML with regular expressions. Usage is simple: Sub Process_Globals Dim sax...

www.b4x.com

 

[JohnC]

ChatGPT sums up what the other users in this thread are saying:

For the issues you're experiencing with your B4A (Basic4Android) application, let's tackle them one by one.

Issue 1: android.os.NetworkOnMainThreadException​

This error occurs because your app is performing a network operation (fetching data from the internet) on the main thread of the application. Android does not allow network operations on its main thread to prevent the UI from becoming unresponsive. The solution is to run these network operations on a separate thread. Here’s how you can resolve this using B4A:

Using Threads​

B4A provides support for running operations in the background using threads. Here's how you can modify your btnLoadFromURL_Click method to utilize threads and avoid the NetworkOnMainThreadException:

Private Sub btnLoadFromURL_Click
    Dim Job1 As HttpJob
    Job1.Initialize("Job1", Me)
    Job1.Download("https://www.zacks.com/stock/quote/" & tx_tkt.Text.ToUpperCase)
End Sub

'Sub that gets called when the Job is done
Sub JobDone(Job As HttpJob)
    If Job.Success Then
        Dim response As String
        response = Job.GetString
        ' Process the response string here
        ExtractAndDisplayValues(response)
    Else
        ToastMessageShow("Error: " & Job.ErrorMessage, True)
    End If
    Job.Release
End Sub

Private Sub ExtractAndDisplayValues(response As String)
    Dim n As Int
    n = response.IndexOf("<header>")  ' Beginning of the price block of text
    If n > 0 Then
        response = response.SubString(n)
        n = response.IndexOf("change")  ' End of the price block of text
        If n > 0 Then
            response = response.SubString2(0, n + 200)
            tx_texto.Text = response
            ExtractValues(tx_texto.text)
        End If
    End If
End Sub

In this example:

• HttpJob is a class available in B4A that handles HTTP requests asynchronously.
• JobDone is a callback that gets executed once the HTTP request is complete.
• You should replace ExtractValues(tx_texto.text) with your logic inside ExtractAndDisplayValues to ensure it runs after the data is fetched.

Issue 2: Yahoo Page Detects Unusual Traffic​

Scraping websites like Yahoo Finance can often lead to issues like being blocked or receiving outdated data because these sites can detect scraping behavior and restrict access. Here are a couple of strategies:

1. User-Agent String: Sometimes, setting a user-agent in your HTTP request that mimics a standard web browser can help avoid detection. You can set this in your HttpJob as follows:
   
   Job1.GetRequest.SetHeader("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36")
2. Rate Limiting: Ensure your requests are spaced out to mimic human browsing patterns rather than rapid, automated access.
3. API Usage: Consider using an official API if available, which would be more reliable and legal compared to scraping web pages. For stock prices, services like Alpha Vantage, IEX Cloud, or even a direct API from Yahoo Finance, if available, might be more appropriate and stable.

Lastly, always ensure that your methods of data retrieval comply with the terms of service of the website you are accessing to avoid any legal issues.

 

[DanteS]

Thanks JohnC

It looks very well.
Is it necesary to add some library?

 

[JohnC]

Thanks JohnC

It looks very well.
Is it necesary to add some library?

I think just the OKHttpUtils2 lib

 

[DanteS]

Thanks to all that sugested, insisted and demmanded the use of the okHtmlUtils. Specially JohnC, who sugested part of the code itself.

I replaced the jSoup library by the okHtmlUtils library. Then, under the button code, I used the method Job1.Download to call the page and inside the Job.success subrutine, I received the text response with the method Job.GetString. The rest remained equal.
The project was compiled using release and runs cool