Android Question Http digest authorization fails in B4A

fs007

Member
I need to write an app which sends a HTTP GET to devices named "Shelly" (these are so called "SmartSwitches, shelly.com).
They need digest authorization as described here: https://shelly-api-docs.shelly.cloud/gen2/General/Authentication/
My code (which does NOT work) is:
B4X:
Dim hj As HttpJob
hj.Initialize("",Me)
hj.Username="admin"
hj.Password="password"
hj.Download("http://192.168.5.188/relay/0?turn=on")
Wait for (hj) JobDone(hj As HttpJob)
If hj.Success Then
    label1.Text = hj.GetString
Else
    label1.Text ="Error"
End If

I get an error "Unauthorized", but the credentials are correct. What's wrong here ???
 
Solution
what would be the proper format for sha-256 digest auth ? code snippet would be nice
Try the code below. I've tested it against https://httpbin.org/digest-auth/auth/test/test3/SHA-256 and it seems to work. The code is not all-encompassing regarding the RFC spec and only handles GET requests, but hopefully, it implements enough to work for your needs.

Usage example:

B4X:
    Wait For(AuthenticateUrl("https://httpbin.org/digest-auth/auth/test/test3/SHA-256", "test", "test3")) Complete (j As HttpJob)
    If j.Success = True
        'successfully authenticated
        Log(j.GetString)
    Else
        Log(j.ErrorMessage)
    End If

The implementation code:

B4X:
'See...

aeric

Expert
Licensed User
Longtime User
thanks for trying it. you know, i missed this way back: httpjob. you're using
httpjob, not okhttpjob. you can dump my mod. the original apache http.jar (which
we haven't used in years) handles digest authentication differently than okhttp.jar.
when used with okhttputils2, my mod is called. sorry for the waste of your time.

the last thing i'll have to say in this matter has to do with whether or not the device
has your username (at least) stored. according to rfc2617, that has to be the case
for digest authentication to work. if you (the server) send me a nonce and i use it
to create a hash of my username and password and send the hash back to you, it
serves no purpose if you don't already have my username and password. if you
already have my username and password stored, you can perform the same operation
with your nonce and compare the result with what i send you. if you don't already
have the username and password, you have nothing to compare my hash with.
The OP is not calling a RPC or HTTP RPC (POST) API call, just a normal GET HTTP REST API call. So nonce or hashing are not required.
He does have the username and password set up because he is successfully called the API through web browser (GET request) with username and password.
That works. Relay is switched. Confirmed with Google Chrome
 
Upvote 0

aeric

Expert
Licensed User
Longtime User
Maybe here are my last questions.
1. Are you owning the switch and it is set up by you from the box?
2. Can you share the screenshot of the web UI? I just want to check if there is a menu for restrict login.
3. Can you share the screenshot of the Chrome browser to show the URL that is is not auto redirect to https secure protocol. Just want to check in case someone set up this switch before with SSL. However, I think this is unlikely.
4. Can you can show the web developer console by pressing Ctrl + Shift + J (or right-click the page and Inspect) and choose Network tab. In the list of resources, click the url endpoint. Check the Response Headers on the right. There may be some clues.

1706717542650.png


Extra note:
I have thought of why not I purchase one switch and try out. I look at 2 e-commerce sites that ship to my place. The price of Shelly+ 2M is MYR200+ while 1M is half the price. It is more cheaper to get a Tuya switch which is 8 times cheaper. So I decided not to proceed.
 
Upvote 0

fs007

Member
Maybe here are my last questions.
1. Are you owning the switch and it is set up by you from the box?
Yes. I own 3 Shelly plus 2PM

2. Can you share the screenshot of the web UI? I just want to check if there is a menu for restrict login.
The UI of these devices consists of dozens pages with 100+ settings. They are much more sophisticated than those cheap Thuya switches. They are even script-able (some sort of Javascript) ... but i can tell you: there is no such thing as "restricted login"

3. Can you share the screenshot of the Chrome browser to show the URL that is is not auto redirect to https secure protocol.
one can activate TLS or deactivate it. I have it deactivated and that is the default setting btw. see the attached screenshot.

4. Can you can show the web developer console by pressing Ctrl + Shift + J
See below. What do you want to learn from this ???

Tuya switch which is 8 times cheaper
A Shelly plus 1 costs about € 15 which is roughly US-$ 15. If a "Tuya switch" is 8 times cheaper, i can guess the quality.
 

Attachments

  • screenshot.png
    screenshot.png
    159.4 KB · Views: 33
Upvote 0

aeric

Expert
Licensed User
Longtime User
See below. What do you want to learn from this ???
I forgot to mention. When you are on the Network tab, you need to refresh the page again.
Do this with the url you are calling. i.e http://admin:[email protected]/relay/0?turn=toggle

A Shelly plus 1 costs about € 15 which is roughly US-$ 15. If a "Tuya switch" is 8 times cheaper, i can guess the quality.
Honestly, I have no experience with this home automation thing. I am afraid of electricity shock. :oops:
If I am going to start, I will first go for the one where my pocket doesn't hurt.

Just for clarification, are you confirmed that the 3rd option does not work?
B4X:
Private Sub CallRelayEndPoint
    Dim hj As HttpJob
    hj.Initialize("", Me)
    
    ' Method 1
    'hj.Username = "admin"
    'hj.Password = "password"
    'hj.Download("http://192.168.5.188/relay/0?turn=toggle")
    
    ' Method 2
    'hj.Download("http://192.168.5.188/relay/0?turn=toggle")
    'hj.GetRequest.SetHeader("Authorization", "Basic " & SU.EncodeBase64("admin:password".GetBytes("UTF8")))
    
    ' Method 3
    hj.Download("http://admin:[email protected]/relay/0?turn=toggle")
    
    Wait For (hj) JobDone(hj As HttpJob)
    If hj.Success Then
        Log(hj.GetString)
    Else
        Log(hj.ErrorMessage)
    End If
    hj.Release
End Sub
 
Upvote 0

OliverA

Expert
Licensed User
Longtime User
I forgot to mention
Hi @aeric. You are so close. Your code in post #9 has most of the ingredients if you want to take a stab at solving this issue w/o waiting for the library to be updated. Here are the steps:

1) Do a get to the URL you want to access
2) The above request should return a status code of 401. Read the fields out of the WWW-Authenticate header. Make sure to check the algorithm it is asking for. In this case it should be SHA-256
3) Use the information to create all the pieces needed to put back into an Authorization header field
4) Do another get to the Same URL, setting an Authorization header with the parts from step #3
5) You should be good to go (as long as the Authorization header is properly formatted for the algorithm that is seen in step #2)

No JSON, not POST, just 2 GETs properly executed. Do not set the username and password for either GET or you will trigger the build in MD5 response of the underlying OkHttpClientWrapper class.

Links:
Your post #9: https://www.b4x.com/android/forum/threads/http-digest-authorization-fails-in-b4a.158831/post-975228
Some more info on digest authorization with sample header fields: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/WWW-Authenticate
 
Upvote 0

OliverA

Expert
Licensed User
Longtime User
I forgot to mention
Hi @aeric. You are so close. Your code in post #9 has most of the ingredients if you want to take a stab at solving this issue w/o waiting for the library to be updated. Here are the steps:

1) Do a get to the URL you want to access
2) The above request should return a status code of 401. Read the fields out of the WWW-Authenticate header. Make sure to check the algorithm it is asking for. In this case it should be SHA-256
3) Use the information to create all the pieces needed to put back into an Authorization header field
4) Do another get to the Same URL, setting an Authorization header with the parts from step #3
5) You should be good to go (as long as the Authorization header is properly formatted for the algorithm that is seen in step #2)

No JSON, not POST, just 2 GETs properly executed. Do not set the username and password for either GET or you will trigger the build in MD5 response of the underlying OkHttpClientWrapper class.

Links:
Your post #9: https://www.b4x.com/android/forum/threads/http-digest-authorization-fails-in-b4a.158831/post-975228
Some more info on digest authorization with sample header fields: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/WWW-Authenticate
 
Upvote 0

fs007

Member
Just for clarification, are you confirmed that the 3rd option does not work?
3rd option does not work, as well as all other proposed code in this thread did not work so far.

as long as the Authorization header is properly formatted for the algorithm
what would be the proper format for sha-256 digest auth ? code snippet would be nice !

waiting for the library to be updated
is it known, when this will happen ?
 
Upvote 0

OliverA

Expert
Licensed User
Longtime User
No JSON doesn't mean it cannot be a POST
I meant to say just keep it simple. For the OP an operational GET is all that is needed. Nothing else fancy (besides the digest)
 
Upvote 0

OliverA

Expert
Licensed User
Longtime User
what would be the proper format for sha-256 digest auth ? code snippet would be nice
Try the code below. I've tested it against https://httpbin.org/digest-auth/auth/test/test3/SHA-256 and it seems to work. The code is not all-encompassing regarding the RFC spec and only handles GET requests, but hopefully, it implements enough to work for your needs.

Usage example:

B4X:
    Wait For(AuthenticateUrl("https://httpbin.org/digest-auth/auth/test/test3/SHA-256", "test", "test3")) Complete (j As HttpJob)
    If j.Success = True
        'successfully authenticated
        Log(j.GetString)
    Else
        Log(j.ErrorMessage)
    End If

The implementation code:

B4X:
'See https://github.com/AnywhereSoftware/B4A/blob/b22087de45d1556b105b71c36a6555243ddf6211/Libs_OkHttp/src/anywheresoftware/b4h/okhttp/OkHttpClientWrapper.java#L265
'See https://www.b4x.com/android/forum/threads/http-digest-authorization-fails-in-b4a.158831/post-975228
Public Sub AuthenticateUrl(Url As String, Username As String, Password As String) As ResumableSub
    Dim j As HttpJob
    Dim initialCall As HttpJob
    initialCall.Initialize("",Me)
    initialCall.Download(Url)
    Wait For (initialCall) JobDone(initialCall As HttpJob)
    If initialCall.Success = False And initialCall.Response.StatusCode = 401 Then
        Log(initialCall.Response.GetHeaders)
        Dim headers As List = initialCall.Response.GetHeaders.Get("www-authenticate") ' looks like all headers are lowercased when retrieved
        If headers <> Null And headers.Size <> 0 Then
            If headers.Size > 1 Then Log($"[WARNING] URL (${Url}) returned multiple WWW-Authenticate challenges. Processing only first challenge entry"$)
            Dim header As String = headers.Get(0)
            Log($"[Debug] WWW-Authenticate header: ${header}"$)
            If header.StartsWith("Digest") Then
                Dim directives As String = header.SubString("Digest".Length+1)
                Log($"[Debug] WWW-Authenticate directives: ${directives}"$)
                Dim retVals() As String = HandleDigest(initialCall.Response, directives, Username, Password)
                If retVals(1) <> "" Then
                    j = initialCall
                    j.ErrorMessage = retVals(1)
                Else
                    initialCall.Release
                    Dim authenticationCall As HttpJob
                    authenticationCall.Initialize("",Me)
                    authenticationCall.Download(Url)
                    authenticationCall.GetRequest.SetHeader("Authorization", retVals(0))
                    Wait For (authenticationCall) JobDone(authenticationCall As HttpJob)
                    j = authenticationCall
                    If j.Success = False Then
                        Log($"[ERROR] StatusCode=${authenticationCall.Response.StatusCode}"$)
                        Log($"[Debug] ${authenticationCall.Response.GetHeaders}"$)
                    End If
                End If
            Else
                j = initialCall
                j.ErrorMessage = $"[ERROR] Wrong authentication scheme. Expected [Digest]"$
            End If
        Else
            j = initialCall
            j.ErrorMessage = $"URL (${Url}) did not have a WWW-Authenticate header"$
        End If

    Else
        ' The proper response was not received
        j = initialCall
        If j.Success Then ' In case it was a successful non-proper response, overwrite Success and ErrorMessage
            j.Success = False
            j.ErrorMessage = $"URL (${Url}) did not respond with a 401 status code"$
        End If
    End If

    Return j
End Sub

'See https://github.com/AnywhereSoftware/B4A/blob/b22087de45d1556b105b71c36a6555243ddf6211/Libs_OkHttp/src/anywheresoftware/b4h/okhttp/OkHttpClientWrapper.java#L290
'See https://www.b4x.com/android/forum/threads/http-digest-authorization-fails-in-b4a.158831/post-975228
Private Sub HandleDigest(response As OkHttpResponse, directives As String, username As String, password As String) As String()
    Dim digestInfo(2) As String
    Dim sb As StringBuilder
    sb.Initialize
    Dim rawParams() As String = Regex.Split(",", directives)
    If rawParams.Length > 0 Then
        Dim params As Map
        params.Initialize
        For x = 0 To rawParams.Length - 1
            Log($"[Debug] Directive #${x}: ${rawParams(x)}"$)
            Dim keyValue() As String = Regex.Split("=", rawParams(x))
            If keyValue.Length = 1 Then
                params.Put(keyValue(0).Trim, "")
            else if keyValue.Length = 2 Then
                params.Put(keyValue(0).Trim, keyValue(1).Trim)
            Else
                Log($"[Warning] Unhandled WWW-Authenticate directive: ${rawParams(x)}"$)
            End If

        Next
        'Technically RFC7616 requires a qop (see https://datatracker.ietf.org/doc/html/rfc7616#section-3.3),
        ' but @Erel's code makes it optional, so it's optional here
'        If params.GetDefault("qop","") = "" Then
'            digestInfo(1) = "[ERROR] Digest authentication requires the qop directive"
'        else if params.GetDefault("nonce","") = "" Then
        If params.GetDefault("nonce","") = "" Then
            digestInfo(1) = "[ERROR] Digest authentication requires the nonce directive"
        Else
            Dim algorithms As List = Array As String ("MD5", "SHA-256")
            'algorithms.Initialize2(Array As String ("MD5", "SHA-256"))
            Dim algorithm As String = params.GetDefault("algorithm", "MD5")
            If algorithms.IndexOf(algorithm) <> - 1 Then
                Dim md As MessageDigest
                Dim bc As ByteConverter
                Try
                    'https://datatracker.ietf.org/doc/html/rfc7616#section-3.4.1
                    'Create H(A1) - 
                    Dim realm As String = params.GetDefault("realm", "")
                    Dim unq_realm As String = realm.Replace($"""$,"")
                    Dim A1 As String = $"${username}:${unq_realm}:${password}"$
                    Log($"A1 = ${A1}"$)
                    Dim H_A1 As String = bc.HexFromBytes(md.GetMessageDigest(A1.GetBytes("ISO-8859-1"), algorithm)).ToLowerCase
                    'Create H(A2)
                    Dim jo As JavaObject = response
                    jo = jo.GetField("response")
                    Dim joRequest As JavaObject = jo.RunMethod("request", Null)
                    Dim method As String = joRequest.RunMethod("method", Null)
                    Dim uri As String = requestPath(joRequest.RunMethod("url", Null))
                    Dim A2 As String = $"${method}:${uri}"$
                    Log($"A2 = ${A2}"$)
                    Dim H_A2 As String = bc.HexFromBytes(md.GetMessageDigest(A2.GetBytes("ASCII"), algorithm)).ToLowerCase
                    
                    'Build the response directive
                    Dim digestResponseValue As String    ' contains the text that will be hashed/hex-encoded
                    Dim NC As String = "00000001"
                    'Dim NC As String = "00000002"
                    Dim cnonce As String
                    Dim nonce As String = params.Get("nonce") ' Already checked if available
                    
                    'nonce = $""812682c2e8c20123364385b8723b0fed""$
                    
                    Dim unq_nonce As String = nonce.Replace($"""$,"")
                    
                    Dim qop As String = params.GetDefault("qop", "")
                    If qop = "" Then
                        digestResponseValue = $"${H_A1}:${unq_nonce}:${H_A2}"$
                    Else
                        If qop.Contains("auth") Then
                            qop = "auth"
                            cnonce =bc.HexFromBytes( md.GetMessageDigest($"${DateTime.Now}"$.As(String).GetBytes("ASCII"), algorithm)).ToLowerCase
                            'cnonce = "c2c6d5e0f80c4ef4"
                            digestResponseValue = $"${H_A1}:${unq_nonce}:${NC}:${cnonce}:${qop}:${H_A2}"$
                        Else
                            digestInfo(1) = $"[ERROR] No suported qop option found. Server returned qop="${qop}""$
                        End If
                    End If
                    Log($"digestResponsValue = ${digestResponseValue}"$)
                    
                    If digestInfo(1) = "" Then
                        Dim digestResponse As String = bc.HexFromBytes(md.GetMessageDigest(digestResponseValue.GetBytes("ASCII"), algorithm)).ToLowerCase
                        Dim sb As StringBuilder
                        sb.Initialize
                        sb.Append($"Digest username="${username}",realm=${realm},algorithm=${algorithm},nonce=${nonce},uri="${uri}","$)
                        If qop <> "" Then 
                            sb.Append($"qop=${qop},nc=${NC},cnonce="${cnonce}","$)
                        End If
                        sb.Append($"response="${digestResponse}""$)
                        If params.GetDefault("opaque","") <> "" Then
                            sb.Append($", opaque=${params.Get("opaque")}"$)
                        End If
                        digestInfo(0) = sb.ToString
                        Log($"[Debug] return digest value=${digestInfo(0)}"$)
                    End If
                    
                    'digestInfo(1) = $"[ERROR] Incomplete implemenation"$
                Catch
                    digestInfo(1) = $"[ERROR] Exception error during digest creation: ${LastException.Message}"$
                End Try
                '
                
            Else
                digestInfo(1) = $"[ERROR] Unsupported digest algorithm: ${algorithm}"$
            End If
            
        End If
    Else
        digestInfo(1) = $"[ERROR] No directives given for digest authentication scheme"$
    End If
    If digestInfo(1) <> "" Then Log(digestInfo(1))
    Return digestInfo
End Sub

'See https://github.com/square/okhttp/blob/master/okhttp/src/main/kotlin/okhttp3/internal/http/RequestLine.kt
Private Sub requestPath(url As JavaObject) As String
    Dim path As String = url.RunMethod("encodedPath", Null)
    Dim query As String = url.RunMethod("encodedQuery", Null)
    Return IIf(query <> "null", $"${path}?${query}"$, path)
End Sub
 
Upvote 1
Solution
Top