Android Question Http digest authorization fails in B4A

fs007

Member
I need to write an app which sends a HTTP GET to devices named "Shelly" (these are so called "SmartSwitches, shelly.com).
They need digest authorization as described here: https://shelly-api-docs.shelly.cloud/gen2/General/Authentication/
My code (which does NOT work) is:
B4X:
Dim hj As HttpJob
hj.Initialize("",Me)
hj.Username="admin"
hj.Password="password"
hj.Download("http://192.168.5.188/relay/0?turn=on")
Wait for (hj) JobDone(hj As HttpJob)
If hj.Success Then
    label1.Text = hj.GetString
Else
    label1.Text ="Error"
End If

I get an error "Unauthorized", but the credentials are correct. What's wrong here ???
 
Solution
what would be the proper format for sha-256 digest auth ? code snippet would be nice
Try the code below. I've tested it against https://httpbin.org/digest-auth/auth/test/test3/SHA-256 and it seems to work. The code is not all-encompassing regarding the RFC spec and only handles GET requests, but hopefully, it implements enough to work for your needs.

Usage example:

B4X:
    Wait For(AuthenticateUrl("https://httpbin.org/digest-auth/auth/test/test3/SHA-256", "test", "test3")) Complete (j As HttpJob)
    If j.Success = True
        'successfully authenticated
        Log(j.GetString)
    Else
        Log(j.ErrorMessage)
    End If

The implementation code:

B4X:
'See...

OliverA

Expert
Licensed User
Longtime User
When communicating over HTTP, this process must be repeated for each request you send to the device.
Correct. And HTTP Digest via HttpJob will do that for you (that is how digest works). It's just that currently the underlying OkHttpClientWrapper does not support it. I do understand what you are doing, but the API doc also says that it would work with HTTP Digest. If it would, the HTTP Digest would take care of all the realm, nonce, hashing, etc, without the programmer having to do it himself.
 
Upvote 0

aeric

Expert
Licensed User
Longtime User
I guess one thing for sure, the documentation is confusing and misleading.

I may want to correct my code. To enable the authentication, the method to be call should be
Shelly.SetAuth

But the documentation is showing a call to another method
Shelly.GetStatus
which should not be called before calling the method above first.

I think once authentication is enabled, any subsequent (GET) API calls can be made with the username and password like the code posted by the OP.

If authentication is disabled, no point to provide the credentials.
 
Upvote 0

aeric

Expert
Licensed User
Longtime User
Another correction : RPC call only uses POST while the REST API calls are not limited to POST.
So my code should be only support POST for calling RPC.
For REST API, just normal http call.
 
Upvote 0

aeric

Expert
Licensed User
Longtime User
I searched the internet for "Shelly HTTP 401"...

Maybe one reason is related to restricted login enabled.
hakana commented on Dec 18, 2019
Look like you have restricted login enabled on Shelly, you need to specify the same user and password in config.yaml.

paparapapapapa commented on Dec 19, 2019
This was the issue thank you. It worked adding:
shelly:
username: xxx
password: xxx

source: https://github.com/StyraHem/ShellyForHASS/issues/123
 
Upvote 0

aeric

Expert
Licensed User
Longtime User
I think all these 3 ways are equivalent. So it is not related to httputils library.

B4X:
hj.Username = "admin"
hj.Password = "password"
hj.Download("http://192.168.5.188/relay/0?turn=on")

B4X:
hj.Download("http://192.168.5.188/relay/0?turn=on")
hj.GetRequest.SetHeader("Authorization", "Basic " & SU.EncodeBase64("admin:password".GetBytes("UTF8")))

B4X:
hj.Download("http://admin:[email protected]/relay/0?turn=on")
 
Upvote 0

OliverA

Expert
Licensed User
Longtime User
The interesting part about that one is that the issue poster uses BASIC authentication with the Shelly implementation he is working with

The doc provided by the poster though mentions
Communication through HTTP and Websocket channels is secured by a digest authentication mechanism using the SHA256 hmac algorithm as defined in RFC7616.
So here we are looking at the digest authentication method. The reason OP is getting the 401, is that the following occurs (with the code posted the first post).

1) The underlying library is set up to handle a digest authentication via a custom Authenticator
2) The URL call is made w/o passing any user/password information
3) The server answers with 401, with various pieces of information in the WWW-Authenticate header used by the client to create the response
4) The underlying library calls the custom Authenticator.
5) The custom Authenticator checks if the request is for a digest authentication. If so it creates the MD5 digest response (using the information provided in the WWW-Authenticate header) and passes a new request back to the underlying library that has the digest information set in the Authorization header
6) The underlying library makes another call to the server
7) The server, expecting a SHA256-generated response, fails to authenticate the user and returns another 401
8) This second 401 is what is returned to the B4X application

Where the B4A digest implementation fails is in step#5. The custom Authenticator does not check the algorithm entry in the WWW-Authenticate header but just assumes that the algorithm is MD5.
 
Upvote 0

aeric

Expert
Licensed User
Longtime User
The interesting part about that one is that the issue poster uses BASIC authentication with the Shelly implementation he is working with

The doc provided by the poster though mentions

So here we are looking at the digest authentication method. The reason OP is getting the 401, is that the following occurs (with the code posted the first post).

1) The underlying library is set up to handle a digest authentication via a custom Authenticator
2) The URL call is made w/o passing any user/password information
3) The server answers with 401, with various pieces of information in the WWW-Authenticate header used by the client to create the response
4) The underlying library calls the custom Authenticator.
5) The custom Authenticator checks if the request is for a digest authentication. If so it creates the MD5 digest response (using the information provided in the WWW-Authenticate header) and passes a new request back to the underlying library that has the digest information set in the Authorization header
6) The underlying library makes another call to the server
7) The server, expecting a SHA256-generated response, fails to authenticate the user and returns another 401
8) This second 401 is what is returned to the B4X application

Where the B4A digest implementation fails is in step#5. The custom Authenticator does not check the algorithm entry in the WWW-Authenticate header but just assumes that the algorithm is MD5.
At this point, my strong believe are:
1. Post #22
2. The authentication mentioned in the documentation is for calling RPC (POST) API, not related to the /relay/x EndPoint which is a HTTP GET request.

Let say I am creating an app for Public use, the user who downloaded my app just need to enter his username and password to make the API call in this case the /relay/x to turn the switch on or off. Provided that the user has updated the config.yaml as mentioned in post #22.

Nothing to do on the SHA-256 or the code I posted for authenticating with HMAC.

Let's wait for the OP replies.
 
Upvote 0

fs007

Member
config.yaml as mentioned in post #22.
where can i find config.yaml ???
Look: any browser can log in these shelly devices without knowing anything about "config.yaml", so Username and password is enough.

I don't know what "restricted login" means.
There are two ways to access the web ui of the device: 1. without authentification, pure http, 2. with this so called digest auth.
Maybe here is some misunderstanding: My issue is login on the web ui of the device, NOT login on the shelly cloud service.
 
Upvote 0

OliverA

Expert
Licensed User
Longtime User
Look: any browser can log in these shelly devices without knowing anything about "config.yaml", so Username and password is enough.
To be fair, according to https://caniuse.com/mdn-http_headers_www-authenticate_digest_sha-256 most browsers did not implement sha256 digest until late last year. Firefox was on the forefront of this, implementing this feature int the fall of 2021. I did place a wish to expand HttpJob's capabilities to support RFC7616 algorithms. See Thread '[B4X] Support for RFC7616 algorithms for digest authentication' https://www.b4x.com/android/forum/t...-algorithms-for-digest-authentication.158862/
 
Upvote 0

aeric

Expert
Licensed User
Longtime User
I don't know what "restricted login" means.
Does it have something call ioBroker?

My issue is login on the web ui of the device
What happen if you use web browser to open this url?

Do you also try the second code in post #25?
B4X:
hj.Download("http://192.168.5.188/relay/0?turn=on")
hj.GetRequest.SetHeader("Authorization", "Basic " & SU.EncodeBase64("admin:password".GetBytes("UTF8")))
* SU is StringUtils
 
Upvote 0

fs007

Member
Does it have something call ioBroker?
No, never seen. I'm not interested in "integrating Shelly devices in MQTT" , but simply make the well-known http-requests to switch the relays.

What happen if you use web browser to open this url?
http://admin:[email protected]/relay/0?turn=on
That works. Relay is switched. Confirmed with Google Chrome

Do you also try the second code in post #25?
B4X:
hj.Download("http://192.168.5.188/relay/0?turn=on")
hj.GetRequest.SetHeader("Authorization", "Basic " & SU.EncodeBase64("admin:password".GetBytes("UTF8")))
Yes i tried that, but it fails. Error message is: ResponseError. Reason: Unauthorized

I think, the reason for this mess is plain and simple: B4A does not support sha256 digest
 
Upvote 0

drgottjr

Expert
Licensed User
Longtime User
while waiting to see if op tried my mod to okhttp, i was taking a look
at RFC7616 (which, of course, has been superseded by RFC 7235)

if tl;dnr, skip to bottom.

first, the version of okhttpclient used in our internal libraries, supports
all the methods described in RFC7616 (among which MD5 and SHA-256),
technically, my mod should work in cases where SHA-256 was required.
i also did a version that allows for MD5 (by default) or SHA-256 if that's
what the server wanted. (@OliverA's wish for an updated okhttp duly noted).
unfortunately, there are no test sites (none that i could find) that allow one
to debug the full process. i found 3 which didn't really put much effort into
implementing the protocol (httpbin.org, among others).

among some interesting statements in RFC7616:
"both client and server know the username." and "the server need not know
the password" (the authorization protocol deals with that). furthermore,
use of the digest authentication "requires that the authenticating agent (usually
the server) store some data derived from the user's name and password
in a "password file" associated with a given realm." the term password can
refer to, eg, a hash rather than the actual password. without getting into a
longer discussion about this aspect, suffice it to say if you look at a typical
unix password file, you'll see the usernames plus some binary garbage where
the passwords would be. passwords are unknown. in the case of digest
authentication, the client creates a hash of his password based on part of
the server's challenge. the server compares the hash with what it has on
file.

the server "might challenge the client by responding with "401 Unauthorized"
response and include one or more WWW-Authenticate header fields." in
other words, the client might get a choice of algorithm (which choice has to
be indicated in the response to the server).

the specification indicates that SHA2-256 is must be implemented by a
conforming client. MD5 is also required for backward compatibility

lastly, "if digest authentication is being used, it should be over a secure
channel like https".

----------------------------------------------------------------------------------------------------
i found the most interesting statement related to the server's having prior
knowledge of the username and password (or hash). has the op carried
out Shelly.SetAuth at some point? does the device already have the
username and password? if not, then logging in, whether by any means,
would fail.
 
Upvote 0

aeric

Expert
Licensed User
Longtime User
I think, the reason for this mess is plain and simple: B4A does not support sha256 digest
That works. Relay is switched. Confirmed with Google Chrome
Is that mean Google Chrome supports and knows that automatically it should use SHA-256 instead of Base64Encode ?

What I understand is,
Opening this URL in browser is same as using job.Download

B4X:
hj.Download("http://admin:[email protected]/relay/0?turn=on")
 
Last edited:
Upvote 0

fs007

Member
Is that mean Google Chrome supports and knows that automatically it should use SHA-256 instead of Base64Encode
Obviously yes; btw: other browsers work too, not only Google Chrome. Maybe these browser try different encodings consecutively.


What I understand is,
Opening this URL in browser is same as using job.Download
No. The browser does not use B4A's job.Download.
Browser works, B4A fails.

I'll try @drgottjr 's mod, but i don't know, if i find the time this week ...

Sorry for my seemingly late posting, but in fact i didn't answer late, instead my posts need approval by a moderator and that needs some time ...
 
Upvote 0

OliverA

Expert
Licensed User
Longtime User
automatically it should use SHA-256 instead of Base64Encode
No, they know this by the response header they receive when they first try to access an URL the "normal" way and then receive a status code of 401. Included in the headers returned from the server, there should be a WWW-Authenticate entry. This entry consists of various fields (see https://datatracker.ietf.org/doc/html/rfc7616#section-3.3). One of those fields may be the "algorithm" field. This field tells the requestor (be it a browser, or some library) what algorithm to use. If the field does not exist, the default is MD5. For SHA-256, it's, well, SHA-256. The information returned by the server is used to create an authorization response header and re-request the same URL.

As of fall 2023 (see post #19 for a link to check browser support: https://www.b4x.com/android/forum/threads/http-digest-authorization-fails-in-b4a.158831/post-975451), many browsers now support SHA-256. Currently, OkHttpUtils2 only supports MD5 (I linked to the source in post #17 here https://www.b4x.com/android/forum/threads/http-digest-authorization-fails-in-b4a.158831/post-975244). I've made a wish (https://www.b4x.com/android/forum/t...for-digest-authentication.158862/#post-975264) to have SHA-256 (and maybe others covered by RFC7616) added to B4X's HttpJob implementation.

RFC7616 (which, of course, has been superseded by RFC 7235)
I'm pretty sure RFC 7235 supersedes RFC 2616. I don't see anything really superseding RFC 7616.
 
Upvote 0

OliverA

Expert
Licensed User
Longtime User
Upvote 0

fs007

Member
have op unzip the attached to his additional libraries folders.
then, in his project in the additional libraries tab, have him
uncheck "okhttp" and check "okhttp2". if the project is already
open when he unzips the archive, he will need to refresh the
additional libraries tab (right click).
@drgottjr i tried your mod today, but i'm sorry, it doesn't work. Server returns still 401.

Moreover: First i tried it with my code in post #1 . The B4A IDE delivers a warning: "Library 'OkHttp2 not used'. So one can assume, that HttpJob doesn't use your mod at all, but uses OkHttpUtils2

Then i tried another code using HttpClient/HttpRequest:
B4X:
Dim hr As OkHttpRequest
hr.InitializeGet("http://192.168.5.188/relay/0?turn=on")
hc.ExecuteCredentials(hr,66,"admin","password")
This code indeed relies on your OkHttp2, but returns again 401 error
 
Upvote 0

drgottjr

Expert
Licensed User
Longtime User
thanks for trying it. you know, i missed this way back: httpjob. you're using
httpjob, not okhttpjob. you can dump my mod. the original apache http.jar (which
we haven't used in years) handles digest authentication differently than okhttp.jar.
when used with okhttputils2, my mod is called. sorry for the waste of your time.

the last thing i'll have to say in this matter has to do with whether or not the device
has your username (at least) stored. according to rfc2617, that has to be the case
for digest authentication to work. if you (the server) send me a nonce and i use it
to create a hash of my username and password and send the hash back to you, it
serves no purpose if you don't already have my username and password. if you
already have my username and password stored, you can perform the same operation
with your nonce and compare the result with what i send you. if you don't already
have the username and password, you have nothing to compare my hash with.
 
Upvote 0

fs007

Member
the last thing i'll have to say in this matter has to do with whether or not the device has your username (at least) stored
the username is always "admin" (unchangeable). So, yes, this name is certainly stored.

when used with okhttputils2, my mod is called
okhttp2 and okhttputils2 are activated, but code fails to authenticate.
If i uncheck okhttp2 AND okhttp, no error or warning is reported by the ide.
If i check okhttp2, the IDE delivers the warning "Library 'OkHttp2' is not used".

you're using httpjob, not okhttpjob
when i enter okhttpjob instead httpjob, B4A IDE says "unknown type"
 
Last edited:
Upvote 0
Top