Android Question [solved] JSON encoding

Discussion in 'Android Questions' started by ALBRECHT, Jul 13, 2019.

Similar threads

Games [XUI2D] Example Pack
B4A Tutorial Integrating Firebase Services
B4A Code Snippet [B4X] GUID
B4A Tutorial Text files
B4A Tutorial [B4X] OkHttpUtils2 with Wait For
  1. ALBRECHT

    ALBRECHT Active Member Licensed User

    Hello,

    Please, i have a Json provided from an external DB Table like that :
    Code:
    [{"Id":"1","Libelle":"CONFORT ET BIEN-ÊTRE","PictureId":"2428"},
    {"Id":"14","Libelle":"AIDES TECHNIQUES","PictureId":"3465"},{"Id":"40","Libelle":"MOBILITE","PictureId":"3464"},{"Id":"53","Libelle":"MATERNITE","PictureId":"3466"},
    {"Id":"66","Libelle":"HYGIENE","PictureId":"3452"},{"Id":"79","Libelle":"DIAGNOSTIQUE","PictureId":"4533"},{"Id":"92","Libelle":"PROTECTION","PictureId":"4534"},
    {"Id":"105","Libelle":"MATERIEL D’INJECTION","PictureId":"3469"},
    {"Id":"32","Libelle":"EQUIPEMENT DE CABINET","PictureId":"4535"},
    {"Id":"36","Libelle":"SOINS ET PANSEMENTS","PictureId":"3437"}]
    but when i read that with :
    Code:
    Dim parser As JSONParser
    parser.Initialize(Job.GetString)
    Dim rows As List
    rows = parser.NextArray
    Thats show me into the listview some special char like : ♦ replacing the accented characters

    and if i use the function (readed into the b4x forum):
    Code:
    public Sub UnicodeEscape (s As StringAs String
        
    Dim sb As StringBuilder
        sb.Initialize
        
    For i = 0 To s.Length - 1
            
    Dim u As String = Bit.ToHexString(Asc(s.CharAt(i)))
            sb.Append(
    "\u")
            
    For i2 = 1 To 4 - u.Length
                sb.Append(
    "0")
            
    Next
            sb.Append(u)
        
    Next
        
    Return sb.ToString
    End Sub
    Thats return something like that : \u00xx for each characters into the strings

    Should i use a special B4X function to read all kind of international characters
    or should i analyse that before the json creation ?

    Thank you
    Michel
     
  2. DonManfred

    DonManfred Expert Licensed User

    Sounds like you are not using the correct charset.

    Can you post a link to request some data from your external DB?

    Can you upload a small project which shows the issue?
     
    BillMeyer and Peter Simpson like this.
  3. ALBRECHT

    ALBRECHT Active Member Licensed User

    The return is exactly what you see (with no header):

    Code:
    [{"Id":"1","Libelle":"CONFORT ET BIEN-ÊTRE","PictureId":"2428"},
    {"Id":"14","Libelle":"AIDES TECHNIQUES","PictureId":"3465"},
    {"Id":"40","Libelle":"MOBILITE","PictureId":"3464"},
    {"Id":"53","Libelle":"MATERNITE","PictureId":"3466"},
    {"Id":"66","Libelle":"HYGIENE","PictureId":"3452"},
    {"Id":"79","Libelle":"DIAGNOSTIQUE","PictureId":"4533"},
    {"Id":"92","Libelle":"PROTECTION","PictureId":"4534"},
    {"Id":"105","Libelle":"MATERIEL D’INJECTION","PictureId":"3469"},
    {"Id":"32","Libelle":"EQUIPEMENT DE CABINET","PictureId":"4535"},
    {"Id":"36","Libelle":"SOINS ET PANSEMENTS","PictureId":"3437"}]
    with 2 special Char : "CONFORT ET BIEN-ÊTRE" at the line 1
    and : a quote : "D’INJECTION" at line 105
     
  4. DonManfred

    DonManfred Expert Licensed User

    yes, but the online service returns them with WHICH charset?
     
  5. emexes

    emexes Well-Known Member Licensed User

    Did you get this going? I tried it with the following code, using the string from your first post:
    Code:
    Sub Process_Globals

        
    Dim J As String = _
        
    $"[{"Id":"1","Libelle":"CONFORT ET BIEN-ÊTRE","PictureId":"2428"},"$ & _
        
    $"{"Id":"14","Libelle":"AIDES TECHNIQUES","PictureId":"3465"},"$ & _
        
    $"{"Id":"40","Libelle":"MOBILITE","PictureId":"3464"},"$ & _
        
    $"{"Id":"53","Libelle":"MATERNITE","PictureId":"3466"},"$ & _
        
    $"{"Id":"66","Libelle":"HYGIENE","PictureId":"3452"},"$ & _
        
    $"{"Id":"79","Libelle":"DIAGNOSTIQUE","PictureId":"4533"},"$ & _
        
    $"{"Id":"92","Libelle":"PROTECTION","PictureId":"4534"},"$ & _
        
    $"{"Id":"105","Libelle":"MATERIEL D’INJECTION","PictureId":"3469"},"$ & _
        
    $"{"Id":"32","Libelle":"EQUIPEMENT DE CABINET","PictureId":"4535"},"$ & _
        
    $"{"Id":"36","Libelle":"SOINS ET PANSEMENTS","PictureId":"3437"}]"$
      
    End Sub

    Sub Activity_Create(FirstTime As Boolean)

        
    Log(J)
      
        
    Dim parser As JSONParser
        parser.Initialize(J)
        
    Dim rows As List
        rows = parser.NextArray
      
        
    For I = 0 To rows.Size -1
            
    Dim X As Map = rows.Get(I)
            
    Log(I & " = " & X)
        
    Next
      
    End Sub
    and get these results:
    Code:
    ** Activity (main) Create, isFirst = true **
    [{
    "Id":"1","Libelle":"CONFORT ET BIEN-ÊTRE","PictureId":"2428"},{"Id":"14","Libelle":"AIDES TECHNIQUES","PictureId":"3465"},{"Id":"40","Libelle":"MOBILITE","PictureId":"3464"},{"Id":"53","Libelle":"MATERNITE","PictureId":"3466"},{"Id":"66","Libelle":"HYGIENE","PictureId":"3452"},{"Id":"79","Libelle":"DIAGNOSTIQUE","PictureId":"4533"},{"Id":"92","Libelle":"PROTECTION","PictureId":"4534"},{"Id":"105","Libelle":"MATERIEL D’INJECTION","PictureId":"3469"},{"Id":"32","Libelle":"EQUIPEMENT DE CABINET","PictureId":"4535"},{"Id":"36","Libelle":"SOINS ET PANSEMENTS","PictureId":"3437"}]
    0 = (MyMap) {Id=1, Libelle=CONFORT ET BIEN-ÊTRE, PictureId=2428}
    1 = (MyMap) {Id=14, Libelle=AIDES TECHNIQUES, PictureId=3465}
    2 = (MyMap) {Id=40, Libelle=MOBILITE, PictureId=3464}
    3 = (MyMap) {Id=53, Libelle=MATERNITE, PictureId=3466}
    4 = (MyMap) {Id=66, Libelle=HYGIENE, PictureId=3452}
    5 = (MyMap) {Id=79, Libelle=DIAGNOSTIQUE, PictureId=4533}
    6 = (MyMap) {Id=92, Libelle=PROTECTION, PictureId=4534}
    7 = (MyMap) {Id=105, Libelle=MATERIEL D’INJECTION, PictureId=3469}
    8 = (MyMap) {Id=32, Libelle=EQUIPEMENT DE CABINET, PictureId=4535}
    9 = (MyMap) {Id=36, Libelle=SOINS ET PANSEMENTS, PictureId=3437}
    ** Activity (main) Resume **
    which includes the non-ASCII characters from the original, and seems to have been delineated correctly.
     
  6. DonManfred

    DonManfred Expert Licensed User

    In fact you started with UTF-8 Data. So the result is that it is working.

    The online data from the TO probably send the data in any other charset than UTF-8.

    So the solution is:
    1. to use
    Code:
    Wait For (j) JobDone(j As HttpJob)
        
    If j.Success Then
            
    Dim res As String = j.GetString2("charset"' replace charset with the correct charsetname
            Log(res)
            
    ' Now parse the data using JsonParser
        Else
            
    Log(j.ErrorMessage)
        
    End If
        j.Release
    2. Change the online tool to return UTF-8

    PS: Thats the reason i asked for a sample project or a url which returns some data.
    Using this url we would see the used charset encoding.
     
    Last edited: Jul 13, 2019
    Jorge M A and mangojack like this.
  7. emexes

    emexes Well-Known Member Licensed User

    I started with the Unicode string that Albrecht supplied in post #1, which appears to be before the "♦ replacing the accented characters" issue occurred. Whether it was encoded in UTF-8 or UTF-7 or UTF-16 or Windows-1250 is not relevant, as long as it decoded back to the original Unicode here. Because it is JSON, I expect that it is what is being returned from the database to B4A and thus includes the effects of any Unicode transfer encoding/decoding. But I agree that perhaps the supplied string has been obtained by some other means, and is not an exact copy of the string returned by Job.GetString.
    It does. I posted my code and string so that Albrecht could run it using the configuration that is causing the replacement of (non-ASCII) characters. If the same processing that works ok with "my" string, fails with the same string retrieved from the database, then we can compare those two input strings to see what the cause might be.
    What is TO?
    Perhaps, but then the string in post#1 should already have contained the replacement characters. Like you:
    I agree that this would be useful to see.
     
  8. ALBRECHT

    ALBRECHT Active Member Licensed User

    Hello,

    In fact, my link (api) : https://www.materiel-medical-a-domicile.com/Asp/ListCatJson.asp

    with that kind of settings before the creation of the Json :
    response.ContentType = "application/json"
    response.Charset = "UTF-8"


    that return :

    [{"Id":"1","Libelle":"CONFORT ET BIEN-�TRE","PictureId":"2428"},{"Id":"14","Libelle":"AIDES TECHNIQUES","PictureId":"3465"},{"Id":"40","Libelle":"MOBILIT�","PictureId":"3464"},{"Id":"53","Libelle":"MATERNIT�","PictureId":"3466"},{"Id":"66","Libelle":"HYGIENE","PictureId":"3452"},{"Id":"79","Libelle":"DIAGNOSTIQUE","PictureId":"4533"},{"Id":"92","Libelle":"PROTECTION","PictureId":"4534"},{"Id":"105","Libelle":"MATERIEL D'INJECTION","PictureId":"3469"},{"Id":"32","Libelle":"EQUIPEMENT DE CABINET","PictureId":"4535"},{"Id":"36","Libelle":"SOINS ET PANSEMENTS","PictureId":"3437"}]

    so, that kind of Special unknowed Char are displayed in in the listview as is ! : �

    (the chars should be : Ê , É)

    Please, what script to use to fix this when reading the JSON in the listview

    thanks for your helps
    Michel
     
  9. DonManfred

    DonManfred Expert Licensed User

    the data seems to be not utf8

    This code is working for me

    Code:
    Dim j As HttpJob
        j.Initialize(
    "",Me)
        j.Download(
    "https://www.materiel-medical-a-domicile.com/Asp/ListCatJson.asp")
        
    Wait For (j) JobDone(j As HttpJob)
        
    If j.Success Then
            
    Dim res As String = j.GetString2("ISO-8859-1"' replace charset with the correct charsetname
            Log(res)
            
    Dim parser As JSONParser
            parser.Initialize(res)
            
    Dim root As List = parser.NextArray
            
    For Each colroot As Map In root
                
    Dim PictureId As String = colroot.Get("PictureId")
                
    Dim Id As String = colroot.Get("Id")
                
    Dim Libelle As String = colroot.Get("Libelle")
                
    Log($"ID ${Id},PictureID ${PictureId}: Libelle ${Libelle}"$)
            
    Next
            
    ' Now parse the data using JsonParser
        Else
            
    Log(j.ErrorMessage)
        
    End If
        j.Release
     
    Jorge M A, Erel and ALBRECHT like this.
  10. ALBRECHT

    ALBRECHT Active Member Licensed User

    indeed, the charset replacement will works the fine and the européan Chars are correctly displayed,

    but the utf-8 Charset cover more :

    so if i change by :
    Code:
    j.GetString2("utf-8")
    the unknowed Chars � appear again, why ?
     
  11. DonManfred

    DonManfred Expert Licensed User

    because the server is NOT sending utf-8 even if specified in the header. The data is in ISO charset
     
    Jorge M A likes this.
  12. emexes

    emexes Well-Known Member Licensed User

    I started explaining Unicode mappings, but... the simplest answer here is Manfred's, ie: what is received from the database is NOT UTF-8 encoded Unicode.

    It is some 8-bit subset of Unicode, probably ISO-8559-1 or windows-1252 (a superset of ISO-8559-1).

    UTF-8 encodes Unicode characters 0..127 to byte values 0..127, and Unicode characters > 127 to a multibyte sequence. These multibyte sequences are marked by the high bit being set to 1. If you decode a stream of 8-bit characters using UTF-8, then any characters that have the high bit set (eg, your accented characters) will instead be misinterpreted as a multibyte sequence and thus not decode correctly.
     
    Last edited: Jul 13, 2019
  13. yfleury

    yfleury Active Member Licensed User

    Your .asp file is not set to utf-8, then your web server send a file to same charset. Save your .asp to utf-8
     
  14. Rubsanpe

    Rubsanpe Member Licensed User

    Web Browser Page Information (Firefox) say the encode is "windows-1252".

    Rubén
     
    emexes likes this.
  15. ALBRECHT

    ALBRECHT Active Member Licensed User

    hello, I insert back the initial settings on the.asp page :
    Code:
    response.ContentType = "application/json"
    response.Charset = 
    "UTF-8"
    Said by chrome consol : document.characterSet = "UTF-8"

    But even like that, reading the JSON in the listview shows the unknown characters! with or not the function : Job.GetString2("UTF-8")

    Code:
    Sub JobDone (Job As HttpJob)  
        
    If Job.Success = True Then      
            
    Dim res As String = Job.GetString2("UTF-8"' or ISO-8859-1
            Dim parser As JSONParser
            parser.Initialize(res)
            
    Dim rows As List
            rows = parser.NextArray

            
    Dim LibTxt As String
            
    For i = 0 To rows.Size - 1
                
    Dim m As Map
                m = rows.Get(i)
                LibTxt = m.Get(
    "Libelle")
                ListView1.AddSingleLine(LibTxt & 
    " : " & i)
            
    Next
        
    Else
            
    Log ("Db Error !!")
        
    End If  
        Job.Release
    End Sub
    and UTF-8 stay the better charset choice ...
     
    Last edited: Jul 13, 2019
  16. emexes

    emexes Well-Known Member Licensed User

    I am getting a header that says it is UTF-8

    upload_2019-7-14_1-40-7.png

    but it still looks like an 8-bit encoding:

    upload_2019-7-14_1-42-7.png

    so right now I am inclined towards: if it ain't broke, don't fix it

    What does it look like in your browser, using the various character mapping settings?
     
  17. hatzisn

    hatzisn Active Member Licensed User

    @yfleury 's answer is the correct solution. Open your asp code with notepad > File > Save As > (select encoding UTF-8) and you are all set.
     
  18. ALBRECHT

    ALBRECHT Active Member Licensed User

    hello hatzisn,

    that asp (as the others of my api) file is already record in UTF-8 :

    encoding charset settings : Unicode (UTF-8 without sign) - Codes Page 65001
     
  19. hatzisn

    hatzisn Active Member Licensed User

    Have you saved it with notepad?
     
  20. ALBRECHT

    ALBRECHT Active Member Licensed User

    Of course,

    and as i said above, chrome consol confirm : document.characterSet = "UTF-8"

    thanks
     
Loading...
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice