Android Question java.net.IDN. [Solved]

trepdas

trepdas

Active Member
Licensed User
Longtime User
Hi Erel and good people,

I am trying to use erel's code to convert text to punycode and vice verse , using the internal lib IDN,

B4X: 
Sub ToACE (s As String) As String
    Dim jo As JavaObject
    jo.InitializeStatic("java.net.IDN")
    Return jo.RunMethod("toASCII", Array(s))
End Sub

Sub FromACE (s As String) As String
    Dim jo As JavaObject
    jo.InitializeStatic("java.net.IDN")
    Return jo.RunMethod("toUnicode", Array(s))
End Sub



but it fails if I add a number to the string.

for example,

using the converter on https://www.punycoder.com/ ,

entering text : מסמך2
will print out : xn--2-kiclao

but b4a crashes when it get that string.

how can it be fixed ?

? ? ? ?
here's the log :


B4X: 
Logger connected to:  samsung SM-J710F
--------- beginning of main


** Activity (main) Pause, UserClosed = true **
*** Service (starter) Create ***
** Service (starter) Start **
** Activity (main) Create, isFirst = true **
user id : LX13E124H6
** Activity (main) Resume **
*** Service (httputils2service) Create ***
** Service (httputils2service) Start **
Url Exists.
Internet Connection OK!
Error occurred on line: 615 (Main)
java.lang.reflect.InvocationTargetException
    at java.lang.reflect.Method.invoke(Native Method)
    at anywheresoftware.b4j.object.JavaObject.RunMethod(JavaObject.java:131)
    at teletag.b4a.main._toace(main.java:1654)
    at teletag.b4a.main._createidn(main.java:1349)
    at teletag.b4a.main._goteletag(main.java:1507)
    at teletag.b4a.main._btngo_click(main.java:1443)
    at java.lang.reflect.Method.invoke(Native Method)
    at anywheresoftware.b4a.shell.Shell.runMethod(Shell.java:732)
    at anywheresoftware.b4a.shell.Shell.raiseEventImpl(Shell.java:351)
    at anywheresoftware.b4a.shell.Shell.raiseEvent(Shell.java:255)
    at java.lang.reflect.Method.invoke(Native Method)
    at anywheresoftware.b4a.ShellBA.raiseEvent2(ShellBA.java:144)
    at anywheresoftware.b4a.BA.raiseEvent2(BA.java:197)
    at anywheresoftware.b4a.BA.raiseEvent(BA.java:193)
    at anywheresoftware.b4a.objects.ViewWrapper$1.onClick(ViewWrapper.java:80)
    at android.view.View.performClick(View.java:6935)
    at android.widget.TextView.performClick(TextView.java:12752)
    at android.view.View$PerformClick.run(View.java:26211)
    at android.os.Handler.handleCallback(Handler.java:790)
    at android.os.Handler.dispatchMessage(Handler.java:99)
    at android.os.Looper.loop(Looper.java:164)
    at android.app.ActivityThread.main(ActivityThread.java:7000)
    at java.lang.reflect.Method.invoke(Native Method)
    at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:441)
    at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1408)
Caused by: java.lang.IllegalArgumentException: Invalid input to toASCII: מסמך1
    at java.net.IDN.toASCII(IDN.java:112)
    at java.net.IDN.toASCII(IDN.java:134)
    ... 25 more
Caused by: The input does not conform to the rules for BiDi code points.. line:  0. preContext:  . postContext: ך1
    at android.icu.text.StringPrep.prepare(StringPrep.java:556)
    at android.icu.impl.IDNA2003.convertToASCII(IDNA2003.java:180)
    at android.icu.impl.IDNA2003.convertIDNToASCII(IDNA2003.java:277)
    at android.icu.text.IDNA.convertIDNToASCII(IDNA.java:654)
    at java.net.IDN.toASCII(IDN.java:110)
    ... 26 more
** Activity (main) Pause, UserClosed = true **
** Service (starter) Destroy (ignored)**
** Service (httputils2service) Destroy **
*** Service (starter) Create ***
** Service (starter) Start **
** Activity (main) Create, isFirst = true **
user id : LX13E124H6
** Activity (main) Resume **
*** Service (httputils2service) Create ***
** Service (httputils2service) Start **
Url Exists.
Internet Connection OK!
*** Service (starter) Create ***
** Service (starter) Start **
** Activity (main) Create, isFirst = true **
user id : LX13E124H6
** Activity (main) Resume **
*** Service (httputils2service) Create ***
** Service (httputils2service) Start **
Url Exists.
Internet Connection OK!
Error occurred on line: 678 (Main)
java.lang.reflect.InvocationTargetException
    at java.lang.reflect.Method.invoke(Native Method)
    at anywheresoftware.b4j.object.JavaObject.RunMethod(JavaObject.java:131)
    at teletag.b4a.main._toace(main.java:1654)
    at teletag.b4a.main._createidn(main.java:1349)
    at teletag.b4a.main._goteletag(main.java:1507)
    at teletag.b4a.main._btngo_click(main.java:1443)
    at java.lang.reflect.Method.invoke(Native Method)
    at anywheresoftware.b4a.shell.Shell.runMethod(Shell.java:732)
    at anywheresoftware.b4a.shell.Shell.raiseEventImpl(Shell.java:351)
    at anywheresoftware.b4a.shell.Shell.raiseEvent(Shell.java:255)
    at java.lang.reflect.Method.invoke(Native Method)
    at anywheresoftware.b4a.ShellBA.raiseEvent2(ShellBA.java:144)
    at anywheresoftware.b4a.BA.raiseEvent2(BA.java:197)
    at anywheresoftware.b4a.BA.raiseEvent(BA.java:193)
    at anywheresoftware.b4a.objects.ViewWrapper$1.onClick(ViewWrapper.java:80)
    at android.view.View.performClick(View.java:6935)
    at android.widget.TextView.performClick(TextView.java:12752)
    at android.view.View$PerformClick.run(View.java:26211)
    at android.os.Handler.handleCallback(Handler.java:790)
    at android.os.Handler.dispatchMessage(Handler.java:99)
    at android.os.Looper.loop(Looper.java:164)
    at android.app.ActivityThread.main(ActivityThread.java:7000)
    at java.lang.reflect.Method.invoke(Native Method)
    at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:441)
    at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1408)
Caused by: java.lang.IllegalArgumentException: Invalid input to toASCII: מסמך2
    at java.net.IDN.toASCII(IDN.java:112)
    at java.net.IDN.toASCII(IDN.java:134)
    ... 25 more
Caused by: The input does not conform to the rules for BiDi code points.. line:  0. preContext:  . postContext: ך2
    at android.icu.text.StringPrep.prepare(StringPrep.java:556)
    at android.icu.impl.IDNA2003.convertToASCII(IDNA2003.java:180)
    at android.icu.impl.IDNA2003.convertIDNToASCII(IDNA2003.java:277)
    at android.icu.text.IDNA.convertIDNToASCII(IDNA.java:654)
    at java.net.IDN.toASCII(IDN.java:110)
    ... 26 more
 
Last edited:
Sort by date Sort by votes
Upvote 0
Thank you Erel for the quick response. ?

I need this for strings manipulation. I do not necessarily need it as string , but using the output string , I create a file that is named as the convertion result.

for example :
מסמך2.txt

would resolve into :
xn--2-kiclao.txt

and not crash (log above)
 
Upvote 0
If I send a string with no digits - it works ok.
(java error in grey in the log for not accessing url - but program keeps running.
I don't care for this error since I would need only the generated idn string. don't want to use it for domains.

if I add a number to the string - it crashes.
webview though - gets it correctly (it I try to send the webview to
אראל_הצילו2.com

it converts it well to xn--_2-uldavg1avd2hra.com)
(webview1.url)


[CODE lang="b4x" title="log"]*** Service (starter) Create ***
** Service (starter) Start **
** Activity (main) Create, isFirst = true **
user id : G74IXV5CGJ
** Activity (main) Resume **
*** Service (httputils2service) Create ***
** Service (httputils2service) Start **
Url Exists.
Internet Connection OK!
GFP OK.
User Input : אראל_הצילו
IdnString : xn--_-zhcarfxsd2gp
Invalid link: https://
ResponseError. Reason: java.net.UnknownHostException: Unable to resolve host "invalid-url": No address associated with hostname, Response:
java.net.UnknownHostException: Unable to resolve host "invalid-url": No address associated with hostname
Error occurred on line: 695 (Main)
java.lang.reflect.InvocationTargetException
at java.lang.reflect.Method.invoke(Native Method)
at anywheresoftware.b4j.object.JavaObject.RunMethod(JavaObject.java:131)
at teletag.b4a.main._toace(main.java:1661)
at java.lang.reflect.Method.invoke(Native Method)
at anywheresoftware.b4a.shell.Shell.runMethod(Shell.java:732)
at anywheresoftware.b4a.shell.Shell.raiseEventImpl(Shell.java:351)
at anywheresoftware.b4a.shell.Shell.raiseEvent(Shell.java:255)
at java.lang.reflect.Method.invoke(Native Method)
at anywheresoftware.b4a.ShellBA.raiseEvent2(ShellBA.java:144)
at anywheresoftware.b4a.BA.raiseEvent2(BA.java:197)
at anywheresoftware.b4a.debug.Debug.delegate(Debug.java:262)
at teletag.b4a.main._createidn(main.java:1353)
at teletag.b4a.main._goteletag(main.java:1514)
at teletag.b4a.main._btngo_click(main.java:1447)
at java.lang.reflect.Method.invoke(Native Method)
at anywheresoftware.b4a.shell.Shell.runMethod(Shell.java:732)
at anywheresoftware.b4a.shell.Shell.raiseEventImpl(Shell.java:351)
at anywheresoftware.b4a.shell.Shell.raiseEvent(Shell.java:255)
at java.lang.reflect.Method.invoke(Native Method)
at anywheresoftware.b4a.ShellBA.raiseEvent2(ShellBA.java:144)
at anywheresoftware.b4a.BA.raiseEvent2(BA.java:197)
at anywheresoftware.b4a.BA.raiseEvent(BA.java:193)
at anywheresoftware.b4a.objects.ViewWrapper$1.onClick(ViewWrapper.java:80)
at android.view.View.performClick(View.java:6597)
at android.view.View.performClickInternal(View.java:6574)
at android.view.View.access$3100(View.java:778)
at android.view.View$PerformClick.run(View.java:25885)
at android.os.Handler.handleCallback(Handler.java:873)
at android.os.Handler.dispatchMessage(Handler.java:99)
at android.os.Looper.loop(Looper.java:193)
at android.app.ActivityThread.main(ActivityThread.java:6669)
at java.lang.reflect.Method.invoke(Native Method)
at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:493)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:858)
Caused by: java.lang.IllegalArgumentException: Invalid input to toASCII: אראל_הצילו2
at java.net.IDN.toASCII(IDN.java:110)
at java.net.IDN.toASCII(IDN.java:133)
... 34 more
Caused by: The input does not conform to the rules for BiDi code points.. line: 0. preContext: . postContext: ו2
at android.icu.text.StringPrep.prepare(StringPrep.java:558)

at android.icu.impl.IDNA2003.convertToASCII(IDNA2003.java:180)
at android.icu.impl.IDNA2003.convertIDNToASCII(IDNA2003.java:277)
at android.icu.text.IDNA.convertIDNToASCII(IDNA.java:654)
at java.net.IDN.toASCII(IDN.java:108)
... 35 more
Invalid link: https://
--------- beginning of crash
java.lang.RuntimeException: java.net.SocketException: Socket closed
at anywheresoftware.b4a.ShellBA.raiseEvent2(ShellBA.java:175)
at anywheresoftware.b4a.BA$2.run(BA.java:387)
at android.os.Handler.handleCallback(Handler.java:873)
at android.os.Handler.dispatchMessage(Handler.java:99)
at android.os.Looper.loop(Looper.java:193)
at android.app.ActivityThread.main(ActivityThread.java:6669)
at java.lang.reflect.Method.invoke(Native Method)
at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:493)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:858)
Caused by: java.net.SocketException: Socket closed
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:124)
at java.net.SocketOutputStream.write(SocketOutputStream.java:161)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at anywheresoftware.b4a.shell.ShellConnector.sendControlMessage(ShellConnector.java:62)
at anywheresoftware.b4a.shell.Shell.raiseEventImpl(Shell.java:305)
at anywheresoftware.b4a.shell.Shell.raiseEvent(Shell.java:255)
at java.lang.reflect.Method.invoke(Native Method)
at anywheresoftware.b4a.ShellBA.raiseEvent2(ShellBA.java:144)
... 8 more
[/CODE]



in main I call it like this :
B4X: 
IdnString = ToACE(TtgString,flag)
    Log ("User Input : "&TtgString)
    Log ("IdnString : "&IdnString)


and the sub, as for now , looks like this ;

B4X: 
Sub ToACE (s As String, flag As Int) As String
    
        Dim jo As JavaObject
        jo.InitializeStatic("java.net.IDN")
        
        Return jo.RunMethod("toASCII", Array(s))

End Sub
 
Last edited:
Upvote 0
for now, I'll settle with this solution.
simply just got to stay with idn algorithm. this is the base of the project.

B4X: 
str1=TtgString
    StrNum=Regex.Replace("\D",str1,"")
    StrChars=Regex.Replace("\d",str1,"")



Try
        IdnString = ToACE(StrChars,flag)
        IdnString = IdnString.Replace("xn--","xn--"&StrNum)
        IdnPassed=True

    Catch
        Log(LastException)
        Log ("IDN Failed.")
        IdnFailed=True
        IdnString=""
    End Try

    Log ("User Input : "&TtgString)
    Log ("IdnString : "&IdnString)
 
Upvote 0
The above solution failed. it converts the string differently if I try to play with it after converted. :(

this is from a php page I have in my archives that used to handle idn convertions perfectly.


Unicode input might be given as either UTF-8 string, UCS-4 string or UCS-4 array.
Available options and values:
* [encoding - Use either UTF-8, UCS4 as array or UCS4 as string as input ('utf8' for UTF-8,
* 'ucs4_string' and 'ucs4_array' respectively for UCS4); The output is always UTF-8]
* [overlong - Unicode does not allow unnecessarily long encodings of chars,
* to allow this, set this parameter to true, else to false;
* default is false.]
* [strict - true: strict mode, good for registration purposes - Causes errors
* on failures; false: loose mode, ideal for "wildlife" applications
* by silently ignoring errors and returning the original input instead
Click to expand...


How do I convert UTF-8 to UCS-4 ?????
 
Last edited:
Upvote 0
Erel said:
As I wrote above, IDN is not expected to support arbitrary strings.
Click to expand...

I wish I could change that.
must use this stupid IDN algorithm as base of the project.

webview is processing it correctly so for now , I will have no choice but to 'smell' the code ,create unvisible webview and do something like this:
B4X: 
webview1.loadurl(string1&".com")
Wait For (WebView2_PageFinished(string1&".com")) complete (flag As Boolean)


Private Sub WebView1_PageFinished (Url As String)
string2=webview1.url
string2=string2(Replace ".com","")
end sub

Horrible.



solution must be to convert the string , probably to UCS-4. (UTF-32) , and successfully use the crashing idn lib. (crashing who?...)

how do I do that?

I kind of feel embarrassed from being such a noob and not able to make a working open source code of this lib.
developer.android.com

IDN | Android Developers

developer.android.com developer.android.com

but...as signed below...
 
Last edited:
Upvote 0
so, regarding this...
how would I convert a string (the user string) to Object >> to UTF32 >> use the idn lib >> convert back to string

possible? or did I get completely lost?
found this in the docs :

Available options and values:
* [encoding - Use either UTF-8, UCS4 as array or UCS4 as string as input ('utf8' for UTF-8,
* 'ucs4_string' and 'ucs4_array' respectively for UCS4); The output is always UTF-8]
* [overlong - Unicode does not allow unnecessarily long encodings of chars,
* to allow this, set this parameter to true, else to false;
* default is false.]
* [strict - true: strict mode, good for registration purposes - Causes errors
* on failures; false: loose mode, ideal for "wildlife" applications
* by silently ignoring errors and returning the original input instead

* @param mixed Parameter to set (string: single parameter; array of Parameter => Value pairs)
* @param string Value to use (if parameter 1 is a string)
* @return boolean true on success, false otherwise
Click to expand...
 
Upvote 0
Hi Erel.
I return to this thread since this is something that is not solved in my project.

right now, I send a fake url that contains foreign letters to webview. then, take the error log (url) and extarct the IDN result from it...(webview convert automatically to IDN once it finds a foreign letter in the sent url)

can you make the java idn work ?

so I can slowly recover the embarrassment of using this bad workaround...


right now - it works only if I use letters as parameters. once I use digits - it crashes.

quoting from a resource that I found here
github.com

idna-convert/README.md at main · algo26-matthias/idna-convert

Pure PHP IDNA Converter. Contribute to algo26-matthias/idna-convert development by creating an account on GitHub.
github.com github.com

Unicode strings are expected to be UTF-8 strings. ACE strings (the Punycode form) are always 7bit ASCII strings.
Click to expand...

maybe this is the key?

convert the string or using something else instead of string?


B4X: 
Sub ToACE (s As String) As String
    Dim jo As JavaObject
    jo.InitializeStatic("java.net.IDN")
    Return jo.RunMethod("toASCII", Array(s))
End Sub


Sub FromACE (s As String) As String
    Dim jo As JavaObject
    jo.InitializeStatic("java.net.IDN")
    Return jo.RunMethod("toUnicode", Array(s))
End Sub
 
Last edited:
Upvote 0
1) What version of Android are you running? Looks like there was a change of the underlying library used for IDN between 6 and 7
2) At one time, domain names (any parts) could not start with a digit. Looks like Android's version of IDN is sticking to this
 
Upvote 0
1) What version of Android are you running? Looks like there was a change of the underlying library used for IDN between 6 and 7
Click to expand...
8

2) At one time, domain names (any parts) could not start with a digit. Looks like Android's version of IDN is sticking to this
Click to expand...

I am not interested in domain names rules. only in the coding / encoding of the algorithm itself ,
just as webview is doing right now. try :

B4X: 
webview1.Load ("b4x://65點看.com")



and check the log :

webview1.url = "b4x://xn--65-il8e891l.com
Click to expand...

the ' xn--65-il8e891l ' is what I need.

www.punycoder.com

Punycode Converter — Unicode to IDN Encoder & Decoder 🧰

Convert single labels and complete domain names between Unicode and their ASCII Punycode form. Uses nontransitional Unicode UTS #46 processing and accepts one item per line. The free tool runs in your browser without an account or installation. 🔧🛠
www.punycoder.com www.punycoder.com


b4xמכור-ל
converts to :
xn--b4x--duf9ajm6g


I guess that the string should be modified/converted somehow to something before using it. ( 7bit ASCII strings? ah?!)
 
Last edited:
Upvote 0
trepdas said:
I am not interested in domain names rules. only in the coding / encoding of the algorithm itself ,
Click to expand...
History sometimes does explain why stuff may not work
trepdas said:
just as webview is doing right now. try :
Click to expand...
I would ascertain that webview may use its own IDN encoding mechanism. Not unusual that different teams within the same company/same project use different resources to get things done

Solution(s)?
A) Use another IDN library
B)
trepdas said:
I guess that the string should be modified/converted somehow to something before using it. ( 7bit ASCII strings? ah?!)
Click to expand...
Since it only breaks with leading numbers, why not strip any leading numbers, convert, then insert the numbers back into the string
 
Upvote 0
OliverA said:
Since it only breaks with leading numbers, why not strip any leading numbers, convert, then insert the numbers back into the string
Click to expand...

I tried that. it seems like if I remove the digit and convert only the chars - it converts to something completely different....
 
Upvote 0
You must log in or register to reply here.

Similar Threads

GMan
  • Question Question
B4J Question Make native Slider "stepable" ?
Replies
0
Views
22
GMan
GMan
mcqueccu
  • Question Question
Android Question Conflict in merge for keyvaluestore and sqlcipherb4a when R8optimize is true
Replies
0
Views
36
mcqueccu
mcqueccu
J
  • Question Question
Android Question error QR scanner with R8
Replies
1
Views
59
Erel
Erel
Erel
Server software is being upgraded
Replies
3
Views
140
LucaMs
LucaMs
A
eMedNY
Replies
0
Views
103
Alex_197
A
Back
Top