ICUB4J
When experimenting with subtitles, I noted that very often characters were not displayed correctly due to wrong character-encoding being used during the loading of a subtitle file to display. You have surely all seen and heard of UTF-8, ISO-8859-1 and so forth. These abbreviations represent various character-encoding formats.
Unfortunately, there is no 100% method for a program/application to know which character-encoding format to use when loading a file unless the file-format provides it. Therefore, most software uses a detection-algorithm to guess the character-encoding format in use.
I tried several detection-algorithms but at the end, I decided to use ICU4J which is continously updated and maintained and which according to my tests furnished the best results/guesses. The size of the original ICU4J-library is very large (it provides other functionaility as well) so I made a wrapper for B4A and B4J using a subset of the APIs available (total size is approximately 75KB).
Anyway - what is ICU4J?
ICU4J is an open-source, widely used set of Java libraries providing Unicode and globalization support for software applications.
Java provides a very strong foundation for global programs, and IBM and the ICU team played a key role in providing globalization technology into Sun's Java. But because of its long release schedule, Java cannot always keep up-to-date with evolving standards. The ICU team continues to extend Java's Unicode and internationalization support, focusing on improving performance, keeping current with the Unicode standard, and providing richer APIs, while remaining as compatible as possible with the original Java text and internationalization API design.
Companies that use ICU4J are for instance Google, Apple, Adobe, Amazon (Kindle), Debian and so forth. Google has announced that with Android N they will also include a subset of ICU4J's APIs.
Usage in B4J:
I'm attaching sample-project, test-files and library. You can use this library with any text-files.
I have also posted a wrapper for B4A which you can find here.
I hope it may be useful for someone.
When experimenting with subtitles, I noted that very often characters were not displayed correctly due to wrong character-encoding being used during the loading of a subtitle file to display. You have surely all seen and heard of UTF-8, ISO-8859-1 and so forth. These abbreviations represent various character-encoding formats.
Unfortunately, there is no 100% method for a program/application to know which character-encoding format to use when loading a file unless the file-format provides it. Therefore, most software uses a detection-algorithm to guess the character-encoding format in use.
I tried several detection-algorithms but at the end, I decided to use ICU4J which is continously updated and maintained and which according to my tests furnished the best results/guesses. The size of the original ICU4J-library is very large (it provides other functionaility as well) so I made a wrapper for B4A and B4J using a subset of the APIs available (total size is approximately 75KB).
Anyway - what is ICU4J?
ICU4J is an open-source, widely used set of Java libraries providing Unicode and globalization support for software applications.
Java provides a very strong foundation for global programs, and IBM and the ICU team played a key role in providing globalization technology into Sun's Java. But because of its long release schedule, Java cannot always keep up-to-date with evolving standards. The ICU team continues to extend Java's Unicode and internationalization support, focusing on improving performance, keeping current with the Unicode standard, and providing richer APIs, while remaining as compatible as possible with the original Java text and internationalization API design.
Companies that use ICU4J are for instance Google, Apple, Adobe, Amazon (Kindle), Debian and so forth. Google has announced that with Android N they will also include a subset of ICU4J's APIs.
Usage in B4J:
B4X:
Sub Process_Globals
'Private fx As JFX
Private MainForm As Form
Private guessEncoding As IcuB4J
End Sub
Sub AppStart (Form1 As Form, Args() As String)
'MainForm = Form1
'MainForm.SetFormStyle("UNIFIED")
'MainForm.RootPane.LoadLayout("Layout1") 'Load the layout file.
'MainForm.Show
Try
Dim fileName As String
Dim fileLocation As String
Dim detectionResult As String
fileName = "bbchinese.srt"
fileLocation = File.Combine(File.DirApp,fileName)
detectionResult = guessEncoding.readFileAsStringGuessEncoding(fileLocation)
Log(detectionResult) ' --> UTF-8
Catch
Log(LastException)
End Try
End Sub
I'm attaching sample-project, test-files and library. You can use this library with any text-files.
I have also posted a wrapper for B4A which you can find here.
I hope it may be useful for someone.
Attachments
Last edited: