B4J Question Counting Tokens for OpenAI

[Tim Chapman]

I need to be able to count tokens in a user's text submission (before it is submitted) to OpenAI API.

I have researched and found that TikToken is the best way to do that.
There are two TikToken libraries available at this time. One is in Python. The other in Node.js. Those are shown here:

OpenAI API: How do I count tokens before(!) I send an API request?

OpenAI's text models have a context length, e.g.: Curie has a context length of 2049 tokens. They provide max_tokens and stop parameters to control the length of the generated sequence. Therefore the

stackoverflow.com

What is the easiest way for me to get TikToken into B4J?

The closest I have seen is to compile a Node.js project into a jar file as shown here:

How to compile nodejs project into jar-Front-end Q&A-php.cn

Node.js is a very popular JavaScript runtime environment. Large web frameworks such as Vue.js and React Native run on Node.js. However, for Java programmers, compiling Node.js projects into Jar packages may be more convenient for integration into Java projects. In this article, we will introduce...

www.php.cn

This looks like the correct JS code to work with:

tiktoken/js at main · dqbd/tiktoken

JS port and JS/WASM bindings for openai/tiktoken. Contribute to dqbd/tiktoken development by creating an account on GitHub.

github.com

Any better answers anyone can see for this? Doing the conversion using the instructions above is a bit out of my league.

 

[jkhazraji]

Send the text directly to the OpenAI API although it will consume your quota.

 

[Tim Chapman]

Send the text directly to the OpenAI API although it will consume your quota.

I need the token count before sending to OpenAI API. That is why I need TikToken in B4J.

 

[DonManfred]

GitHub - knuddelsgmbh/jtokkit: JTokkit is a Java tokenizer library designed for use with OpenAI models.

JTokkit is a Java tokenizer library designed for use with OpenAI models. - knuddelsgmbh/jtokkit

github.com

Download jar

https://mvnrepository.com/artifact/com.knuddels/jtokkit/1.1.0

You probably can use it with JavaObject

 

[Tim Chapman]

GitHub - knuddelsgmbh/jtokkit: JTokkit is a Java tokenizer library designed for use with OpenAI models.

JTokkit is a Java tokenizer library designed for use with OpenAI models. - knuddelsgmbh/jtokkit

github.com

Download jar

https://mvnrepository.com/artifact/com.knuddels/jtokkit/1.1.0

You probably can use it with JavaObject

Hi Don,

I have delayed replying to you until I try to use the jar file you recommended. I apologize for not thanking you sooner. I have had to take a detour and work on software for my business. It is now giving me trouble which I can't figure out. I am posting in the B4J forum about it. When I get back to my OpenAI project, I will try to use the jar file you suggested and will let you know what I find.

I posted my question here: https://www.b4x.com/android/forum/threads/combobox-not-showing-anything-when-i-click-on-it.163439/

 

[DonManfred]

I posted my question here

See my answer/solution in that that. ;-)

 

[DonManfred]

until I try to use the jar file you recommended

you are in luck. i already wrote a small wrapper for it ?

Download the jar from the link in #4 and put it into the additional libs folder

Basically you need to have an

    Private tokkit As EncodingRegistry

and then use it

    tokkit.Initialize("") ' Eventname is not used
    tokkit.SetModelEncoding("GPT_3_5_TURBO")
    
    Dim tokens As List = tokkit.encode("This is a Test!")
    Log(tokens)
    
    Dim decoded As String = tokkit.decode(tokens)
    Log($"Decoded ${decoded}"$)

 

[Tim Chapman]

you are in luck. i already wrote a small wrapper for it ?

Download the jar from the link in #4 and put it into the additional libs folder

Basically you need to have an

    Private tokkit As EncodingRegistry

and then use it

    tokkit.Initialize("") ' Eventname is not used
    tokkit.SetModelEncoding("GPT_3_5_TURBO")
   
    Dim tokens As List = tokkit.encode("This is a Test!")
    Log(tokens)
   
    Dim decoded As String = tokkit.decode(tokens)
    Log($"Decoded ${decoded}"$)

Thank you Don!