com.lzhpo.chatgpt.utils

类 TokenUtils

java.lang.Object

com.lzhpo.chatgpt.utils.TokenUtils

public class TokenUtils
extends Object

How_to_count_tokens_with_tiktoken

 
   def num_tokens_from_messages(messages, model="gpt-3.5-turbo-0301"):
     """Returns the number of tokens used by a list of messages."""
     try:
         encoding = tiktoken.encoding_for_model(model)
     except KeyError:
         print("Warning: model not found. Using cl100k_base encoding.")
         encoding = tiktoken.get_encoding("cl100k_base")
     if model == "gpt-3.5-turbo":
         print("Warning: gpt-3.5-turbo may change over time. Returning num tokens assuming gpt-3.5-turbo-0301.")
         return num_tokens_from_messages(messages, model="gpt-3.5-turbo-0301")
     elif model == "gpt-4":
         print("Warning: gpt-4 may change over time. Returning num tokens assuming gpt-4-0314.")
         return num_tokens_from_messages(messages, model="gpt-4-0314")
     elif model == "gpt-3.5-turbo-0301":
         tokens_per_message = 4  # every message follows <|start|>{role/name}\n{content}<|end|>\n
         tokens_per_name = -1  # if there's a name, the role is omitted
     elif model == "gpt-4-0314":
         tokens_per_message = 3
         tokens_per_name = 1
     else:
         raise NotImplementedError(f"""num_tokens_from_messages() is not implemented for model {model}. See https://github.com/openai/openai-python/blob/main/chatml.md for information on how messages are converted to tokens.""")
     num_tokens = 0
     for message in messages:
         num_tokens += tokens_per_message
         for key, value in message.items():
             num_tokens += len(encoding.encode(value))
             if key == "name":
                 num_tokens += tokens_per_name
     num_tokens += 3  # every reply is primed with <|start|>assistant<|message|>
     return num_tokens
 
 

作者:

lzhpo

构造器概要

构造器

构造器和说明
TokenUtils()

方法概要

限定符和类型	方法和说明
static com.knuddels.jtokkit.api.Encoding	getEncoding(com.knuddels.jtokkit.api.ModelType modelType) Returns the encoding that is used for the given model type.
static com.knuddels.jtokkit.api.EncodingRegistry	getRegistry() Get the REGISTRY.
static Long	tokens(com.knuddels.jtokkit.api.ModelType modelType, String content) Encodes the content into a list of token ids and returns the amount of tokens.
static Long	tokens(String model, List<ChatCompletionMessage> messages) Encodes the messages into a list of token ids and returns the amount of tokens.
static Long	tokens(String modelTypeName, String content) Encodes the content into a list of token ids and returns the amount of tokens.

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

构造器详细资料

TokenUtils

public TokenUtils()

方法详细资料

getRegistry

public static com.knuddels.jtokkit.api.EncodingRegistry getRegistry()

Get the REGISTRY.

返回:

EncodingRegistry

getEncoding

public static com.knuddels.jtokkit.api.Encoding getEncoding(com.knuddels.jtokkit.api.ModelType modelType)

Returns the encoding that is used for the given model type.

参数:

modelType - ModelType

返回:

the encoding

tokens

public static Long tokens(com.knuddels.jtokkit.api.ModelType modelType,
                          String content)

Encodes the content into a list of token ids and returns the amount of tokens.

参数:

modelType - ModelType

content - content

返回:

the tokens

tokens

public static Long tokens(String modelTypeName,
                          String content)

Encodes the content into a list of token ids and returns the amount of tokens.

参数:

modelTypeName - ModelType name

content - content

返回:

the tokens

tokens

public static Long tokens(String model,
                          List<ChatCompletionMessage> messages)

Encodes the messages into a list of token ids and returns the amount of tokens.

参数:

model - model

messages - messages

返回:

tokens