Class TokenTextSplitter

java.lang.Object

org.springframework.ai.transformer.splitter.TextSplitter

org.springframework.ai.transformer.splitter.TokenTextSplitter

All Implemented Interfaces:

Function<List<Document>, List<Document>>, DocumentTransformer

public class TokenTextSplitter
extends TextSplitter

A TextSplitter that splits text into chunks of a target size in tokens.

Author:

Raphael Yu, Christian Tzolov, Ricken Bazolo, Jemin Huh

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static final class	TokenTextSplitter.Builder

Constructor Summary

Constructors

Constructor	Description
TokenTextSplitter()	Deprecated, for removal: This API element is subject to removal in a future version. since 2.0.0-M3, use builder() instead.
TokenTextSplitter(boolean keepSeparator)	Deprecated, for removal: This API element is subject to removal in a future version. since 2.0.0-M3, use builder() instead.
TokenTextSplitter(int chunkSize, int minChunkSizeChars, int minChunkLengthToEmbed, int maxNumChunks, boolean keepSeparator, List<Character> punctuationMarks)	Deprecated, for removal: This API element is subject to removal in a future version. since 2.0.0-M3, use builder() instead.
TokenTextSplitter(com.knuddels.jtokkit.api.EncodingType encodingType)	Deprecated, for removal: This API element is subject to removal in a future version. since 2.0.0-M3, use builder() instead.
TokenTextSplitter(com.knuddels.jtokkit.api.EncodingType encodingType, boolean keepSeparator)	Deprecated, for removal: This API element is subject to removal in a future version. since 2.0.0-M3, use builder() instead.

Method Summary

Modifier and Type	Method	Description
static TokenTextSplitter.Builder	builder()
protected List<String>	doSplit(String text, int chunkSize)	Splits text into chunks based on token count.
protected int	getLastPunctuationIndex(String chunkText)
protected List<String>	splitText(String text)

Methods inherited from class TextSplitter

apply, isCopyContentFormatter, setCopyContentFormatter, split, split

Methods inherited from class Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface DocumentTransformer

transform

Methods inherited from interface Function

andThen, compose

Constructor Details

TokenTextSplitter

@Deprecated(since="2.0.0-M3",
            forRemoval=true)
public TokenTextSplitter()

Deprecated, for removal: This API element is subject to removal in a future version.

since 2.0.0-M3, use builder() instead.

TokenTextSplitter

@Deprecated(since="2.0.0-M3",
            forRemoval=true)
public TokenTextSplitter(boolean keepSeparator)

Deprecated, for removal: This API element is subject to removal in a future version.

since 2.0.0-M3, use builder() instead.

TokenTextSplitter

@Deprecated(since="2.0.0-M3",
            forRemoval=true)
public TokenTextSplitter(com.knuddels.jtokkit.api.EncodingType encodingType)

Deprecated, for removal: This API element is subject to removal in a future version.

since 2.0.0-M3, use builder() instead.

TokenTextSplitter

@Deprecated(since="2.0.0-M3",
            forRemoval=true)
public TokenTextSplitter(com.knuddels.jtokkit.api.EncodingType encodingType,
 boolean keepSeparator)

Deprecated, for removal: This API element is subject to removal in a future version.

since 2.0.0-M3, use builder() instead.

TokenTextSplitter

@Deprecated(since="2.0.0-M3",
            forRemoval=true)
public TokenTextSplitter(int chunkSize,
 int minChunkSizeChars,
 int minChunkLengthToEmbed,
 int maxNumChunks,
 boolean keepSeparator,
 List<Character> punctuationMarks)

Deprecated, for removal: This API element is subject to removal in a future version.

since 2.0.0-M3, use builder() instead.

Method Details

builder

public static TokenTextSplitter.Builder builder()

splitText

protected List<String> splitText(String text)

Specified by:

splitText in class TextSplitter

doSplit

protected List<String> doSplit(String text,
 int chunkSize)

Splits text into chunks based on token count.

Punctuation-based splitting only applies when the token count exceeds the chunk size (tokens.size() > chunkSize). Text that exactly matches or is smaller than the chunk size is returned as a single chunk without punctuation-based truncation.

Parameters:

text - the text to split

chunkSize - the target chunk size in tokens

Returns:

list of text chunks

getLastPunctuationIndex

protected int getLastPunctuationIndex(String chunkText)