Splitters

mash.ai.splitters · AI

Split text for AI processing

Methods

character

splitters.character(chunk_size?: int = 1000, chunk_overlap?: int = 200, separator?: string = "\n\n")

Create a character-based text splitter.

htmlHeaderTextSplitter

splitters.htmlHeaderTextSplitter(headers_to_split_on: array, return_each_element?: bool = false)

Create an HTML header-based text splitter.

htmlSectionSplitter

splitters.htmlSectionSplitter(headers_to_split_on: array)

Create an HTML section splitter.

htmlSemanticPreservingSplitter

splitters.htmlSemanticPreservingSplitter(headers_to_split_on: array, max_chunk_size: int, chunk_overlap: int)

Create an HTML semantic-preserving splitter.

markdownTextSplitter

splitters.markdownTextSplitter(chunk_size?: int = 1000, chunk_overlap?: int = 200, separators?: array = ["\n\n","\n"," ",""])

Create a Markdown-aware text splitter.

recursiveCharacter

splitters.recursiveCharacter(chunk_size?: int = 1000, chunk_overlap?: int = 200, separators?: array = ["\n\n","\n"," ",""])

Create a recursive character text splitter.

recursiveJsonSplitter

splitters.recursiveJsonSplitter(chunk_size?: int = 1000, chunk_overlap?: int = 200)

Create a recursive JSON splitter.