Split text into chunks under a hard token cap without cutting a table, sentence or code fence

Overview

Plans chunk boundaries for text an agent must feed to a model in pieces: every chunk is at most maxTokens in the named tokenizer (o200k_base or cl100k_base, verified by counting each finished chunk), boundaries fall only between sentences, blocks, whole Markdown or HTML tables and whole code fences, headings stay with the text that follows, and the chunks tile the input exactly (concatenating the spans gives the original). Spans carry offsets in UTF-16, code points, UTF-8 byt

Health