技能 人工智能 LLM 上下文文本分块

LLM 上下文文本分块

v20261016
xberg-chunking
该技能用于将提取的文本分割为适合 LLM 上下文窗口或 RAG 检索增强的块。支持配置块大小、重叠设置,并提供文本、Markdown、YAML 及语义分块等多种策略。适用于文档处理、向量存储准备及模型上下文优化,支持基于字符或 Token 的尺寸控制。
获取技能
189 次下载
概览

Chunking

Use this when feeding documents into an LLM context window or a vector
store. Xberg chunks two ways: inline during extraction (chunks land on
each document's chunks field), or standalone via the chunk command for text you
already have. Sizing is character-based by default, or token-based when a
tokenizer model is supplied.

Inline during extraction

Turn on chunking with --chunk and the chunks appear on the structured
result under chunks:

# 1000-char chunks, 200-char overlap (defaults when --chunk is on)
xberg extract report.pdf --chunk --format json | jq '.result.chunks | length'

# Explicit size + overlap
xberg extract report.pdf --chunk --chunk-size 1500 --chunk-overlap 300 --format json

Overlap must be smaller than chunk size — the CLI rejects
--chunk-overlap >= --chunk-size. When you set only --chunk-overlap
against an existing config, an overlap that exceeds the size is clamped to
chunk_size / 4.

Standalone chunk command

Chunk text you already have, from --text or stdin. Output defaults to
JSON:

# From a flag
xberg chunk --text "long document text ..." --chunk-size 800 --chunk-overlap 100

# From stdin (pipe extracted content straight in)
xberg extract notes.md | xberg chunk --chunk-size 500 --format json

JSON output carries chunks (array of strings), chunk_count, the
resolved config (max_characters, overlap, chunker_type), and
input_size_bytes. Use --format text for a human-readable dump with
--- chunk N --- separators.

Note: in the JSON output, chunker_type is rendered capitalized ("Text",
"Markdown", "Yaml", "Semantic") because it is emitted via Rust's Debug
formatting, whereas the --chunker-type input flag is lowercase
(text, markdown, yaml, semantic). Lowercase the value before
comparing if you parse it back.

Chunker types

--chunker-type selects the splitting strategy (standalone chunk
command):

TypeBehavior
textDefault. Plain character-window splitting with overlap.
markdownMarkdown-aware — splits on structure (headings, blocks) where possible.
yamlYAML-aware splitting for structured config/data documents.
semanticTopic-boundary splitting driven by --topic-threshold (0.0–1.0, default 0.75).
# Markdown-aware chunking keeps headings and blocks intact
xberg chunk --text "$(cat README.md)" --chunker-type markdown

# Semantic chunking — lower threshold = more, smaller topic chunks
xberg chunk --text "$(cat transcript.txt)" --chunker-type semantic --topic-threshold 0.6

Token-based sizing

By default --chunk-size counts characters. To size chunks by tokens for
a specific model, pass --chunking-tokenizer with a HuggingFace tokenizer
id. On the extract command this implicitly enables chunking. Requires the
chunking-tokenizers feature (present in the default CLI build).

# Size chunks by GPT-4o tokens during extraction
xberg extract report.pdf --chunking-tokenizer Xenova/gpt-4o --format json

# Or on the standalone command
xberg chunk --text "$(cat doc.txt)" --chunking-tokenizer Xenova/gpt-4o --chunk-size 512

With a tokenizer set, --chunk-size is interpreted in tokens, not
characters.

Config file alternative

Field names in config files are snake_case under [chunking]:

[chunking]
max_characters = 1000
overlap = 200
chunker_type = "markdown"
xberg extract report.pdf --config xberg.toml --format json
CLI flags map to config fields as --chunk-size → max_characters and
--chunk-overlap → overlap. In config files use the snake_case names.

Programmatic access

From Python, enable chunking on the config and read the chunks off the
document in the result envelope (result.results[0].chunks):

from xberg import ExtractInput, extract, ExtractionConfig, ChunkingConfig

config = ExtractionConfig(
    chunking=ChunkingConfig(max_characters=1000, overlap=200),
)
result = await extract(ExtractInput(uri="report.pdf"), config)
for chunk in result.results[0].chunks or []:
    print(len(chunk.content))
The public Python ChunkingConfig (a dataclass) uses constructor kwargs
max_characters / overlap; the Rust core struct fields are also
max_characters / overlap. TOML/JSON config keys are max_chars /
max_overlap (with max_characters / overlap accepted as serde aliases),
and dict-form config passed to ExtractionConfig likewise accepts the
max_chars / max_overlap aliases; Node's ChunkingConfig interface uses
maxCharacters / overlap. See references/python-api.md and
references/rust-api.md.

Picking parameters

  • RAG / vector store — 500–1000 chars (or 256–512 tokens) with

10–20% overlap. Use markdown chunking for docs to keep sections whole.

  • LLM summarization — larger chunks (1500–4000 chars) with small

overlap; size by tokens to stay under the model window.

  • Topic segmentation — semantic chunker; tune --topic-threshold

down for finer splits, up for coarser ones.

Common pitfalls

  • Overlap ≥ size — rejected on extract; clamped to size / 4 when

only overlap is changed against an existing config.

  • Tokenizer without the feature — --chunking-tokenizer errors if the

CLI was built without chunking-tokenizers. The default build includes it.

  • Empty input — the standalone chunk command bails on empty text;

provide --text or pipe non-empty stdin.

See references/configuration.md for the full [chunking] schema and
references/cli-reference.md for every chunk flag.

信息
分类 人工智能
名称 xberg-chunking
版本 v20261016
大小 46.86KB
更新时间 2026-10-17
语言