Skills Artificial Intelligence Keyword and Embedding Extraction

Keyword and Embedding Extraction

v20261011
extracting-keywords
Extract keywords with YAKE or RAKE, detect document language, and generate vector embeddings for RAG and search using the xberg CLI or Python API, with local, LLM, or plugin providers.
Get Skill
107 downloads
Overview

Extracting keywords, language, and embeddings

Use this for the enrichment surface around extraction: statistical keyword
extraction, language detection, and vector embeddings. Keywords and
language detection ride along with extraction and land on the result;
embeddings are produced by a dedicated embed command.

Keywords (YAKE / RAKE)

Keyword extraction is configured via the [keywords] config block (or
inline JSON) — there is no single --keywords CLI flag. When enabled,
extracted keywords appear on result.extracted_keywords (extractedKeywords
in Node.js; the CLI JSON field is extracted_keywords). Two algorithms are
available:

  • YAKE ("yake") — statistical, unsupervised single-document

extraction. Good general default.

  • RAKE ("rake") — co-occurrence / phrase-based. Favors multi-word

key phrases.

Feature-gated: keyword extraction requires the CLI to be built with the
keywords-yake and/or keywords-rake Cargo features (both are in the
default/full build). If the CLI was built without them, the [keywords]
config block is silently ignored — result.extracted_keywords simply stays empty
rather than erroring. The "yake" algorithm needs keywords-yake; "rake"
needs keywords-rake.

Enable via inline JSON on the CLI:

xberg extract paper.pdf --format json \
  --config-json '{"keywords":{"algorithm":"yake","max_keywords":15,"language":"en"}}' \
  | jq '.extracted_keywords'

Or in a config file:

[keywords]
algorithm = "rake"       # "yake" or "rake"
max_keywords = 10        # default 10
min_score = 0.0          # filter below this score (normalized 0.0-1.0 for both algorithms)
ngram_range = [1, 3]     # unigrams..trigrams (default); config-file only
language = "en"          # stopword language; omit to skip stopword filtering
xberg extract report.pdf --config xberg.toml --format json | jq '.extracted_keywords'

Field notes:

  • max_keywords caps how many keywords are returned (default 10).
  • min_score filters low-scoring keywords. Both YAKE and RAKE normalize

their scores to the 0.0-1.0 range with higher-is-better, so
min_score retains keywords with score >= min_score identically for
either algorithm.

  • ngram_range is [min, max]: [1,1] unigrams only, [1,2] adds

bigrams, [1,3] (default) adds trigrams. Config-file only — it is not a
field on the language bindings' KeywordConfig.

  • language enables stopword filtering for that language; omit it to

disable stopword filtering entirely.

Language detection

Language detection is a real CLI flag: --detect-language. Detected
languages appear on result.detected_languages:

xberg extract multilingual.pdf --detect-language true --format json \
  | jq '.detected_languages'

In a config file it lives under [language_detection]:

[language_detection]
enabled = true
min_confidence = 0.8
detect_multiple = false

The CLI flag enables detection with min_confidence = 0.8 and
single-language mode; use the config block to detect multiple languages or
tune confidence.

Embeddings (embed command)

The standalone embed command produces vector embeddings for text from
--text (repeatable) or stdin. It does not run extraction — pipe
extracted content in if you want document embeddings.

# Local ONNX preset model (default provider)
xberg embed --text "first passage" --text "second passage" --preset balanced

# Embed extracted document text
xberg extract report.pdf | xberg embed --preset quality

Presets for the local provider: fast, balanced (default), quality,
multilingual. Output defaults to JSON (--format json).

--provider selects the embedding source:

ProviderFlagNotes
local`--preset balanced\
llm--model <id> --api-key <key>liter-llm routing, e.g. openai/text-embedding-3-small.
plugin--plugin <name>A backend pre-registered in-process via the plugin API.
# Provider-hosted embeddings via an LLM
xberg embed --text "query text" \
  --provider llm --model openai/text-embedding-3-small --api-key "$OPENAI_API_KEY"

Local embedding presets must be downloaded first if not cached. Pre-warm
them with the cache command:

xberg cache warm --embedding-model balanced   # one preset
xberg cache warm --all-embeddings             # all available presets (currently 8)

Programmatic access

Keywords and detected languages live on the document in the result envelope:

from xberg import ExtractInput, extract, ExtractionConfig, KeywordConfig, KeywordAlgorithm

config = ExtractionConfig(
    keywords=KeywordConfig(algorithm=KeywordAlgorithm.YAKE, max_keywords=15, language="en"),
)
result = await extract(ExtractInput(uri="paper.pdf"), config)
doc = result.results[0]
print(doc.extracted_keywords)   # extracted keywords (when enabled)
print(doc.detected_languages)   # detected languages (when enabled)

See references/python-api.md and references/configuration.md in the
sibling xberg skill for the keyword / language-detection config
classes and the embedding presets.

Common pitfalls

  • No --keywords flag — keyword extraction is config-only. Use

--config-json '{"keywords":{...}}' or a [keywords] config block.

  • min_score direction — scores are normalized to 0.0-1.0 with

higher-is-better for both YAKE and RAKE, so the same threshold behaves
identically for either algorithm.

  • Embeddings ≠ extraction — embed only takes raw text. Pipe

xberg extract output into it for document vectors.

  • Cold embedding models — first local run downloads the preset; run

xberg cache warm --all-embeddings to pre-populate.

See references/advanced-features.md for the embeddings pipeline and
references/cli-reference.md for the embed and cache warm flag sets.

Info
Name extracting-keywords
Version v20261011
Size 6.6KB
Updated At 2026-10-12
Language