LlamaIndex

Both are token counts, not characters. chunk_size (default 1024) caps how many tokens a node may hold, measured with the tokenizer LlamaIndex is configured with; chunk_overlap (default 200) repeats that many trailing tokens at the start of the next node.

VectorStoreIndex.from_documents() splits each document into nodes, calls the embedding model once per node, and writes those vectors plus the node text into a vector store — by default an in-memory SimpleVectorStore that vanishes when the process exits.

SimpleDirectoryReader walks a folder, picks a reader based on each file's extension, and returns Document objects with file metadata attached. You narrow what it touches with input_files, required_exts, exclude, exclude_hidden, recursive and num_files_limit.

In LlamaIndex, as_retriever() returns an object whose retrieve() call gives back scored nodes and makes no LLM call. as_query_engine() wraps a retriever with a response synthesizer, so query() also calls the LLM and returns an answer plus its source nodes.

FunctionAgent uses the model's native tool-calling API, so it needs a model that supports it. ReActAgent prompts the model to write reasoning and actions as text and parses them, which works with any model but costs more tokens and can fail to parse.

Metadata is copied onto every node, prepended to the node's text for embedding and for the LLM, and counted against chunk_size — so long metadata shrinks the real text budget and can even make it non-positive. Use excluded_embed_metadata_keys and excluded_llm_metadata_keys to control what leaks where.

FaithfulnessEvaluator checks whether the answer is supported by the retrieved context. RelevancyEvaluator checks whether the retrieved context and answer address the query. CorrectnessEvaluator scores the answer against a labelled reference answer, so only it needs ground truth.

Choose by the shape of the question. VectorStoreIndex suits targeted lookups over a large corpus. SummaryIndex keeps every node in order and is for whole-document synthesis over a small set. TreeIndex spends LLM calls at build to create a summary hierarchy for large-scope questions.

Call index.storage_context.persist(persist_dir="./storage") to write the docstore, index store and default vector store to disk, then rebuild with StorageContext.from_defaults(persist_dir="./storage") and load_index_from_storage(storage_context). Re-calling from_documents() would re-embed everything.

A Document is one whole source item as loaded — a file, a page, an API record. Nodes are the pieces produced from it, each with its own id, metadata, embedding, and a stored reference back to the source Document.

refine calls the LLM once per node, each call improving the running answer. compact, the default, packs nodes into as few context-sized prompts as possible before refining, so it makes far fewer calls. tree_summarize summarizes batches and then summarizes the summaries, bottom-up.

similarity_top_k is how many nodes a LlamaIndex retriever asks the vector store for, defaulting to 2 on VectorIndexRetriever. Raising it buys recall at the cost of prompt tokens, latency and distraction, so the usual pattern is a wide top_k narrowed by a reranker.

Nothing declares the order. Each @step method's parameter type says which event it consumes and its return type says which events it emits, so LlamaIndex wires steps by matching event types. A run starts with StartEvent and ends when a step returns StopEvent.

They are pointers stored in each node's relationships dict that link it to the chunks immediately before and after it in the original document. They let you expand a retrieved chunk back into its surrounding context instead of relying only on chunk_overlap.

Pick TokenTextSplitter when the text has no reliable sentence structure — logs, code, transcripts, scraped markup — and you want tight, uniform token windows. SentenceSplitter is the better default for prose because it refuses to cut mid-sentence.

BatchEvalRunner takes a dict of named evaluators and runs them concurrently over many queries with asyncio, capped by its workers setting. It returns a dict keyed by evaluator name holding one EvaluationResult per row. The cost is judge LLM calls: queries times evaluators.

Subclass BaseReader and implement load_data(), returning a list of Document objects with text, a metadata dict, and a stable id_ derived from the source system. Use lazy_load_data() to stream large sources instead of materialising everything.

Serialize the run's Context with ctx.to_dict() and store it, then rebuild it with Context.from_dict(workflow, data) and pass it to the next run. WorkflowCheckpointer captures per-step snapshots, but they live in memory unless you persist them yourself.

AgentWorkflow gives each agent a handoff tool listing the agents named in its can_handoff_to. Calling it transfers control to that agent, which continues with the same shared Context and conversation history rather than starting fresh.

It chunks each document several times at decreasing sizes and links the levels with PARENT and CHILD relationships. You embed only the leaf nodes returned by get_leaf_nodes, but every node of every level must go into the docstore or the parent links dangle.

It requires an embed_model and embeds sentence groups across the whole corpus before any node exists, so ingestion becomes an extra full embedding pass with the latency, spend and rate limits that implies. It also gives no hard token ceiling on the chunks it emits.

Attach a TokenCountingHandler to a CallbackManager (usually via Settings.callback_manager), give it a tokenizer matching your model, then read prompt_llm_token_count, completion_llm_token_count and total_embedding_token_count. Counters accumulate until you call reset_counts(), so scope or reset them per request.

Install the Phoenix callback integration and call set_global_handler("arize_phoenix") before building anything; for hosted LlamaTrace pass its endpoint and an API key header. Every query then emits a span tree covering retrieval and synthesis, with retrieved node text, scores and the exact prompts.

A VectorStoreIndex build spends one embedding call per node. A PropertyGraphIndex additionally runs LLM extractors over every chunk to pull out entities and relations, so the build is an LLM bill, is far slower, and produces non-deterministic structure that changes between runs.

Give every document a stable id, then call index.refresh_ref_docs(documents): LlamaIndex compares each document's hash against the docstore and re-inserts only what changed. Removals need an explicit index.delete_ref_doc(doc_id, delete_from_docstore=True).

Set excluded_embed_metadata_keys on the Document or node to hide keys from the embedded text, and excluded_llm_metadata_keys to hide keys from the prompt. Verify with get_content(metadata_mode=MetadataMode.EMBED) or MetadataMode.LLM.

Two separate mechanisms: an IngestionCache that skips a transformation when the same input hash and transformation were seen before, and an attached docstore that compares each document's id and hash to decide insert, skip or upsert. Both need stable document ids.

Either use a vector store that supports native hybrid search through as_retriever(vector_store_query_mode="hybrid"), or run a BM25Retriever and a vector retriever side by side and merge them with QueryFusionRetriever using mode="reciprocal_rerank".

Rerankers are node postprocessors: pass CohereRerank or SentenceTransformerRerank in node_postprocessors, and they rescore the retrieved nodes with a cross-encoder and keep only top_n. They run in list order between retrieval and synthesis, and they replace the embedding scores with their own.