Incremental Indexing: Keeping Embedding Stores Fresh

Embedding stores are derived, lossy projections of a mutable source corpus, and "freshness" in retrieval-augmented systems is a reconciliation contract between that corpus and its vector index, not a property of the database. This article treats the spectrum of incremental indexing strategies — delta indexing, time-windowed shards, append-only with compaction, change-data-capture pipelines — as graded responses to scale and latency pressure, and argues that the right default for most systems is rebuildable indexing with incremental optimization layered on top, not the inverse.

Coverage note: verified through May 6, 2026.

The materialized-view frame

The most common architectural mistake in retrieval-augmented systems is treating the vector index as if it were the source of truth. It is not. An embedding is a vector representation that preserves "aspects" of meaning, and choosing a smaller dimensionality is explicitly a tradeoff against accuracy (OpenAI embeddings concept, OpenAI embeddings guide). Once you have decided to embed, you have already decided to discard information. The store that holds those vectors is a derived artifact — a Materialized View over a canonical corpus, not the corpus itself.

This framing has immediate consequences. A materialized view depends on stable identity for the rows it derives from, deterministic transformations from source to projection, and a reconciliation strategy when the source changes. Drop any of these and "incremental indexing" stops being an engineering optimization and becomes a guess about whether the index still describes the world. Most production retrieval failures trace back to violations of one of those three properties, not to the choice of HNSW versus IVF or cosine versus dot-product.

The lossy-compression metaphor sharpens the same point. Chunking discards document structure. Single-vector encoding blurs detail across a token window. Reranker metadata, parser output, and OCR confidence are routinely dropped at the index boundary. A change in any upstream stage — a new chunker, a new embedding model, a fixed bug in the PDF extractor — silently invalidates vectors whose source documents have not changed at all. Mixedbread's own engineering writeup on billion-scale retrieval emphasizes that semantic-search failures often look plausible and arrive from any stage in the pipeline: parsing, chunking, embedding, metadata extraction, indexing, or scoring (Mixedbread billion-scale). Treating the embedding store as the place where freshness is reconciled — rather than as one of several derived artifacts that must be re-derived when their inputs change — pushes complexity to the wrong layer.

The article's working definition: an embedding store is fresh when, for every document and chunk in the canonical corpus at time t, a query at time t + Δ returns a result computed from the current source content, current chunking, current embedding model, and current access-control state, where Δ is the system's published freshness budget. Anything else is an SLA violation, even if the index "has" the document.

Why full reindex fails — and the cases where it doesn't

The standard argument for incremental indexing is that full reindex does not scale. This is true at certain scales and for certain change patterns, and substantially overstated at others.

The strongest evidence for the cost claim comes from production case studies. Notion's two-year writeup describes a search index that grew through hundreds of millions, then billions, of objects, with online updates flowing through Kafka and offline backfills running on Spark. The team eventually found that re-embedding whole pages on small edits was wasteful enough to motivate a span-level architecture: hashing page spans and their metadata separately reduced indexing data volume by roughly 70% and was the precondition for further cost reductions (Notion vector search). At that scale, full rebuild is not a fallback option for routine change — it is a migration event reserved for model upgrades, schema changes, or corruption.

Research on dynamic ANN maintenance shows the same pressure from a different angle. SPFresh's central claim is that periodic global rebuilds at billion-vector scale are resource-heavy and disrupt both latency and accuracy during the rebuild window, motivating in-place incremental updates (SPFresh paper, Microsoft Research SPFresh). Ada-IVF makes the related point that static IVF-style indexes degrade under continuous updates unless they are reconstructed, and the reconstruction itself is the bottleneck the paper aims to remove (Ada-IVF). Azure's writeup on streaming DiskANN frames inserts, deletes, and replaces as fundamentally unlike one-shot ingestion: graph indices may need links repaired after deletion to preserve recall, and that repair work is what makes streaming workloads hard (Azure Cosmos DB DiskANN streaming).

But the cost-of-rebuild argument generalizes badly. For a personal wiki, a company handbook, an internal product corpus of low millions of chunks, or a research codebase, a nightly or content-hash-triggered rebuild can be cheaper, more auditable, and easier to roll back than a CDC pipeline with idempotent chunk updates and tombstone compaction. Rebuild cost is not a universal: it depends on corpus size, embedding API price per token, index build wall-time, serving disruption tolerance, and the recall budget. An article that asserts "full reindex is too expensive" without naming those variables is laundering a billion-scale conclusion into systems where it does not apply.

The honest position is that rebuildability is the baseline, not the exception. A retrieval system that cannot be rebuilt deterministically from a manifest of source documents, content hashes, parser version, chunker version, embedding model version, and metadata schema is missing something the incremental layer cannot supply: the ability to recover from pipeline drift. Incremental indexing is a layer on top of that baseline that buys lower freshness latency in exchange for operational complexity. It is justified when the freshness SLA is tighter than rebuild wall-time, when corpus size makes rebuild cost dominate, or when the change rate exceeds what rebuild cadence can absorb — and it is unjustified, or at least premature, in systems where none of those pressures bind.

This is the central architectural claim of the article: incremental indexing is an optimization over rebuildable indexing, not a replacement for it. Systems that adopt it without preserving the rebuild path tend to discover, on their first parser-version migration or embedding-model upgrade, that they have lost the ability to restore semantic uniformity across the index.

Stable identity is the prerequisite

Every incremental-indexing pattern below assumes a layer of stable identity that most ad-hoc pipelines lack. Without it, "delta indexing" devolves into reinserting entire documents whose chunk boundaries have shifted by one token, and "deletion" devolves into hoping that orphaned vectors do not surface in top-k.

The minimum identity stack:

Layer	Field	Purpose
Document	`document_id`	Stable across renames, moves, edits
Document	`source_uri`	Where the canonical content lives
Document	`source_version`	Monotonic version or modification timestamp
Document	`content_hash`	Deterministic hash of normalized source content
Document	`acl_state`	Permission-set identifier or hash
Chunk	`chunk_id`	Stable across re-chunking when content didn't move
Chunk	`chunk_hash`	Hash of chunk text plus boundary metadata
Chunk	`chunk_offset`	Byte or token range in source document
Pipeline	`parser_version`	Which parser produced the text
Pipeline	`chunker_version`	Which chunking strategy and parameters
Pipeline	`embedding_model`	Model name, version, dimensionality
Pipeline	`index_build_id`	Which pipeline run produced these vectors

Two of these deserve attention because they are the most often overlooked.

Chunk identity is the load-bearing one. If chunk IDs are derived purely from (document_id, chunk_index), a one-character edit at the top of a document re-chunks everything below it and changes every downstream ID. The result is a re-embed and reinsert of the whole tail of the document — and an orphan-vector problem if the old chunks are not explicitly tombstoned. Notion's span-level hashing is one solution: identify chunks by content hash plus position, so unchanged spans keep their identity even when neighbors change (Notion vector search). Diff-aware chunkers that align edits to chunk boundaries are another. The principle is that chunk identity should track content, not ordinal position, or the system pays an "edit blast radius" tax on every minor update.

Pipeline version is the one that prevents mixed-generation indexes. When a chunker is changed, when a parser fixes an OCR bug, when an embedding model is upgraded, every old vector is in principle invalid — the source document has not changed, but the projection from source to vector has. Without a parser_version and embedding_model recorded on each vector, there is no way to ask "which vectors were produced by the old pipeline and need re-derivation?" The result is an index that mixes vectors from incompatible representation spaces, which silently degrades recall in ways that do not show up in any single query.

The identity stack is what makes both rebuild and incremental update tractable. Rebuild becomes "iterate the canonical document table, re-derive everything, write to a new index, swap the alias." Incremental becomes "for each changed document, diff against content_hash, identify which chunk hashes are new or removed, re-embed only those, upsert by chunk_id, tombstone the rest." Without it, both paths become guessing.

The approach spectrum

With stable identity in place, the design space for incremental indexing covers six recognizable patterns. They are not exclusive: production systems usually combine two or three.

Approach	What it is	When it fits	Failure mode
Full reindex	Rebuild the entire index from the canonical manifest on a schedule or trigger	Small-to-mid corpora; pipeline-version migrations; recovery from drift	Wall-time and cost grow with corpus; serving disruption during cutover unless blue-green
Delta indexing	Detect changed documents (hash, mtime, version), re-derive only those	Most pipelines; the default optimization	Orphan vectors when chunks shift; permission-only changes invisible to content hashing
Time-windowed shards	Partition the index by ingestion time; rebuild or expire windows independently	Append-heavy streams with natural decay (logs, news, agent memory)	Cross-window queries pay shard fan-out; rare historical lookups slow
Append-only with periodic compaction	New writes go to an append log or hot shard; merges produce compacted segments offline	High write rates with deletes/updates rare or batched	Deleted content remains queryable until compaction; read amplification across segments
Change-data-capture (CDC)	Subscribe to a source database's WAL or replication stream; transform rows into chunk operations	Sources that are themselves databases (Postgres, Mongo, etc.) with a real change log	Row-level changes do not map cleanly to chunk-level changes; non-database sources excluded
Hot delta overlay + cold compacted index	A small recent-writes overlay merged with a large stable index at query time	Tight read-after-write requirements with low overlay churn	Query-side merge complexity; overlay must stay small or query latency suffers

Several patterns deserve elaboration because their failure modes are not obvious from the table.

Delta indexing and the chunking problem

Delta indexing is the default optimization for systems that have outgrown nightly rebuilds. The mechanism is a content hash check: for each document in the source manifest, compute the current hash, compare against the index's recorded hash, and re-derive only documents whose hash has changed. This is straightforward at the document level. It is not straightforward at the chunk level.

Consider a document whose chunker produces 40 chunks. Edit the third paragraph. If the chunker is position-based (every N tokens), every chunk after the edit may shift, and every chunk hash may change. The "delta" for this document is now 38 chunks, not the one paragraph that actually changed. Worse, the old 40 chunks are still in the index unless explicitly removed, and the new 38 are inserted alongside them. The result is duplicate content under different chunk IDs, retrievable by similarity to overlapping query text, with no way for the ranker to know they describe the same source.

Three patterns address this. First, content-hashed chunking: chunk IDs are derived from a hash of the chunk text and surrounding context, so unchanged content keeps its ID even when neighbors change. Notion's span-level approach is the production reference here, and it is the architectural change that unlocked the 70% data-volume reduction (Notion vector search). Second, diff-aware chunkers that align chunk boundaries to the structural seams of the document (headings, paragraph breaks, code-block edges), reducing the blast radius of point edits. Third, explicit tombstoning by document version: when a document's source_version changes, mark all chunks at the previous version as deleted, then insert the new version's chunks. This is simpler but reverts to the "re-embed every chunk on any edit" tax that delta indexing was supposed to avoid.

The choice is a tradeoff between identity sophistication and write amplification. Most teams reach for the third option, discover the cost, and then back into one of the first two.

Time-windowed shards

For corpora with a natural time axis — chat logs, news articles, agent memory, support tickets, marketplace listings — time-windowed shards are the cleanest fit. Each shard covers a window (a day, a week, a month), is written to actively while it is the "current" window, and becomes immutable when the window closes. Old windows are expired by deletion of the shard, not by point deletes within it. Qdrant documents this pattern explicitly for time-series workloads, including the use of custom shard keys to direct writes to the correct window (Qdrant time-based sharding).

The benefits are significant: the active write surface is small, full rebuilds are bounded to a single window, and data lifecycle management is just shard deletion. The cost is that queries spanning multiple windows must fan out across shards, and any query that lacks a time predicate effectively touches the entire history. For agent-memory systems where recency dominates, this is a feature; for legal or compliance workloads where the question is "find every document mentioning X across all of history," it is a problem.

Time-windowed shards also do not solve the deletion problem within a window. If a document inside an active window is deleted or its permissions revoked, the system still needs a within-shard mechanism — typically tombstones with periodic compaction. The pattern reduces but does not eliminate the underlying mutation surface.

Append-only with compaction

Pinecone's serverless architecture is the most documented production example of an append-only design at vector-database scale. Writes land in a write-ahead log and a memtable, which together form a hot region; periodically, those writes are compacted into immutable slabs that form the cold region; queries merge results across both (Pinecone serverless). The pattern is borrowed from LSM-tree-style key-value stores, adapted to ANN structures.

This works well for the consistency-versus-latency tradeoff because it makes the tradeoff explicit and tunable. Writes are fast because they only append. Reads are slower than a single static index because they must merge across the hot and cold regions, but the latency cost is bounded by the size of the hot region. Deletions are tombstones until compaction reclaims the space. Compaction itself can be scheduled to avoid serving disruption, and graph or partition repair (the part that makes streaming ANN hard) happens during compaction, not during user-facing reads or writes.

The failure mode is compaction backlog. If the write rate exceeds the compaction throughput, the hot region grows, query latency grows with it, and at some point the system either pauses writes or accepts degraded reads. Production deployments need a metric on hot-region size and a clear policy for what happens when the budget is breached. Solr's documentation of segment-merge thresholds is the same pattern in a different domain (Solr segments and merging); Elasticsearch's near-real-time search guarantee is the user-facing consequence of append-then-refresh semantics (Elasticsearch near-real-time). Vector indexes inherit the timing model and add ANN-graph-quality concerns on top.

Change-data-capture, and what it doesn't capture

CDC pipelines are appealing because they reuse infrastructure that already exists for database replication: AWS DMS supports ongoing replication via CDC (AWS DMS CDC); Debezium captures row-level changes from Postgres, MySQL, and others through their write-ahead logs (Debezium docs). For embedding stores fed primarily from a transactional database, CDC is the natural way to get sub-minute freshness without polling.

The pitfalls are specific. A database row change is not a semantically valid chunk change. Updating a last_modified timestamp triggers a CDC event but does not change the document's content. Changing a single field in a row that contributes to a multi-field document chunk requires a per-document re-derivation, not a per-row update. CDC delivers atomic row deltas; the embedding pipeline needs document-level diffs. The translation layer between them — the consumer that turns a stream of row updates into a stream of chunk operations — is where most of the engineering complexity lives, and it is the layer most likely to ship with subtle bugs around batching, ordering, and idempotency.

CDC also does not help when the source is not a database. Filesystem corpora, SaaS APIs, object stores, and document management systems each need their own change-detection strategy. mgrep's watch command is the filesystem analog: it runs as a background process that detects file changes and re-indexes affected files into a Mixedbread store (mgrep README). For SaaS sources without webhooks or change feeds, the only option is periodic polling with content hashing — effectively delta indexing on a polling schedule.

Hot delta overlays

The hot-overlay pattern combines a small, fresh, frequently-rebuilt index with a large, stable, occasionally-rebuilt index, merged at query time. New writes land in the overlay. The overlay is rebuilt or replaced on a short cadence (minutes). The cold index is rebuilt rarely (days or weeks). Queries hit both and merge results.

This is the cleanest answer to read-after-write requirements where the rest of the corpus is too large to rebuild on every change. It is also the most operationally complex of the patterns: query-side merging requires either a system that natively supports multiple indexes per query (which most managed vector stores do not expose cleanly) or an application-layer fan-out and merge. It introduces a second consistency horizon (the overlay's lag) on top of the cold index's. And it requires a graduation path — when does an item move from the overlay to the cold index? — that has to handle the case where the cold rebuild has not happened yet but the overlay is full.

In practice, hot overlays show up most often in two contexts: very large corpora where cold rebuild is a measured-in-days operation, and agent-memory systems where the fresh writes are the entire point of the query workload and the cold index is mostly a reference corpus. Outside those contexts, the operational complexity often outweighs the freshness gain compared to a more aggressive append-with-compaction design.

Updates and deletions in vector indexes

A persistent piece of folk wisdom in retrieval engineering is that "vector databases don't support updates and deletes." This was approximately true in 2020. It is no longer accurate, and the article-shaped version of the claim ("native vector indexes are inherently append-only") needs to be retired. The actual landscape is more nuanced: most production vector stores support some form of mutation, but the semantics, latency, and recall consequences vary significantly.

System	Update	Delete	Async?	Notes
Qdrant	Vector and payload updates supported via point operations	Point delete and delete-by-filter	Mutations are asynchronous unless the caller waits on the operation	Qdrant points
Weaviate	Object and vector updates supported	Object delete	Async indexing path queues imports, deletes, and updates; explicit consistency tradeoffs documented	Weaviate update, Weaviate vector index, Weaviate consistency
Milvus	Upsert supported	Logical delete with later compaction reclamation	Tunable consistency levels because recent streaming data may not be visible without latency tradeoffs	Milvus consistency, Milvus FAQ
Pinecone	Upsert supported in serverless and pod-based	Delete by ID and by filter	Append-only internally with compaction; merge across WAL/memtable/slabs at query	Pinecone serverless
OpenAI vector stores	File-level operations; chunks managed by the platform	File removal	Vector-store file operations are async; removals are eventually consistent	OpenAI retrieval guide
Mixedbread stores	External-ID upsert workflows	File-level delete	Files move through `pending` → `in_progress` → `completed` before being searchable	Mixedbread ingestion, Mixedbread concepts
FAISS (raw)	Reinsert; some index types support `remove_ids`	`remove_ids` on supported index types only; may scan the full database	Synchronous, but practical use requires periodic rebuild	FAISS special operations

Several distinctions matter more than the matrix suggests.

Logical versus physical deletion. A logical delete marks a vector as removed but leaves it in the index until compaction or merge reclaims the space. The vector is filtered out of query results by the ID list. A physical delete removes the vector from the underlying graph or partition, which may require repairing graph links to preserve recall. Most managed systems do logical deletion immediately and physical deletion lazily; this is the right tradeoff for write throughput, but it means the system carries deleted content as visible-to-storage-but-invisible-to-queries until compaction runs. For privacy and right-to-delete obligations, the gap between logical and physical deletion is the gap between "the user can no longer retrieve their data" and "the data is no longer present" — which are distinct legal claims.

Update versus reinsert. An "update" in many vector stores is implemented as a delete followed by an insert under the same ID, atomically or nearly so. The query-time effect is identical, but the index-quality effect is not: an in-place update may not trigger graph repair, while a delete-then-insert pair may briefly leave the graph in a state where the new vector is reachable but not optimally connected. For systems with high update rates, this is the source of the slow recall degradation that motivates papers like Ada-IVF and SPFresh.

Tombstones and query-side filtering. Systems that defer physical deletion typically expose a metadata filter that excludes tombstoned IDs from query results. This works correctly only if the filter is honored at every query path — including approximate similarity search, hybrid retrieval, and any reranker stage. Bugs in the filter path produce the most damaging class of failure: the system continues to retrieve content that has been "deleted," typically without any error signal.

The deletion problem the matrix doesn't show. Permission revocation. When a user's access to a document is revoked, the document content has not changed, but the set of users for whom that document is retrievable has. Many production systems handle this by encoding ACLs as metadata on the vector record and filtering at query time. This works only if the ACL metadata is updated synchronously with the source ACL system — and the link between an embedding store and an external identity provider is, in most architectures, the most asynchronous, most often-lagging link in the entire pipeline. A stale ACL on a vector record is a privacy incident waiting for a query.

The conservative posture: separate content deletion, ACL revocation, legal deletion, and model-cache deletion as distinct pipelines with distinct SLAs. Content deletion can tolerate seconds-to-minutes of staleness. ACL revocation cannot. Legal deletion has its own audit and proof-of-deletion requirements. Model-cache deletion (purging vectors generated by a specific model version) is rarely user-facing but is the precondition for clean migrations.

Consistency, latency, and the freshness SLA

The right way to specify a freshness requirement is as a service-level objective with measurable percentiles, not as a slogan. "Real-time" and "eventually consistent" are the two extremes; everything useful is between them.

A workable SLA model has four tiers, each tied to a query pattern:

Read-your-writes for individual users editing their own content. The user just changed a document; the next query they issue should see the change. This typically requires a synchronous indexing path or a hot overlay with bounded lag.
Bounded staleness for shared corpora. New content should be retrievable within some published budget — minutes for handbooks, seconds for incident docs, sub-second for fraud detection. The budget is the ceiling, not the average.
Eventual consistency for slowly-changing reference content. The system promises that a write will eventually be visible; the application does not depend on a specific bound.
Snapshot consistency for analytical or audit workloads. The query is run against a known index version, with no expectation that newer writes are visible.

Most production retrieval systems implement multiple tiers simultaneously, with different tiers served by different paths. Mixedbread's lifecycle model — files move through pending, in_progress, and completed states before becoming searchable (Mixedbread ingestion) — is a tier-2 pattern: the system publishes a state machine the application can poll, and "fresh" means "a file that has reached completed since its last upsert." Weaviate's documented consistency tradeoffs (Weaviate consistency) and Milvus's tunable consistency levels (Milvus consistency) expose tier choice as a per-query parameter. Qdrant's asynchronous point modifications (Qdrant points) require the caller to opt into synchronous waiting if read-your-writes semantics are needed.

The metrics that make these SLAs measurable are unglamorous and usually missing in early deployments:

Indexing lag p50/p95/p99 — time from source change to retrievability, measured end-to-end across the pipeline, not just inside the vector store.
Delete latency p50/p95/p99 — time from source deletion to query non-retrievability, separate from content-update latency because it is often longer.
ACL propagation latency — time from permission change to query enforcement, the metric most likely to be unmeasured in early deployments.
Orphan vector count — vectors in the index whose source document_id no longer exists in the canonical corpus.
Re-embedding backlog — number of chunks queued for re-derivation due to pipeline-version changes.
Source/index divergence — count of documents where content_hash in the canonical corpus does not match content_hash recorded in the index.
Compaction backlog — for append-with-compaction systems, the size of the hot region against its budget.

Without these metrics, "the index is fresh" is an assertion the system cannot defend. With them, freshness is a measurable SLO with the same operational discipline as availability or latency.

Embeddings as lossy compression: what reconciliation actually means

The "embeddings as lossy compression" framing is useful as a corrective to the implicit assumption that the vector store is the system of record. It is also strong enough to be misleading if pushed too hard.

The framing's strongest form is this: every embedding is a deterministic projection from (source_content, parser, chunker, embedding_model, dimensionality) to a fixed-dimensional vector. Every component of that tuple can change. When any of them changes, the vectors derived from the previous tuple are no longer in the same representation space as the vectors derived from the new tuple. The index that holds them is now mixed-generation. Similarity search across mixed generations is undefined behavior in a strict sense, even though it returns results.

This is why pipeline-version metadata on every vector is not optional. It is the only way the system can ask "which vectors were produced by an obsolete projection?" and re-derive them. In a single-model, single-pipeline deployment, this rarely matters; the moment a team upgrades from text-embedding-3-small to a successor, or fixes a chunker bug that affected long documents, or improves the OCR model that fed the parser, it matters absolutely. Without pipeline-version metadata, the only safe response to any of those changes is a full rebuild.

The framing's weak form is the claim that embeddings should be treated as a cache. They should not, in the sense that "cache" implies. A cache holds copies of authoritative state; an embedding holds a derived projection that cannot be inverted. Cache invalidation is straightforward: drop the cached value, recompute on the next read. Embedding invalidation is more like materialized-view maintenance: you need to know which vectors are stale, what they are stale with respect to, and whether the stale vector is still useful (a slightly outdated embedding is usually better than no embedding at all, but a vector from an incompatible model is worse than nothing).

The right operational stance:

The canonical corpus is the source of truth. It owns document IDs, content, ACLs, versions, hashes, and timestamps. It should be queryable independently of any index.
The embedding store is one of several derived indexes. Others may include a sparse keyword index, a metadata index, a graph index, a full-text store. Each has its own freshness SLA, its own derivation pipeline, and its own reconciliation strategy.
Reconciliation compares the index against the canonical corpus, not against itself. A reconciliation job iterates the source manifest, checks each document's current state against the index's recorded state, and queues operations to close the gap. The job's correctness depends on the canonical corpus, not on the index.
Rebuild is the recovery path for the failure modes incremental cannot handle. Pipeline-version migrations, large schema changes, recovery from a corrupted index, model upgrades that change the dimensionality or the representation space — these are rebuild events, and the system must preserve the ability to do them without serving disruption (typically via blue-green index promotion behind an alias).

This is not an argument against incremental indexing. It is an argument that incremental indexing is most reliable when the system has kept the rebuild path, not replaced it. The teams that struggle most with embedding-store maintenance are usually the ones who treated the vector store as authoritative and discovered, on the first model migration, that they had no way to go back.

Failure modes and what to measure

A non-exhaustive inventory of failures that show up in production retrieval systems, organized by where they originate:

Identity layer

Document IDs that change when documents are renamed or moved, producing duplicate records.
Chunk IDs that derive from ordinal position, producing orphan chunks on every edit.
Missing pipeline-version metadata, producing mixed-generation indexes invisible to monitoring.

Change-detection layer

Polling that misses changes between intervals.
Webhooks that fire on metadata updates without content changes.
CDC streams that lose ordering during partition reassignment.
Filesystem watchers that miss changes during their own restart windows.

Transformation layer

Parser bugs that produce different output for unchanged input across versions.
Chunkers whose output depends on prior internal state, producing nondeterministic chunk sequences.
Embedding API rate limits that cause silent fallback to retry queues with their own lag.
Reranker metadata schemas that drift, invalidating cached scores.

Index layer

Tombstone filters not honored at every query path.
Compaction backlogs that grow beyond the hot-region budget.
Graph repair after deletion that degrades recall in regions of the embedding space.
Replica drift where two index replicas have different post-write states.

Query layer

Read-after-write violations on tier-1 SLAs.
Stale ACL filtering that returns content the user no longer has access to.
Cross-shard queries that miss results because a shard is being rebuilt.
Hybrid-search merging that weights stale and fresh results inconsistently.

Reconciliation layer

Reconciliation jobs that scan the index instead of the canonical corpus, missing source-only changes.
Drift detection thresholds set too loose, allowing meaningful divergence to accumulate.
Rebuild paths that have not been exercised recently and have silently broken.

Each of these has a corresponding metric. The discipline is to instrument before deploying, not after the first incident.

The minimum dashboard for an embedding store with any production load:

Metric	Target
Indexing lag p95	Under SLA, with explicit alerting
Delete latency p95	Tracked separately from indexing lag
ACL propagation latency p95	Tracked separately, often the most asynchronous
Orphan vector count	Monotonically zero or trending toward zero
Source/index divergence count	Tracked per pipeline version
Compaction backlog size	Bounded by configured hot-region budget
Re-embedding backlog	Drained on a known cadence
Recall@k against a held-out query set	Trending stable or improving
Query latency p95	Tracked across hot/cold/overlay paths separately

The recall metric matters most when changes are happening, because recall degradation under update pressure is exactly what the dynamic-ANN literature (Ada-IVF, SPFresh paper, Azure Cosmos DB DiskANN streaming) describes as the central operational risk. A system that does not measure recall continuously cannot detect the slow degradation that motivates incremental-maintenance algorithms in the first place.

A decision framework

The cumulative argument of the article points toward a small decision tree rather than a single recommended pattern. The tree depends on freshness SLA, corpus size, source type, and write rate.

Step 1: Establish the rebuildable baseline. Regardless of what comes next, the system should support deterministic rebuild from a canonical manifest. This means stable document and chunk IDs, content hashes, pipeline-version metadata on every vector, and a blue-green index promotion mechanism. If any of these are missing, fix them before adding incremental complexity. The cost is high once, low forever after; the reverse architecture (incremental on a non-rebuildable foundation) tends to compound.

Step 2: Measure the change pattern. Estimate change rate (documents per minute), edit blast radius (chunks affected per document edit), deletion rate, and ACL change rate. These determine which incremental patterns are viable. A corpus with high deletion rate but low content edit rate is a different problem from one with the inverse profile.

Step 3: Choose an incremental strategy by source and SLA:

Situation	Strategy
Source is a transactional database, freshness budget seconds-to-minutes	CDC pipeline (Debezium-class) feeding a chunk-aware consumer
Source is a filesystem or document repository	Hash-based delta indexing with a watcher (mgrep-style)
Source has a natural time axis with decay	Time-windowed shards
High write rate, low-to-moderate deletion rate	Append-only with periodic compaction
Read-after-write required, large cold corpus	Hot overlay + cold compacted index
None of the above bind, corpus under low millions of chunks	Periodic rebuild on a content-hash trigger; defer incremental

Step 4: Treat ACL changes as a separate pipeline. Whatever the content-freshness strategy is, ACL revocation should have its own faster path with its own SLO. The cost of getting this wrong is qualitatively different from the cost of stale content.

Step 5: Run the replay test before declaring victory. Take a representative slice (10k–100k documents), apply a realistic change log (creates, edits, permission-only changes, deletes, moves), and measure write-to-search latency p50/p95, stale-delete leakage, read-your-writes success rate, recall@k against a fixed query set, and recovery from interrupted ingestion. The decisive canary: edit or delete 100 known documents, query for them every few seconds, and record when old content disappears and new content becomes retrievable. This is the experiment that exposes the gap between architectural intent and actual behavior, and it is cheap relative to the cost of discovering the same gap in production.

Step 6: Publish the SLA. The system's freshness behavior should be specified with the same rigor as availability and latency: per-tier SLOs, measured percentiles, alerting thresholds, and a documented rebuild path for the failure modes incremental cannot handle. A retrieval system without a published freshness SLA is not making a freshness claim — it is hoping.

The concluding architectural posture: incremental indexing is the right answer to a real problem at real scale, but the obvious version of the answer — "build a CDC pipeline, the vector store will handle the rest" — skips the load-bearing prerequisites. Stable identity, pipeline-version metadata, ACL-as-mutation, separate deletion semantics, and rebuild as a recovery path are not optional. They are the conditions under which incremental indexing is reliable. Without them, the system is not maintaining freshness; it is accumulating drift, slowly, in ways that do not surface until a model migration or an audit forces the question.

Companion entries

Core theory:

Materialized Views over Mutable Corpora
Embeddings as Lossy Projections
Stable Identity in Derived Indexes
Chunk Identity and Edit Blast Radius
Pipeline Versioning for Retrieval Systems

Practice:

Blue-Green Index Promotion
Hash-Based Delta Indexing
Time-Windowed Shards for Streaming Corpora
Append-Only Vector Stores with Compaction
Change-Data-Capture for Embedding Pipelines
Hot Overlay + Cold Index Patterns
Tombstone Semantics in Vector Indexes
ACL Revocation as Index Mutation
Reconciliation Loops for Derived Indexes
Recall Degradation Under Continuous Updates

Tools and systems:

mgrep and Background Filesystem Indexing
Mixedbread Stores Lifecycle
Qdrant Asynchronous Point Operations
Weaviate Async Indexing Tradeoffs
Milvus Tunable Consistency
Pinecone Serverless WAL/Memtable/Slab Architecture
OpenAI Vector Stores Eventual Consistency
FAISS Removal Semantics

Methodology:

Replay Testing for Retrieval Pipelines
Recall@k as a Continuous Metric
Source/Index Divergence Detection
Freshness SLOs and Indexing-Lag Percentiles

Counterarguments and limits:

When Full Reindex Is Cheaper Than Incremental
The CDC Trap: Row Updates Are Not Chunk Updates
Mixed-Generation Indexes and the Migration Problem
Freshness as a Failure Mode: When New Content Hurts Retrieval
Rebuildability as the Default, Incrementality as the Optimization

Adjacent concepts:

Hybrid Search Weighting
Reranker Metadata Drift
Permission-Aware Retrieval
Audit Trails for Derived Indexes
Semantic Drift Across Model Versions