Field notes · ai

Keeping a local document index fresh and trustworthy

A 16% non-indexed tail kept my nightly crawl running 12 to 16 hours until separate skip metadata made unchanged failures cheap to recognize each night.

·8 min read

My first incremental document crawl still spent 12 to 16 hours revisiting files that had not changed. The surprise was not the indexed majority. It was the 16 percent of candidates that produced no vector points, so the next run had no metadata proving they had already been examined.

Freshness became a state-design problem, not a faster-model problem. The fix was to persist a small verdict record for unchanged non-indexed files while keeping those records out of the searchable collection.

Key takeaways

  • Track indexed and intentionally non-indexed files as durable states.
  • Use modification time and size as a cheap first gate, then hashes for stronger change decisions.
  • Keep skip records out of the retrieval collection.
  • Let a full crawl bypass incremental shortcuts on a slower cadence.
  • Preserve per-file progress so a long crawl can resume safely.
  • Report stale, unreadable, duplicate, and missing content separately.

Why did an incremental crawl keep reprocessing unchanged files?

The ordinary delta gate looked up a file’s stored modification time and size in the vector collection. That worked for files with chunks. Unreadable, duplicate, oversized, and failed files had no points, so they had no stored state and were fully extracted again every night.

The pathological loop was:

find file
  -> no vector point exists
  -> extract and classify
  -> verdict says do not index
  -> write no vector point
  -> repeat tomorrow

Those files were concentrated in the most expensive part of the corpus: large PDFs, conversion failures, legacy files, and duplicates discovered only after extraction. A minority of candidates consumed a disproportionate share of the nightly window.

The bug is easy to miss because the main index was correct. Search results did not contain unreadable or duplicate marker records. Operationally, however, the crawler had forgotten that those files existed.

The extraction triage described in Extracting Office Documents for Local RAG made the missing state visible. Every non-indexed verdict needed a durable home, but that home could not be the searchable chunk collection.

What metadata makes an incremental crawl useful?

A practical first gate needs the canonical path, modification time, file size, and prior verdict. Indexed files also benefit from content hashes, source metadata, and captured permissions. The crawler can skip expensive extraction when the cheap attributes match and reprocess when they change.

For an indexed file, I store metadata on its chunks. For a file that produces no chunks, I store a separate record like:

{
  "source_path_hash": "stable-point-id",
  "mtime": 42.0,
  "size": 482113,
  "verdict": "unreadable"
}

The real record uses the canonical source path as lookup data but avoids placing raw paths in unnecessary error messages. The point contains no embedding and lives in a dedicated metadata collection.

Qdrant explains in its fundamentals documentation that a point can contain an identifier and payload without vectors. That is useful for operational state. I still keep this state in a separate collection so a future retrieval change cannot accidentally surface marker records as document evidence.

The decision order is:

  1. If a normal indexed record has matching time and size, skip extraction but refresh permission metadata as required.
  2. Otherwise, if a skip record has matching time and size, reuse its verdict.
  3. If either attribute changed, extract and classify again.
  4. If the file becomes indexable, write chunks and clear the old skip record.
  5. If --full is requested, bypass both shortcuts.

Why not trust modification time alone?

Modification time is a fast network-friendly filter, not a complete identity. Size reduces false matches, and a content hash provides stronger evidence after the file has been read. A periodic full crawl covers rare edits that preserve both metadata fields or expose a bug in incremental state.

Hashing every file on every run would require reading every file, which destroys the main performance benefit of the stat gate over a network share. I therefore use layers:

Signal Cost Use
Canonical path Very low Find prior state
Modification time and size Low Skip most unchanged files before extraction
Extracted content hash Higher Detect duplicate or unchanged content after read
Periodic full crawl Highest Reconcile assumptions and rebuild state

The full mode is not the normal nightly path. It is a deliberate slower audit that ignores cached verdicts and cheap gates. The schedule should leave enough time for it and should not overlap the ordinary crawl.

Moves and deletions need their own policy. A moved file can look like one missing path and one new path. A deleted file leaves stale chunks unless a reconcile pass compares the current inventory with stored source paths. I prefer an explicit full reconciliation for those cases rather than guessing from one failed file open.

How did the skip store change the nightly run?

The separate skip store made unchanged non-indexed files cheap to recognize before extraction. After it was committed and the catch-up crawl completed, the full estate finished in the overnight window instead of repeatedly timing out on the same tail.

The implementation was test-driven around the state transitions:

  • The first unreadable result creates a skip record.
  • An unchanged unreadable file is not extracted again.
  • A changed file is extracted again.
  • Full mode bypasses the record.
  • A skipped file remains present in the verdict report.
  • A newly indexable file clears its stale skip record.
  • Oversize, error, no-chunk, and duplicate outcomes are also remembered.
  • No skip marker is written into the retrieval collection.

That last assertion is load-bearing. Operational state should never become search context. A user asking a question should not retrieve a marker saying a secret or broken document exists.

The fix also demonstrated why performance measurements need a denominator. “The crawl is slow” was not actionable. “A 16 percent no-point tail is parsed on every run” identified the missing state and a direct verification target.

How do permissions stay current when content is skipped?

Content freshness and authorization freshness are different. An unchanged file can receive a new access control list, so an incremental crawl must refresh or revalidate permission metadata even when it skips extraction and embedding.

My indexed points carry an allow-list derived from the file’s NTFS permissions. At query time, Qdrant filters the candidates before ranking. Its official filtering documentation shows how payload conditions narrow searches and recommends indexes for fields used in filtering.

The safe invariant is:

unchanged content may reuse text and vectors
changed ACLs must update authorization payloads
unreadable ACLs must not preserve old broad access silently

If permission capture fails, I would rather remove or withhold the file from search until the crawler can prove an allow-list. Keeping yesterday’s access because today’s check failed turns an operational problem into a disclosure risk.

The complete authorization path is described in Building Private AI Search for a Windows File Share.

What makes a long crawl resumable?

Write durable state per file or per batch instead of waiting until the entire share finishes. The first complete crawl in my pilot needed more than one night, but completed files stayed completed, so the next run advanced instead of starting over.

Resumability needs idempotent writes:

  • A chunk has a deterministic identity tied to its source and position.
  • Reprocessing a file replaces its old point set rather than appending another copy.
  • Content-hash deduplication reaches the same decision on a retry.
  • Verdict reports can be regenerated from current state.
  • Partial failure leaves enough metadata to continue safely.

The crawler also needs bounded concurrency. More parallel extraction can improve throughput until network storage, OCR, memory, or the embedding service becomes the bottleneck. Past that point, concurrency adds timeouts and makes diagnosis harder.

I prefer progress counters that distinguish discovered, unchanged, extracted, indexed, deduplicated, skipped, and failed files. One total percentage hides the exact tail that caused this problem.

How do I know the index is trustworthy today?

Trust comes from a reconciliation report, retrieval evaluations, permission tests, and citations that still resolve to current source files. A successful nightly command only proves the process exited; it does not prove coverage or answer quality.

I would review these signals together:

  1. Candidate files by share and supported extension.
  2. Indexed, unchanged, duplicate, unreadable, unsupported, oversize, and error counts.
  3. Oldest successful crawl and last full reconciliation.
  4. Stored chunks whose source path no longer exists.
  5. Evaluation questions whose known source is absent from the top results.
  6. Authorization fixtures for at least two identities with different access.
  7. User feedback tied to the query and cited source set.

The unindexed report is especially important. It lets records owners distinguish “search found nothing” from “the source was never readable.” That is a better failure than a confident answer built from incomplete evidence.

Frequently asked questions

Index freshness is not one timestamp. It is the alignment of source inventory, content, permissions, vectors, verdicts, and citations.

How often should a full crawl run?

Choose a cadence based on the cost of the corpus and the risk of metadata-preserving changes. Nightly delta crawls plus a slower periodic reconciliation is a practical starting pattern, but measure the real completion window first.

Should failed files be retried forever?

No. Persist the failure verdict and retry when the file changes, the extractor version changes, or an operator requests a full pass. Permanent nightly retries hide genuine new failures inside predictable noise.

Can a vector database be the only crawl ledger?

It can store operational metadata, but keep non-document state logically or physically separate from searchable chunks. The retrieval path must never confuse a crawl marker with evidence.