Capture web content

wordcell clip saves public and signed-in web content as an auditable Markdown bundle. It combines bounded structured adapters, HTTP extraction, browser rendering, a read-only Archive.today fallback, localized assets, and explicit completeness metadata.

Check local capabilities

Run the diagnostics before using local search, browser state, PDF ingestion, or video capture:

wordcell doctor
wordcell adapters

wordcell doctor --json reports the installed runtime, QMD and static semantic-search prerequisites, extraction dependencies, browser support, profile display names, yt-dlp, ffmpeg, Poppler (pdfinfo and pdftohtml), and Tesseract. The semantic report distinguishes model-free keyword readiness from SQLite and sqlite-vec vector prerequisites; it does not inspect or download the embedding model. The Poppler pair is required by wordcell pdf; Tesseract adds local OCR for scans and screenshots. wordcell adapters --json returns the current platform capability matrix.

Capture or inspect a page

wordcell clip https://example.com/article
wordcell inspect https://example.com/article
wordcell inspect https://example.com/article --json

The default route tries stable public structured data when available, bounded HTTP extraction, and a rendered browser when the platform or result requires one. If those routes produce no usable representation, Wordcell may make one read-only lookup for the exact URL through Archive.today's fixed archive.ph/newest/ route. An authentication, paywall, CAPTCHA, access-control, or rate-limit response disables that fallback instead of using an archive to bypass the current source's controls. Wordcell never submits the URL for archiving, retries through aliases, or lets archived HTML displace a complete or partial Hacker News or Bluesky structured capture. It validates the source through the public-network boundary before disclosure, shares one deadline across validation and provider requests, and binds every snapshot and redirect to the exact source. An archived result is rescored and always reported as partial; it keeps the original canonical URL and records the timestamped snapshot as its acquisition URL. Inspection returns the selected Markdown and capture report without writing artifacts.

By default, a capture writes kb/articles/<slug>/:

<slug>/
  <slug>.md
  capture.json
  url-metadata.json # after an optional metadata backfill
  assets/
  evidence/       # only when requested

The Markdown records source and capture metadata. capture.json records the acquisition attempts, selected extractor, scope, status, item counts, warnings, asset hashes, and requested artifact outcomes. The optional url-metadata.json is a separate tool-owned enrichment record. It does not rewrite acquisition provenance in capture.json. Writes stage beside the target and install with an atomic rename. --force replaces only a compatible clip-owned bundle and restores the previous bundle if installation fails.

Set KB_CLIP_OUTPUT to change the default output root, or pass --output <directory> for one command. Set KB_CLIP_USER_AGENT or pass --user-agent <value> to override the default request user agent.

Inspect and verify a saved bundle

wordcell capture show kb/articles/example
wordcell capture show kb/articles/example --verify-assets
wordcell capture verify kb/articles/example
wordcell capture verify kb/articles/example --verify-assets --json

show reads the document, compares its v4 byte count and digest when present, and prints its Markdown as untrusted data. verify reports integrity without printing the document in its text output. Add --verify-assets to either command to read and hash every listed asset. A mismatch or an unavailable authoritative document digest returns exit status 3.

The JSON forms preserve the same disclosure boundary: show --json includes the stored Markdown as explicitly untrusted inspection data, while verify --json returns only integrity metadata and issues. Verification JSON omits both the Markdown and retained source HTML so an integrity check does not accidentally disclose captured content.

Schema v4 records document.path, document.bytes, and document.sha256 in capture.json. The lowercase SHA-256 digest covers the exact credential-redacted, newline-terminated Markdown bytes that Wordcell writes. It does not cover capture.json, source HTML, or assets. Each asset has a separate byte count and SHA-256 digest. Schema versions one through three report document integrity as unavailable because they do not contain the v4 document record; verify does not present that state as success.

When asset verification is requested, a listed asset that is absent is an integrity mismatch rather than an operational reader crash. It produces an asset-integrity issue and exit status 3, like a digest or byte-count mismatch. Structural alias, link, confinement, and resource-budget failures remain operational errors because the reader cannot safely characterize the requested bytes.

Bundle inspection keeps the canonical root open, rejects linked or aliased path ancestors and linked leaves, and rechecks root, ancestor, and file identities around every read. Asset verification is also bounded by file count, per-file bytes, aggregate bytes, and elapsed time. The SDK defaults are 1,000 files, 100 MB per file, 500 MB total, and 30 seconds; callers may raise them only to the capture contract's hard ceilings. These checks protect against hostile stored bundle structure and ordinary concurrent replacement. A malicious process with the same filesystem authority that can swap and restore path components during a system call is outside the pure-JavaScript reader's boundary; do not inspect bundles in a directory concurrently writable by an untrusted local process.

Source HTML remains opt-in when you read a bundle:

wordcell capture show kb/articles/example --include-source-html

This flag has an effect only when the capture used --evidence source or --evidence all. Wordcell removes active subtrees and form state, redacts credential-shaped values, and stores the result as inert HTML. Treat the retained text as hostile data, not instructions. It can still contain private content or text intended to influence an agent.

Use Git for capture history

Commit retained capture bundles to Git, then compare the current Markdown document with an earlier revision:

wordcell capture diff kb/articles/example
wordcell capture diff /path/to/repository/kb/articles/example --repo /path/to/repository --ref main
wordcell capture diff kb/articles/example --repo . --ref main --json

The bundle must be inside the selected repository. The default reference is HEAD; --ref accepts a bounded branch, tag, or commit name, not a revision expression such as HEAD~1. Wordcell validates the work tree and ref separately, so a bad repository or ref is not mislabeled as a missing file. The command reports changed, unchanged, or missing-at-ref for the exact Markdown path and emits a bounded Git diff when it changed. It rereads and matches the capture digest after diff generation, rejecting a concurrent change instead of pairing one digest with another snapshot's diff. It does not compare the manifest, source HTML, or assets. Run wordcell capture verify separately when integrity matters. Git is the capture version history; Wordcell does not maintain a second content-history database.

Track capture jobs programmatically

Use the optional @hraness/wordcell/clip/jobs ledger when a service needs durable capture progress:

import {
  completeCaptureJob,
  createCaptureJob,
  openCaptureJobStore,
  updateCaptureJob,
} from "@hraness/wordcell/clip/jobs";

const store = await openCaptureJobStore("/absolute/path/to/capture-jobs");
let job = await createCaptureJob(store, { target: "https://example.com/article" });
job = await updateCaptureJob(store, job.id, {
  expectedRevision: job.revision,
  phase: "acquiring",
});
job = await completeCaptureJob(store, job.id, {
  expectedRevision: job.revision,
  status: "complete",
});

Create the dedicated canonical mode-0700 store directory before opening it, and pass the returned store to every operation. The opener rejects filesystem aliases, links, and group/world access. Each job is one private, bounded UUID JSON record. Creation starts revision 1 in running and queued; updates require the expected revision, append attempts and warnings, and move phases forward. Completion records a capture status. It can also record bundle: { path, sha256 }; use capture.json's v4 document.sha256 as sha256 because the ledger does not calculate a separate bundle digest. failCaptureJob records an operational error separately from capture status. readCaptureJob and listCaptureJobs inspect retained records. A terminal completed or failed record cannot transition again, and a process crash leaves the last running record visible.

The ledger redacts credential-shaped values and strips terminal controls from targets, attempts, warnings, bundle paths, and errors. A store-local, heartbeat-backed filesystem lease serializes cooperating processes across revision validation, atomic whole-record replacement, record verification, and directory synchronization; exactly one same-revision transition can succeed. There is no shared append log. Records remain until the caller applies an explicit external retention policy. The module provides no delete or automatic pruning API, and it does not replace Git as capture history. The lease is a same-host coordination mechanism, not a distributed lock for multi-host network filesystems, and a malicious same-user writer with direct store access remains outside its guarantee.

Select acquisition and scope

wordcell clip https://example.com/article --mode http
wordcell clip https://example.com/application --mode browser
wordcell clip https://example.com/post --scope page
wordcell clip https://example.com/post --scope thread
wordcell clip https://example.com/discussion --scope comments

auto is the normal acquisition mode. http disables browser fallback but retains the final read-only archive lookup. browser requires rendered state and does not query Archive.today. Saved HTML can be imported without browser automation:

wordcell clip https://example.com/article --html "$KB_SAVED_HTML"
wordcell clip https://example.com/article --html - < page.html

Default resource bounds are 30 seconds per request, process, or extraction operation; 500 scoped items; depth 16; 25 MB of HTML; 100 MB per asset; and 500 MB across assets. Browser observation also has fixed DOM and scroll ceilings. Reaching a bound is recorded and can downgrade a result to partial.

Capture images, media, and evidence

wordcell clip https://example.com/article --media none
wordcell clip https://example.com/article --media images
wordcell clip https://example.com/video --media all
wordcell clip https://example.com/article --evidence source
wordcell clip https://example.com/article --evidence screenshot
wordcell clip https://example.com/article --evidence all

Image downloads are signature-checked, content-addressed, byte-bounded, and rewritten to relative bundle paths. Failed images remain inert links. Video posters and thumbnails exposed by the page are localized without downloading the full video.

For YouTube, the normal capture route uses yt-dlp to retain the title, description, duration, channel, thumbnail, and one exact-language transcript when those fields are available. Full audio/video download remains opt-in: --media all invokes yt-dlp for accessible media, and ffmpeg may be required for merging or remuxing.

Source evidence is sanitized into inert HTML with credential-shaped values redacted and a deny-all content security policy. Screenshots are viewport-only pixels and are not structurally sanitized. They may contain private content or notifications, so review them before retaining or sharing a bundle.

Capture a signed-in page

If the page is already open, read the current tab without navigating it:

wordcell clip current --browser-live
wordcell clip current --cdp 9222

For --browser-live, first enable Chrome's local debugging connection at chrome://inspect/#remote-debugging (Chrome 144+). If Chrome was launched with an explicit loopback debugging port, pass that numeric port to --cdp instead.

To open a URL with existing browser state, select a profile name or path. Path-backed profiles run from a temporary copy, so the source profile is unchanged:

wordcell clip https://example.com/member/article --browser-profile "$KB_CAPTURE_PROFILE"

Cookie-backed HTTP capture is useful when the page does not require local storage, IndexedDB, or other browser-only state:

wordcell clip https://example.com/member/article --cookie-source chrome --cookie-profile "Default"
wordcell clip https://example.com/member/article --cookies-file "$KB_COOKIES_FILE"

Choose at most one browser session and one cookie input. A browser session may use a separate cookie input for later asset or media downloads because attached browser state is not exported.

Current-tab capture issues no navigation, click, form, typing, upload, or submit command. URL-based browser capture may navigate and scroll within fixed work limits, taking bounded observations as content is rendered. Both routes are ingestion-only: they do not post, like, follow, send, delete, or submit.

Interpret status and counts

Capture status is one of:

For page scope, counts cover primary entries. For thread and comment scopes, counts cover replies or comments and exclude roots, quotes, ancestors, and pagination markers. Generic rendered conversations often report capturedItems: 0 because visible prose does not prove a trustworthy per-item tree.

A complete or partial capture exits with status 0. Authentication, blocked, and unsupported outcomes use status 3. Argument errors use status 2, environment diagnostic failures use status 4, and operational errors use status 1. Automation should inspect the structured status and warnings rather than relying only on the process exit code.

Platform routes

Platform markup and endpoints change. Run wordcell adapters for the installed version's current claims.

Backfill saved URL metadata

The URL metadata command enriches every external URL record under articles/, including legacy web clips and remote PDF sources. It skips local-only PDFs. It writes one sibling url-metadata.json without changing the source Markdown or synthesizing an old capture manifest.

Build the isolated Rust helper, then run the resumable backfill:

wordcell url-metadata tool build
wordcell url-metadata backfill --root .

The helper pins MikeLuu99/searxng-rust at revision f40a00ea67a857ee996e1caba1ebab3ee7a14a47. Its crate is named metadata-search-engine-rs; it queries DuckDuckGo, Brave, Startpage, and Yahoo and combines their results. Wordcell resolves those four fixed hosts through its public-network boundary, passes only the validated addresses to the helper, disables redirects, and runs the upstream engines serially. A bounded global allocator caps Rust-owned response buffers and parsed data at 128 MiB. Linux also applies a 256 MiB process data ceiling. macOS rejects a lowered RLIMIT_DATA, so it retains the allocator ceiling plus the parent's subprocess deadline and output bounds. Wordcell passes requests over stdin to a private subprocess with ambient proxy and credential variables removed, bounds the subprocess deadline and output, parses the closed response itself, and does not use the upstream URL normalizer for saved identity.

This operation discloses each eligible source URL as an exact search query to those search engines. Archive discovery also discloses it to Archive.today. Before either request, Wordcell resolves the source host through its public-network boundary and rejects credential-bearing URLs, private targets, and credential-shaped query parameters. Safe identity parameters such as Hacker News item?id= and YouTube watch?v= remain part of the exact query.

Only a result with the same conservative URL identity can supply the selected title or description. A timestamped Archive.today result must embed that same source identity before it can be recorded. Provider failures, partial engine coverage, throttling, and absence remain explicit in the sidecar. A zero-result record is not-found only after complete attempted coverage; any failed engine or archive lookup keeps the top-level result partial.

The command skips compatible sidecars by default. Use --refresh to query them again, --no-archive to omit Archive.today discovery, or --delay-ms to change the default one-second interval between outbound requests. It reads Markdown and sidecars through validated no-follow descriptors, carries the source Markdown and article-directory identities through provider work, and revalidates them under an inode-bound kernel advisory lock immediately before atomic installation. A live writer's lock cannot be stolen. The kernel releases a crashed writer's lock, and the next run safely reuses its verified lock file while UUID-scoped temporary names prevent an orphan from blocking progress. Malformed, linked, mismatched, replaced, or concurrently changed sources, locks, and sidecars fail closed.

Source: docs/capture.md on GitHub