In this article

Start with the retrieval task

The original local PROFAS research library used ParadeDB in an Apple container. That gave us a working place to catalogue papers, retain extracted text and search the corpus. When sharing the library became the next requirement, the obvious question was where to host that system. A more useful question came first: what did the research workflow need from its search engine?

The answer was concrete. A reader should find a passage, understand which paper and source version it belongs to, open the relevant page and inspect the surrounding context. Filters should narrow the material, while notes and reviews should retain their links to it. The immediate requirement was full-text retrieval across a research library, rather than a broader analytical search platform.

We selected D1 with SQLite FTS5 for that hosted workflow. The implemented search uses full-text matching, ranked results and contextual excerpts, with stable passage and page identities carried into the result. This was a judgment about sufficient capability for the library, not a claim that FTS5 and ParadeDB have equivalent features, ranking behaviour or operational characteristics. The richer source model did not require keeping the original infrastructure unchanged.

Keep originals and searchable text distinct

The hosted library separates searchable and collaborative records from original bytes. D1 holds document metadata, sources, pages, passages and contribution history. Private R2 storage holds the originals. A search hit is therefore a route into retained material, rather than an isolated fragment whose provenance has to be inferred from its wording.

That separation also keeps extraction in the right place. PDF and OCR processing remain local owner-operated work. Prepared material is published through a scoped import path, and a new source stays hidden until finalisation checks its bytes and page counts. The hosted Worker does not turn a supplied local filename into an executable instruction or pretend that an incomplete transfer is an accepted source.

Source identity matters when extraction is partial. A reader needs to know whether a missing phrase was absent from the paper, outside the extracted coverage or simply not returned by the query. Preserving source hashes, extraction status and physical page identities gives the application a place to express those limits. Search is an aid to locating evidence; the returned excerpt is not a replacement for reading it.

ENGINEERING NOTE

A source-linked reading path

  1. Search passage

    A ranked hit identifies its document, source version and retained page.

  2. Read the source

    Inspect the exact page and surrounding context with extraction limits visible.

  3. Review the paper

    Separate findings, appraisal, relevance and recall material.

  4. Inspect a connection

    Follow a selected concept or research direction back to its evidence.

The graph is optional context around the reading path. Each accepted connection retains a route back to a specific review passage and its source pages.

Make hosted authority explicit

Sharing introduces a different set of questions from searching. Who is reading, who can contribute and which source material may an agent inspect? The library uses Cloudflare Access at the connection boundary, then resolves the authenticated identity to application membership or a scoped agent grant. Passing the outer identity check does not, by itself, grant permission to edit reviews or export the corpus.

The browser and agent adapters use the same Worker services for accepted operations. Notes and review revisions derive their author from the authenticated principal, while revision checks protect their editing basis. A restricted agent sees only its permitted documents. Authoring scope does not quietly expand into source publication, membership administration or a bulk export grant.

After cutover, hosted notes and reviews became the current shared contribution record. The local library was preserved for extraction and recovery, but it is not a writable replica that silently synchronises with D1. That is an important simplification: there is one place to find current collaborative work. Returning to an older local snapshot would require preserving newer hosted contributions and deliberately changing authority, not merely opening another interface.

Add only a graph worth maintaining

A research corpus also raises questions that text search alone does not present conveniently: which papers examine this concept, which platform elements they inform and which research directions remain worth investigating. We added a small curated graph around four kinds of node—documents, concepts, PROFAS elements and research directions—and five relationship verbs. Three views expose selected connections rather than asking the reader to decipher one dense network.

The constrained vocabulary is only part of the design. A paper relationship pins an immutable review version, a pointer to the relevant review passage and its retained source-page evidence. Findings, our appraisal and proposed applications remain distinguishable. Candidate annotations stay with their candidate; an older set is not silently reassigned to a newer review. When a graph accompanies a new review, acceptance validates them together.

This is a research reading aid, with explicit maintenance costs. It does not establish scientific agreement, discover every relevant relationship or replace reading the source. It is also separate from code navigation: engineering changes still require direct reading of current implementation and design context. A missing edge is not evidence that a dependency, constraint or relevant paper does not exist.

Make portability deliberate

A smaller hosted stack still needs a clear way out. The library supports consistent private bundles containing records and originals, alongside deliberate metadata-and-text exports suitable for private Git history. Contributions pause during the consistent export window while reading continues. The portable record preserves stable identities and histories rather than reducing the corpus to a folder of PDFs and a freshly generated index.

The search index is a projection that can be rebuilt; original bytes and accepted contributions are the assets to preserve. Recovery preparation verifies the complete private bundle, targets an empty isolated destination and disables restored access until it is deliberately reauthorised. The Git derivative is useful on its own terms, but it excludes originals and private identity material and cannot replace that complete bundle.

The same restraint applies to authoring. Codex writes paper reviews, recall material and graph annotations through the accepted services. New eligible papers queue work by default, but the queue launches no background agent or model service and needs no model API key. The library organises and records that work. Choosing less infrastructure meant retaining the distinctions the research needs while avoiding additional systems whose role we could not justify.