ENGINEERING PROFAS / Infrastructure and research tools
Choosing less infrastructure
The research library needed reliable full-text search, readable source pages and attributable collaboration. Choosing its infrastructure began by asking which capabilities that work actually required.
In this article
Start with the retrieval task
The original local PROFAS research library used ParadeDB in an Apple container. That gave us a working place to catalogue papers, retain extracted text and search the corpus. When sharing the library became the next requirement, the obvious question was where to host that system. A more useful question came first: what did the research workflow need from its search engine?
The answer was concrete. A reader should find a passage, understand which paper and source version it belongs to, open the relevant page and inspect the surrounding context. Filters should narrow the material, while notes and reviews should retain their links to it. The immediate requirement was full-text retrieval across a research library, rather than a broader analytical search platform.
We selected D1 with SQLite FTS5 for that hosted workflow. The implemented search uses full-text matching, ranked results and contextual excerpts, with stable passage and page identities carried into the result. This was a judgment about sufficient capability for the library, not a claim that FTS5 and ParadeDB have equivalent features, ranking behaviour or operational characteristics. The richer source model did not require keeping the original infrastructure unchanged.
Keep originals and searchable text distinct
The hosted library separates searchable and collaborative records from original bytes. D1 holds document metadata, sources, pages, passages and contribution history. Private R2 storage holds the originals. A search hit is therefore a route into retained material, rather than an isolated fragment whose provenance has to be inferred from its wording.
That separation also keeps extraction in the right place. PDF and OCR processing remain local owner-operated work. Prepared material is published through a scoped import path, and a new source stays hidden until finalisation checks its bytes and page counts. The hosted Worker does not turn a supplied local filename into an executable instruction or pretend that an incomplete transfer is an accepted source.
Source identity matters when extraction is partial. A reader needs to know whether a missing phrase was absent from the paper, outside the extracted coverage or simply not returned by the query. Preserving source hashes, extraction status and physical page identities gives the application a place to express those limits. Search is an aid to locating evidence; the returned excerpt is not a replacement for reading it.
A source-linked reading path
- Search passage
A ranked hit identifies its document, source version and retained page.
- Read the source
Inspect the exact page and surrounding context with extraction limits visible.
- Review the paper
Separate findings, appraisal, relevance and recall material.
- Inspect a connection
Follow a selected concept or research direction back to its evidence.
Add only a graph worth maintaining
A research corpus also raises questions that text search alone does not present conveniently: which papers examine this concept, which platform elements they inform and which research directions remain worth investigating. We added a small curated graph around four kinds of node—documents, concepts, PROFAS elements and research directions—and five relationship verbs. Three views expose selected connections rather than asking the reader to decipher one dense network.
The constrained vocabulary is only part of the design. A paper relationship pins an immutable review version, a pointer to the relevant review passage and its retained source-page evidence. Findings, our appraisal and proposed applications remain distinguishable. Candidate annotations stay with their candidate; an older set is not silently reassigned to a newer review. When a graph accompanies a new review, acceptance validates them together.
This is a research reading aid, with explicit maintenance costs. It does not establish scientific agreement, discover every relevant relationship or replace reading the source. It is also separate from code navigation: engineering changes still require direct reading of current implementation and design context. A missing edge is not evidence that a dependency, constraint or relevant paper does not exist.
Make portability deliberate
A smaller hosted stack still needs a clear way out. The library supports consistent private bundles containing records and originals, alongside deliberate metadata-and-text exports suitable for private Git history. Contributions pause during the consistent export window while reading continues. The portable record preserves stable identities and histories rather than reducing the corpus to a folder of PDFs and a freshly generated index.
The search index is a projection that can be rebuilt; original bytes and accepted contributions are the assets to preserve. Recovery preparation verifies the complete private bundle, targets an empty isolated destination and disables restored access until it is deliberately reauthorised. The Git derivative is useful on its own terms, but it excludes originals and private identity material and cannot replace that complete bundle.
The same restraint applies to authoring. Codex writes paper reviews, recall material and graph annotations through the accepted services. New eligible papers queue work by default, but the queue launches no background agent or model service and needs no model API key. The library organises and records that work. Choosing less infrastructure meant retaining the distinctions the research needs while avoiding additional systems whose role we could not justify.