AI Knowledge Platform — Filip Iversen

A regional innovation network asked for better search. It connected organizations, funding programs, and startups, but much of its information was hard to find. Resources lived in PDFs, partner websites, and program one-pagers.

I worked on the product strategy and AI engineering. The initial proposal was a search-and-filter plugin for the existing website. As we worked through the requirements, the scope grew to include importing external material, reviewing it, answering questions with source citations, and publishing a directory and map.

The finished platform connects those tasks. Staff can add a PDF or a URL, review the extracted information, and publish it. Approved material becomes available to the search, while organization and program records also appear in the directory.

Workflow showing PDFs, websites, and curated records moving through extraction and organization, staff review, and publication into an approved library. The library branches into answers with citations using keyword and semantic search, and a directory and map of organizations and programs. Human approval controls public access, and scheduled refreshes preserve editorial work. — Staff review and publish source material into a shared library. Approved content powers cited answers, while connected organization and program records populate the directory and map.

From a document to an answer

The first part of the work was getting usable content into the system. The ingestion pipeline accepts PDFs and URLs, extracts their contents, and removes navigation, ads, and other boilerplate. Some partner sites render their text with JavaScript, so those pages need a browser to load before extraction.

A curator reviews the material before publishing it. That approval determines what the public search can retrieve. The answer process draws on this approved library rather than searching the open web.

Retrieval combines keyword search with semantic search, using embeddings stored in PostgreSQL. Keyword matching handles exact names and specialist terms; semantic search helps with questions phrased differently from the source material. We combine the results to select passages for the answer.

The answer includes citations to those passages. A reader can follow the reference and inspect the material behind the response.

An illustrated search answer recommends the Example Lab Program for shared lab space and specialist equipment. A line connects citation 1 to the matching highlighted passage on page 4 of its program guide. — Follow a citation from the answer to the passage that supports it. Illustrative question, answer, and source shown.

Fixing the citations

An early version retrieved a relevant passage but cited the beginning of its document. The answer could look convincing while sending the reader to the wrong evidence.

We traced the problem and corrected the citation handling to use the passage that matched the question, so the reader could check the answer against it.

There also needed to be a path for questions the library couldn't answer. The system compares its strongest match against an adjustable threshold. Below that threshold, it returns a message explaining that it doesn't have a strong match. We calibrated that threshold against the embedding model's behavior on real queries.

Keeping the directory connected

The same curated database holds the network's organizations and programs, their locations, and the categories and audiences used for filtering. That data also powers the public directory and interactive map.

Staff approve an organization once. Its published record can then appear in the directory, on the map, and as context for the search. They maintain those records alongside the source material the answers draw from.

A directory card for Example Innovation Center is connected to a matching highlighted marker and label on a folded map. Both show the same organization and St. Louis region location, illustrating discovery by category or location. — The directory and map use the same approved organization record. Illustrative record and schematic map shown.

Refreshing sources without undoing editorial work

Published web sources refresh on a schedule. The ingestion pipeline retries transient failures, so a temporary network problem doesn't immediately end the attempt to update a resource.

Re-ingestion also has to account for the work a curator has already done. New source passages and embeddings wait for a curator's approval before replacing the approved evidence. Edited titles, summaries, and existing publish status stay intact. Staff can revise a resource's public description without having to redo that work after the next scheduled refresh.

Updated source passages sit in front of approved evidence and are marked Awaiting review. Beside them, an editorial record preserves the curator’s title, summary, and Published status, each marked with a check. — New source evidence waits for review while the curator’s title, summary, and publish status stay intact.