Giving Your AI a Memory: Building a Local Knowledge Base with pgvector and MCP

Model Context Protocol is an open standard for giving LLMs access to external tools. The LLM decides when to call a tool and what to search for. You provide the tools and their descriptions; the model does the rest.

The interesting bit for a knowledge base: because MCP is model-agnostic, the same server acts as shared memory across different tools. I use the same OKB instance from Claude Code, Claude.ai, Le Chat, and local models, and I could point Cursor and Copilot at it too. With the HTTP MCP server, I share a knowledge base with my wife (topic: farming).

The core is extremely simple, you just need a few things: a way to turn documents into vectors, a place to store and an index to search those vectors, and a way to expose that search to an LLM.

Document: Django Performance Notes
Project: student-app          # inferred from path or frontmatter, or set in an MCP call
Section: Query Optimization   # extracted from e.g. org/markdown headers
Topics: django, performance   # from frontmatter tags
Content: Use select_related() to avoid N+1 queries...

# Batch ingest with Modal GPU
okb-admin ingest ~/notes ~/docs

# Or force local embedding (uses CUDA if available, falls back to CPU)
okb-admin ingest ~/notes --local

OKB exposes itself as an MCP server either over stdio (for local tools like Claude Code) or over HTTP with token auth (for browser-based tools or remote setups).

The local configuration (Claude used as example):

{
  "mcpServers": {
    "knowledge-base": {
      "command": "okb-admin",
      "args": ["serve"]
    }
  }
}

That's it. Any MCP-capable tool now has access to search_knowledge, keyword_search, hybrid_search, save_knowledge, and a growing list of others. The tool descriptions tell the LLM what each one does, and the model decides when to use them.

For browser-based or remote access, OKB runs as an HTTP server:

okb token create --db personal -d "browser"
# → okb_personal_rw_a1b2c3d4...

okb serve --http --host 0.0.0.0 --port 8080

This is how my wife and I share a farming KB on the same server, our own (RW) tokens and our own LLM accounts connecting to it. It's also how the same knowledge base serves both my local coding sessions and my browser-based conversations.

The save_knowledge tool for MCP was the killer feature for me, because with it when an LLM learns something useful during a conversation, whether a decision, summary or synthesised insight, it can save it back to the knowledge base, so that next session, that information is searchable.

I also added trigger_sync so the LLM can pull in fresh data from external sources like GitHub issues, Todoist tasks, Slack messages, or Dropbox Paper documents, without me having to run a cron job or remember to re-ingest. All of those are incremental, so reasonably cheap just to run a full resync once in a while, at least for me.

It works, and the rate of 'minor improvements' I add to remove warts and support my workflows has slowed down to a bare trickle. Having the LLM automatically search my notes when I ask about something we discussed previously is really making the LLM more useful, and I no longer maintain a separate document of "things to explain at the start of each session." The MCP integration means it happens with no interruption: I just ask a question, the model searches, I get an answer grounded in my actual notes. With the amount of data I have the semantic search is surprisingly reliable, and I can't recall a case where it failed to surface a relevant document that existed.

The multi-database setup turned out to matter more than I expected, but not for the reason I originally built it. Yes, keeping personal and work separate is good hygiene. But the killer feature is that my wife and I share a farming database – we're running a small operation and she has her own Claude.ai setup pointed at the same KB via the HTTP server. She might add notes about crop planning and tools, I add notes about our apple tree growing project, and both our Claudes can search across all of it. Shared memory for a shared project, without either of us having to forward emails or maintain a document.

Some of the most valuable documents in my knowledge base are the syntheses the LLM created – summaries that connect dots across multiple source documents. OKB has a synthesis pipeline where the LLM proposes connections across your notes, and you approve or reject them before they enter the knowledge base. LLMs are quite good at noticing patterns you've never explicitly connected, and the save_knowledge tool means those insights persist. Less "always right" than the simpler store/retrieve knowledge, but helpful to create more focused knowledge from a large corpus. I started out trying unguided LLM-driven "knowledge enrichment" and "entity extraction", but I could never get the hallucinations to an acceptable level.

If you're using an MCP-capable LLM regularly and you have a meaningful collection of notes, documents, or project knowledge, then yep! The core (pgvector + chunking + MCP server) is a fun project, and only the quality-of-life features (sync plugins, synthesis, multi-database) took weeks of using-building-testing-discarding-rebuilding and started feeling like work.

OKB itself is open source and installable via pipx if you'd rather just use it than build your own. But the interesting part isn't my specific implementation but the pattern. A local database, a chunking strategy that preserves context, an embedding pipeline that does or doesn't require a permanent GPU, and an MCP server that lets the LLM drive the retrieval.

I can't really work with an LLM anymore without the KB - with it, the LLM moves from smart autocomplete with hallucinations and amnesia towards something more like an over-eager PA with perfect memory (and hallucinations). Even if it can't bring me coffee (Kristoffer's probably can soon), it's still a huge help.

The code is at github.com/haard/okb. Install with pipx install okb. If you build something similar or have questions, I'm @motmanniska@mastodon.nu.