Personal Robot

What It Is

This is ORION — a fully local, open-source personal robot, built from the ground up on consumer hardware with no cloud APIs and no external dependencies. The goal is a small physical robot that can perceive its environment, recognise the people in it, hold conversations, and build its own personality over time through accumulated interactions.

ORION currently lives entirely in software, progressing through a phased build plan from a local conversational LLM through memory, voice, and computer vision, toward an eventual physical form on a Raspberry Pi. Every component — language model, speech recognition, text-to-speech, vision pipeline — is designed to run locally, with the only running cost being electricity.

The Origin

The idea of building a personal robot has been in the back of my mind since childhood — but for most of that time it felt firmly in the realm of science fiction rather than something a single person could actually build. The LLM revolution of 2022 changed that perception quickly. The first models were too large for consumer hardware, but the pace at which capable models have been compressed and optimised since then has been remarkable. By 2026, with small models running respectably on a laptop GPU and tools like GitHub Copilot meaningfully accelerating development, a full robotics pipeline on consumer hardware had gone from a fantasy to something worth actually starting.

ORION is named after my first Dungeons & Dragons character — a small personal touch that felt right for a project this long in the making.

The Philosophy

What genuinely drives this project is curiosity about memory — how it forms, how it shapes identity, and what it would mean for an artificial system to have something analogous to it. I took a neurobiology course during my undergraduate years and it left a lasting impression: the mechanisms of neuroplasticity, the way memories are consolidated and retrieved, the degree to which our personalities are shaped by accumulated experience rather than fixed at birth.

I've been influenced equally by philosophers and AI researchers. Daniel Dennett's work on consciousness and the nature of mind; Andrej Karpathy's clarity about what neural networks actually do and don't do; Yann LeCun's arguments about the limits of current architectures and what genuine machine intelligence might require. The philosophy of mind has always fascinated me, and this project is, in a way, a hands-on exploration of those questions: what does it mean to remember? Can a system develop something like a personality through interaction? Does it matter whether the substrate is biological?

I don't have answers. But building ORION is my way of taking the questions seriously.

Build Phases

• Phase 1

  Conversational LLM — A local chat interface running a small LLM via Ollama. The foundation: a capable language model that can hold a conversation with no internet connection and no API key.

• Phase 2

  Memory — Persistent memory via a vector store and RAG pipeline. This phase is ongoing and experimental — different memory architectures behave very differently with small models, and finding what works reliably for context awareness versus fact retrieval is an active area of exploration. ORION has also been given the ability to rewrite its own personality files based on conversations, with mixed but genuinely interesting results.

• Phase 3

  Voice I/O — Whisper for speech-to-text and Kokoro for text-to-speech, developed in parallel through the STARLING project. The speech pipeline is being refined as a standalone system before full integration into ORION.

• Phase 4

  Computer Vision — YOLOv8 for object detection and OpenCV for face recognition. The goal is practical rather than broad: ORION should recognise a small set of familiar faces when they enter the room, notice an unfamiliar face, and be able to ask who that person is and remember the answer.

• Phase 5

  Physical Robot — A basic chassis with wheels, a camera, a microphone, and a speaker, all connected to a Raspberry Pi. Running the full pipeline on the Pi alone will be constrained, and in practice compute may need to be streamed from a local machine over the network — but the direction of travel in on-device AI makes even that constraint feel temporary.

The Local-First Constraint

Building everything locally isn't the easy path. Getting models to run well on specific hardware — a laptop GPU for development, a Pi for deployment — involves real tradeoffs in model selection, quantisation, execution order, and pipeline design. Every phase introduces new constraints that have to be solved rather than bypassed.

But using a cloud API was never seriously on the table. Anyone can plug a toy into an API — that's not the challenge. The point is to build something genuinely functional from first principles, on hardware I own, without depending on any external service. The engineering problems that constraint creates are exactly the ones worth solving.

The Wider Ecosystem

Several of my other projects exist, at least in part, as research for ORION. STARLING is a dedicated exploration of the speech pipeline — STT, LLM, TTS, latency optimisation — on the kind of constrained hardware the robot will eventually use. The Personal Dungeon Master project is a deep exploration of memory architectures: how knowledge graphs handle context versus fact retrieval, how small LLMs behave with different memory structures, what it takes to maintain coherent long-running state.

The overlap is deliberate. Each project advances a specific part of the problem, in a context where it can be developed and tested independently before being folded back into the robot.

What's Next

The project is currently in a considered pause while the memory and speech systems mature through their respective projects. The next concrete step is committing to a Raspberry Pi and beginning the work of getting the existing pipeline to run on constrained hardware — the moment the software meets a physical body.

The longer vision is something more interesting than a capable assistant. In 3–5 years, I'd like to see a small physical ORION exploring my apartment, talking to whoever walks in, and — critically — building its own personality through those interactions. Not a fixed set of traits, but something that evolves. A system that is genuinely shaped by its accumulated experience of the world and the people in it. Whether that constitutes anything philosophically meaningful is a question I'm happy to leave open.