Ralph: A Local-Only Autonomous Coding Agent Loop

The Problem

I’d just finished prototyping a full-autonomy AI coding feature at work, and it left me with a question worth testing on my own time: how much of that capability actually depends on a large cloud-hosted model, versus how much could run entirely on local hardware? Autonomous coding agents (“agent loops” that plan, write, test, and commit code with minimal supervision) almost universally assume a large, cloud-hosted model behind the API call. That assumption brings recurring API cost, network latency, and a hard dependency on a third party staying up and available. So as a self-directed proof-of-concept, I set out to rebuild the same autonomous-loop workflow — describe an intent, walk away, come back to working, tested, committed code — but running entirely locally, with a much smaller open-weight model doing the reasoning. The catch: small local models are far less reliable at following instructions, more prone to hallucinating conversational filler instead of raw commands, and more likely to get stuck repeating a failing action.

The Solution

I designed and built a self-contained agent loop — nicknamed “Ralph” — that runs a full plan-execute-verify-commit cycle against a local Ollama model, with no cloud API in the loop at all. It takes a single markdown “intent” file describing what to build, breaks it into a technical schema and a structured multi-step strategy, then works one atomic task at a time: writing code, running it, checking the result against an automatically generated acceptance test, and committing on success. Because the underlying model is small and local, most of the engineering went into compensating for that: a validation layer that catches unsafe or redundant commands before they run, a “think before you run” critical-review pass, and a five-checkpoint QA gate before any step is considered done. A separate interactive discovery tool turns a vague idea into that structured intent file through a short guided conversation, so the loop always starts from a well-formed spec rather than a one-line prompt.

Outcome & Impact

The result answers the question I set out with: yes, the plan-write-test-commit loop holds up with zero per-token API cost and zero cloud dependency, running a bounded, fully autonomous session of up to 30 iterations, each one gated by a five-checkpoint QA pipeline (build integrity, regression tests, empty-file detection, architecture sync, and a dynamically generated acceptance-test check) before it’s allowed to commit. Every successful step lands as its own git commit, so a full session leaves a readable, revertable history rather than one large opaque change. What it cost to get there was reliability — nearly all the engineering went into scaffolding around a small model’s weaknesses rather than the loop logic itself, which is the main finding: local-only autonomy is achievable, but only if you budget for guardrails a cloud-hosted model would have given you for free. The architecture itself went through a real pivot mid-build: an early version routed every call through a multi-stage LLM proxy for model flexibility, which I killed in favor of a direct connection to the local model once it was clear the proxy added latency and failure surface without adding value — a good example of cutting scope once real usage showed which design goal actually mattered.

My Role & Contributions

Aspect	Detail
Role	Sole designer and implementer
Team size	1
Timeline	Built and iterated solo over about a week
Scope	Agent-loop architecture, safety/validation layer, QA gate design, discovery-tool CLI, packaging as a global command-line tool
Key decisions	Dropped an early multi-stage proxy architecture for a direct local-model connection; chose re-derive-next-step-from-state over a fixed task list; built the guardrail layer specifically to compensate for small-model unreliability rather than assuming a stronger model

Technical Overview

The loop runs as a five-stage lifecycle against a local Ollama model:

A separate front-end tool handles the “before Stage 0” problem: a small interactive CLI walks the user through problem framing, UX preferences, and technical constraints in three phases, then has the model synthesize the answers into the formal intent.md the loop consumes. State (active feature, branch, current step, last test result) persists to a JSON file between iterations so the loop can resume or be inspected mid-run. Ollama models are swapped by task: one model for planning/synthesis, a smaller code-focused model for execution.

Challenges & Key Decisions

Killing the proxy layer

The first working version routed model calls through a multi-stage proxy for flexibility across model backends. Once the loop was actually running end to end, the proxy was pure overhead — added latency, an extra process to keep alive, and a class of failures unrelated to the actual coding task. I cut it and moved to a direct HTTP connection to the local model, archiving the old configuration rather than trying to make the proxy earn its keep. Fewer moving parts turned out to matter more than the theoretical flexibility.

Guardrails for a model that hallucinates commands

Small local models frequently return conversational text (“I will now create the file…”) where a raw shell command is expected, or resubmit the same failing command repeatedly. I built a dedicated sanitization layer that strips conversational filler and code-fence wrappers from model output, a quote-balance and sed-safety check to catch commands that would fail on shell syntax alone, and a redundant-command detector that forces a pivot (e.g., rewriting a no-op touch into a real file-write) instead of letting the loop repeat itself.

Deriving the next step instead of following a fixed plan

Rather than generating a static task list upfront and executing it in order, Stage 2 re-evaluates the actual current file state and completed-task history on every single iteration and asks the model which atomic step is still outstanding. This makes the loop self-correcting: if a step was already completed by a side effect of a previous one, the loop recognizes that from the files on disk rather than blindly re-running a stale plan.

Hitting a hardware ceiling

Running everything locally moves the constraint from “API budget” to “the laptop in front of you.” On a 32GB machine, the model plus the loop’s own process overhead left little headroom — long sessions would churn and occasionally run the system out of memory outright. The practical fix was keeping the context window deliberately small, which controlled memory but traded away the thing a longer context is good for: retaining more history within a single session. That’s a real efficiency ceiling on long-running sessions, not just a tuning knob, and it’s the clearest evidence that “fully local” has a hardware cost the cloud-hosted version of this same workflow doesn’t.

Lessons Learned

• Small, local models need more scaffolding, not less. Everything that a larger hosted model would get right by default — staying on-task, not repeating itself, emitting a raw command instead of prose — had to be enforced explicitly with a validation layer in front of every execution step.
• Re-deriving state beats trusting a plan. Checking “what’s actually true in the codebase right now” before choosing the next action made the loop far more resilient to an individual step going sideways than committing to a fixed sequence up front would have been.
• “Fully local” is a real ceiling, not just a cost switch. Swapping in a smaller open-weight model isn’t a drop-in substitution for a cloud model — it’s a different reliability profile that has to be designed for from the start. As a proof-of-concept, this confirmed the autonomous-loop pattern itself is sound; the open question it left for any production use is whether the guardrail overhead is worth it versus just paying for a stronger hosted model.
• Zero API cost isn’t zero cost. Removing the cloud dependency just moves the bill to local hardware: on 32GB of RAM, the model and loop together left little margin, and keeping the context window small enough to avoid running out of memory directly limited how long a session could usefully run.