Prediction Market Edge Finder

The Problem

This one started as a personal challenge rather than a work problem: could I take a domain I had zero prior background in — prediction markets — and run the full arc, from cold-start discovery research through a resolved architecture to a working, tested application, in a single weekend? The test wasn’t just “can I build fast.” It was whether a disciplined discovery-first process (research the domain, red-team the thesis, resolve every design decision on paper) actually holds up when compressed into a weekend, or whether it collapses under time pressure into hand-waved architecture and untested assumptions.

Prediction markets turned out to be a good stress test for that process: Kalshi and Polymarket are large, liquid, regulated event markets, but most retail traders have no real probabilistic edge over consensus prices on liquid contracts — and naively asking a frontier LLM to “predict the outcome” doesn’t reliably beat the market either, since research shows LLM forecasters are roughly accuracy-competitive with markets, not clearly better. Finding a real angle meant digging past the obvious approach into the market microstructure itself: the opportunity isn’t out-forecasting the crowd, it’s the same pattern as dealer-positioning analysis in options markets — finding structural, behavioral, or attention-driven mispricings rather than competing on raw prediction skill. That thesis, plus two platforms with their own API quirks, fee structures, and edge cases, gave the weekend enough real depth to be a legitimate test of the process, not a toy problem.

The Solution

A pipeline that ingests live prices from both platforms, runs a set of mechanical and heuristic detectors to flag candidate mispricings (near-arbitrage price-ordering violations, cross-platform price gaps, thin markets, extreme “longshot” prices, and news-driven overreactions), then routes the ambiguous cases through a cost-gated ensemble of LLMs for research and probability estimation. Model outputs are explicitly shrunk toward the market price to correct for known LLM overconfidence on genuinely uncertain future events, and ensemble agreement is used as a confidence signal in its own right. The result is a ranked daily digest the user reviews in under a minute, with drill-down detail on each candidate and a running backtest log that closes the loop on which flags were actually accurate. The tool deliberately never issues a “trade this” verdict — it ranks and explains; the human decides and executes manually.

Outcome & Impact

Working end-to-end MVP shipped in a single ~7-hour build session, following a multi-day upfront design phase: 21 commits across 8 phases, a 12-model schema covering every pipeline stage, and 33 test files spanning heuristics, calibration, cost budgeting, and all three LLM provider integrations, with every external network call mocked. Ingest, all five heuristic detectors, the whale/flow signal layer, the three-model LLM research pipeline, and a fully wired three-screen dashboard are implemented and locally runnable — not scaffolding. The system is currently a tested local MVP, not yet deployed to production; per the project’s own red-team assessment, it’s being treated as a research/decision-support tool to validate over a real sample of flagged candidates before any claim of a durable trading edge, rather than as a profit engine from day one.

Metric	Detail
Build timeline	Multi-day design phase + single ~7-hour implementation session
Commits / phases	21 commits across 8 planned build phases
Data model coverage	12 models across base (price/market) and signal (flags, estimates, calibration, backtest) layers
Test coverage	33 test files; all external LLM/API calls mocked, no live network dependency in the suite
Pipeline completeness	Ingest → heuristic detection → LLM scaling gate → 3-model research → calibration → ranked dashboard, all implemented
Deployment status	Locally runnable MVP; cloud deployment intentionally deferred as a separate, deliberate step

My Role & Contributions

Aspect	Detail
Role	Sole architect and engineer, end to end (product/architecture design done in collaboration with Claude)
Team size	1
Timeline	One weekend: domain research and discovery distilled into a dozen working documents and one consolidated architecture brief, followed by a single ~7-hour build session that shipped a working MVP across 8 phases
Scope	Ingest clients for both platforms, the full heuristic detection layer, LLM orchestration and calibration, the data model, and the three-screen dashboard
Key decisions	Layered dependency-boundary enforcement between routing/business-logic/data-access code; a mechanical fast-path for the one near-riskless mispricing category; a shrinkage-based calibration formula to counter LLM overconfidence; curated (not automated) cross-platform market matching; a hard daily cost ceiling on LLM spend

Technical Overview

[Ingest] → [Heuristic Filter/Flag] → [LLM Scaling Gate] → [Multi-LLM Research] → [Calibrate + Rank] → [Dashboard]
                  │                                                                     ▲
                  └── Mechanical near-arbitrage check ─────────────────────────────────┘
                       (bypasses gate, research, and calibration entirely)

The backend is a Node.js/TypeScript service (Express + Prisma/PostgreSQL) split into a web process and an always-on worker process that runs the ingest/detection/research loop on an in-process scheduler — no message queue, since a solo-user research tool doesn’t yet generate the volume that would need one. Both processes share the same business-logic and data-access modules, enforced by a strict one-way dependency rule (routers only handle HTTP, services hold pure business logic, only the repository layer touches the database or external APIs), checked via ESLint import restrictions rather than convention alone.

LLM orchestration

Three independent frontier models research each ambiguous candidate in parallel and return a structured probability/confidence/reasoning estimate. A cheap single-purpose model acts as a pre-filter (“scaling gate”) that decides whether a candidate has enough real information to justify full ensemble research, and fails open on infrastructure errors rather than silently dropping a candidate. A hard daily spend ceiling acts as a safety valve, not a routine constraint.

Data model

Two layers: a factual base layer (markets, series, price history) that’s a useful price-history tool on its own, and a derived signal layer (candidate flags, model estimates, calibrated consensus, backtest outcomes) that’s opt-in enrichment on top — never required to join on every market.

Frontend

A React + TypeScript SPA (Vite, Tailwind, TanStack Query, Recharts, Zustand) implementing all three screens: ranked daily digest, market detail drill-down with overlaid price history and per-model reasoning, and a backtest log.

Challenges & Key Decisions

Doing real discovery in a domain I’d never worked in, on a deadline

The easy failure mode here was skipping straight to building something that looked plausible without actually knowing whether the underlying thesis held up. Instead, discovery came first: reading the actual research on LLM forecasting accuracy and calibration, working out the market-microstructure argument for where an edge could plausibly exist, and explicitly red-teaming that thesis before writing a line of architecture — including writing down the reasons the edge might not be real. That work produced a dozen separate discovery documents, which then had to be consolidated into one coherent, resolved brief before implementation could start, so the weekend’s time budget was spent mostly on getting the “what” and “why” right, not on the “how.”

Correcting for a known LLM failure mode, not just using the model’s output

Research on LLM forecasting shows frontier models are systematically overconfident on genuinely unknown future events, and that “thinking” modes can make calibration worse, not better. Taking a model’s stated probability at face value would have baked that overconfidence into the ranking. Instead, every raw ensemble estimate is shrunk toward the current market price before use, with the shrinkage strength stored as a named, backtestable constant rather than hardcoded logic, so it can be refit once enough resolved outcomes exist.

Finding the one category worth a mechanical shortcut

Most candidate categories genuinely need LLM research to judge whether a price gap is signal or noise. One category — nested markets on the same event priced out of their required logical order — is a mechanical fact, not a probability judgment, and can be detected with zero model calls. Carving that path out as an explicit bypass (skipping the cost gate, ensemble, and calibration entirely) kept the one near-riskless case cheap and fast while routing the genuinely uncertain cases through full research.

Controlling cost without silently degrading the signal

Running a multi-model ensemble against every flagged candidate would be neither cheap nor necessary — many flags are gap-in-price-but-nothing-behind-it noise. Rather than an arbitrary sampling cutoff, a cheap pre-filter model judges whether a candidate carries enough real differentiating information to justify full research, and that judgment is itself logged and backtestable, so its own accuracy — not just the ensemble’s — can be checked over time. A hard daily budget sits underneath as a ceiling; candidates beyond it fall back to a raw-gap-size ranking instead of being silently dropped.

Precision over coverage on cross-platform matching

Comparing the same event’s price across two platforms looks like free arbitrage detection, but differing resolution and tie-breaking rules frequently make two “matching” markets not actually equivalent questions — the main false-positive risk in the system. Rather than automated text or embedding matching, cross-platform pairs are a curated, human-asserted watchlist, and even a matched pair still goes through full research, including an explicit model judgment on whether the two questions are really equivalent, rather than being auto-flagged as riskless.

Tech Stack

Layer	Technologies
Backend	Node.js, TypeScript, Express, Prisma
Database	PostgreSQL
Scheduling	node-cron (in-process worker loop)
Frontend	React, TypeScript, Vite, Tailwind, TanStack Query, Recharts, Zustand
LLM research	Claude, Gemini, and an open-weight model, dispatched directly against provider APIs
Testing	Vitest, with all provider/network calls mocked

Lessons Learned

• A single mechanical bypass can be worth more than it looks. Spending design effort finding the one category that’s a logical fact rather than a probability judgment (the nested-market ordering check) paid off disproportionately — it’s the fastest, cheapest, most confident signal in the whole system, and it shipped first specifically because it needed no external dependencies to prove out the pipeline end to end.
• Calibration has to be a re-tunable constant, not a guess baked into code. Naming the shrinkage factor and the agreement thresholds as isolated, backtestable values (rather than folding them into scoring logic) means they can be refit against real outcomes later instead of needing a rewrite.
• Precision beats coverage when false positives are expensive to reason about. Choosing a curated watchlist over automated cross-platform market matching traded some coverage for confidence that every flagged pair is worth a human’s attention — the right tradeoff at solo-user scale, where every false positive costs a manual review cycle.
• Cost control belongs at the decision layer, not just the pricing layer. A cheap, logged, backtestable pre-filter model turned “should we spend money researching this” into a decision with its own accuracy record, rather than an unaudited assumption.