Paper as an Interface: A reMarkable Layer for a Personal AI System

The Problem

I keep a personal knowledge base — a folder of markdown notes acting as a “second brain,” paired with an AI coding assistant that reads and writes to it. Separately, I do my weekly planning and note-taking by hand on an e-ink tablet, on purpose, because paper without notifications is where I actually think. The two systems didn’t talk: anything I planned on paper had to be manually retyped into my notes, and anything I wanted to read on the tablet — long-form research, PDFs — had to be exported and reformatted by hand, often unreadably, since most print layouts break on a single-column e-ink screen. The obvious fix, syncing the tablet like a folder, has a sharp failure mode: any tool that writes back to the device risks landing generated text on a page you write on, or overwriting a week of handwriting outright.

The Solution

I built a small set of purpose-built integrations — coding-assistant skills plus a read-only server connection to the tablet’s cloud API — that treat the device as an asymmetric interface rather than a sync target. Handwritten planning and notes flow one way, from tablet to notes, automatically and without retyping. Long documents flow the other way, from notes to tablet, reformatted for e-ink reading. Nothing is ever generated onto a page I write on, and nothing on the device can be altered or deleted by the system — that boundary is enforced by the connection itself, not by good intentions.

Outcome & Impact

The weekly planning notebook I write by hand is now reflected across every day’s notes the same day, with zero manual re-entry — previously a recurring block of manual retyping every week, now zero. The read-only device connection exposes six read tools (browse, read, image, export, recent, status) and enforces “no writes” at the server flag level rather than at the skill level, closing off an entire class of accidental-overwrite bugs before they can happen. A magazine or journal PDF that would otherwise be unreadable in its original multi-column layout on an e-ink screen is now converted into a fully reflowed, single-column EPUB — complete with a working table of contents — through a fully deterministic pipeline with no model calls in the extraction path. The system currently runs four independent skills, each scoped to one direction of the paper/digital boundary, with zero incidents of generated content landing on a handwritten page since the read-only redesign.

My Role & Contributions

Aspect	Detail
Role	Sole designer and builder (personal project)
Team size	1 (solo)
Timeline	Built incrementally over several weeks, September 2026
Scope	4 coding-assistant skills, a read-only device server connection, and a security/format investigation that drove the architecture
Key decisions	Chose read-only, one-directional pipelines over bidirectional sync; enforced the read-only boundary at the connection layer, not the skill layer; kept a deterministic (non-AI) pipeline for document reflow

Technical Overview

The system splits cleanly into two directions across the device boundary, with no path that crosses both.

Ingress (tablet → notes): One skill polls the tablet’s own handwriting-to-text conversion output — content I’ve explicitly opted to convert — checkpointed by document ID so nothing is re-processed or missed inside its expiry window. Action items route to a daily task list; everything else routes to a dated capture log. A second skill reads a hand-drawn weekly planner notebook directly off the device through the read-only server connection, using image reads plus vision (not flat text extraction) so it can follow arrows, cross-outs, and margin notes that plain OCR would flatten, then fans each day’s items into that day’s note.

Egress (notes → tablet): A third skill converts PDFs into single-column, reflowable e-books through a deterministic, geometry-based extraction pipeline — no model in the extraction loop, only in a metadata pass and a spot-check. A fourth skill writes long-form research syntheses into my notes and hands off a link that a small local reading pipeline turns into a device-ready document via a public read-view extraction step.

   TABLET (handwritten ink)              PERSONAL NOTES (markdown)
   ┌──────────────────────┐               ┌───────────────────────┐
   │  Converted notes API │ ──ingress──▶  │  Daily notes          │
   │  (opt-in, ID-checkpt)│               │  (tasks + capture log)│
   ├──────────────────────┤               ├───────────────────────┤
   │  Weekly planner      │ ──ingress──▶  │  Weekly planner note  │
   │  notebook (read-only │   (vision)    │  + fanned daily tasks │
   ├──────────────────────┤               ├───────────────────────┤
   │  Reflowed EPUB /     │ ◀──egress───  │  Source PDF           │
   │  outlined PDF        │  (geometry)   │  (magazine/journal)   │
   ├──────────────────────┤               ├───────────────────────┤
   │  Rendered synthesis  │ ◀──egress───  │  Long-form research   │
   │  (via read-view link)│ (extraction)  │  note                 │
   └──────────────────────┘               └───────────────────────┘
        read-only server connection (no upload / delete / rename)

Why deterministic reflow, not a model

The PDF-to-EPUB path needed to run unattended across dozens of differently-designed print layouts and produce reproducible output every time, so the extraction logic — column detection by block geometry, drop-cap re-attachment, same-baseline text splicing — is plain code, not a model call. The assistant’s job is limited to the one genuinely judgment-based step (pulling clean bylines and deck text from a contents page) and a final spot-check before shipping.

Challenges & Key Decisions

Choosing where the AI stops and the pen starts

An earlier version of the planner sync generated a printed planner page and uploaded it back to the device, replacing the previous copy each run. I dropped this design deliberately: generated text landing on a page meant for ink corrupts the one thing that made the device worth using, and a destructive replace risked losing a week of handwriting if anything went wrong. The redesign made the sync strictly one-directional — nothing is ever written back — which also means nothing can be lost, because nothing is ever deleted or replaced.

A read/write credential is bearer access to an entire personal library

While scoping the device connection, I found that the same cloud credential type defaults to full read/write/delete access across the whole document library, with no per-document scoping. Rather than rely on the coding assistant’s instructions to “just not write,” I ran the server itself in a read-only mode, so the restriction is enforced structurally and can’t be bypassed by a bad prompt, a bug, or a future change to the instructions.

Handwriting recognition and OCR are merged with no way to tell them apart

Reading the device’s own conversion output for an annotated document, I found that its handwriting recognition and its OCR of the printed page are merged into a single transcript with nothing marking which text came from which source — my own handwritten note sat in the same list as text lifted verbatim off the printed page, including a truncated word. I resolved this by treating the structured “action items” field as the trustworthy channel and flagging general captured text as raw material to double-check against the source, rather than presenting either to myself as verified.

Tech Stack

Layer	Technologies
Orchestration	Claude Code (skills)
Device connection	MCP (Model Context Protocol), read-only server mode
Ingestion scripting	Python
Document reflow	Python, PyMuPDF
Device API	GraphQL (tablet’s converted-notes endpoint)
Notes storage	Markdown, Obsidian

Lessons Learned

• Enforce a safety boundary at the layer that can’t be talked out of it. A read-only flag on the server survives a bad prompt or a future edit to the instructions; a rule written only in a skill’s own instructions does not.
• A physical interface benefits from asymmetric design, not bidirectional sync. Trying to make the device a two-way sync target created the one real failure mode (overwriting handwriting); splitting ingress and egress into direction-specific pipelines removed it entirely.
• Save the model for genuinely ambiguous judgment calls. The document-reflow pipeline is fully deterministic except for one metadata pass — keeping the bulk of it as plain code made it reproducible across wildly different print layouts, something a model call re-run each time would not guarantee.