Develop · Build
How it came together.
- 01
Telegram as the universal capture surface
Capture began as iOS Shortcuts for photos and a Chrome extension for URLs: two surfaces, two auth flows, two failure modes. Telegram collapses both into one bot reachable from any phone, laptop, or iPad, with a native share sheet from every iOS app. A webhook on the Bot API, user ID allowlisted, turns it into a universal inbox for photos, voice notes, documents, and links, all routing through one /ingest endpoint. New surfaces ship as thin clients.
Backend Engineering The floor is the point. Each rung is faster and dumber than the one above it. The bottom rung never fails, which is what lets the top two be allowed to. - 02
The classifier became a provider ladder
Classification started heuristic-first because Ollama on a Dell CPU took 30 seconds to sort a 1500-char prompt into a 12-option enum. Moving the daemon onto a 4090 changed the economics: the same call is now warm in about a second, so the ladder runs Cloudflare first, local Ollama second, and the phrase-aware heuristic last. The heuristic still earns its place as the floor - about 80 percent accurate in under 50ms - which is what makes the chain safe to time out aggressively: nothing in the ingest path waits on a model that isn't answering.
AI & Machine Learning - 03
Vision moved to Cloudflare after every local tuning failed
Local llava-7b on the Dell was tuned every way available (sharper pre-resize, shorter prompts, thread pinning, longer keep-alive) to beat a 90s timeout; the floor stayed at 162s per image, because vision-token tokenisation dominates prompt-eval on CPU, not generation. Cloudflare Workers AI runs the same class of model on GPU, 10-15s warm round-trip, free tier covering 10k requests a day. Only the photo bytes leave the device for that step, never the vault - and it's one env var to move back local once a GPU mini-PC replaces the Dell.
AI & Machine Learning - 04
Perception in the cloud, the vault always local
Voice started as local Whisper-base ONNX inference via @xenova/transformers - slow enough on CPU that the bot needed a 'transcribing…' ack so the user knew it heard them. Cloudflare's Whisper turbo returns in two to five seconds, so it's now the default; local stays as the offline fallback, one env var to switch. Same rule as vision: only the audio leaves the device, never the vault.
AI & Machine Learning - 05
A webhook that survives the home box being down
The bot used to long-poll from a process on the Dell: a bad socket could strand it in a retry loop for hours, and a reboot at home took capture down with it. It's a Vercel-hosted webhook now, so there's no poll loop to wedge. The remaining failure mode, the home backend being unreachable, is handled by probe-and-queue: anything the webhook can't forward lands in a pending file that a drainer replays once the box answers again.
Backend Engineering - 06
Vault RAG on one retrieval index
One index sits beside the vault and matches three ways at once: on the words themselves, on what a passage means, and on what a saved image looks like, with the three sets of results merged into a single ranking. Ask the bot what you saved about some half-remembered trick and it searches the vault, answers from the passages it found, and cites them, or, when the local model is slow, drops to returning the sources without the prose rather than making you wait. Nothing is trapped in the index: it rebuilds from the markdown at any time, and the vault stays the source of truth.
Data Pipelines
Develop · Forks
Decisions on the record.
The few calls worth defending. Each one is a fork; the other branch would have been a different project.
Decision · 01
Why hybrid retrieval, text and image together
Image-dense retrieval catches 'show me other things I've seen that look like this,' something most personal-knowledge tools skip. Pair it with semantic-text and keyword search so named entities stay findable, and fuse the three without a weight tournament.