TabbyTab
AI transcription of guitar audio is still an unsolved problem. I built a full-stack app with an ML audio pipeline around that limitation, turning any song into sheet music I could play along, kinda.
At a glance
Built a full-stack app with zero prior coding background: Next.js, TypeScript, Drizzle ORM, ML audio pipeline.
Directed an AI-built system, splitting work between Claude for planning and reasoning and OpenCode models for execution.
Caught two AI missteps before they shipped: a 1.9GB unintended repo push, and a permission scope that quietly expanded mid-task.
Cut token usage 85% by auditing and restructuring project docs after noticing an AI agent burning context on every prompt.
Why I made this
I'm a beginner guitarist who is into melodic rock and jazz. Half the songs I want to learn don't have tabs, and the ones that do are written for someone way better than me.
So I'm building a tool that turns any song into sheet music with adjustable difficulty, like a video game difficulty slider, so I can actually play along instead of giving up.
The real goal is learning to build with AI: picking up engineering concepts and workflows as I go. This project is my long-term playground for testing ideas and tools.
Modular by design
The system splits into two halves: audio processing stays local, the web app lives online, and a shared hosted database acts as the bridge between them. Local processing relies on open source models to avoid ongoing costs.
I use Claude for planning, reasoning, and verifying, and OpenCode models for execution. Each demands a different kind of thinking, which challenges me every day.
Constrained from the start
Just a laptop and a Claude Pro subscription made it possible to start building something I couldn't have before.
I set some ground rules before writing a single line:
Process and store audio locally.
Build with modularity and contracts in mind so I could swap modules later.
Read every plan myself before it ran. Never accept a decision I didn't understand.
The last one became real fast. Partway through, Claude wanted to push 1.9GB of my project files straight to the repo. I stopped the leak mid-process.
Hearing before building
I didn't want to build a whole app around an idea that might not work. Before anything else, I needed to test instrument separation and note extraction, the rest of the app hinges on both actually working.
I picked songs I already knew how to play by ear, so I could judge the results instead of guessing, and set a rating scale from unusable → good.
The audio pipeline runs in four steps on my Mac: ingest audio, separate instruments, extract notes, visualize as sheet music.
The first pass was rough like listening to a walkie-talkie. Audio quality was bad and the extracted notes were full of noise from other instruments. I went through three different solutions before settling on htdemucs and Basic Pitch.
Recognizable not accurate
Turning a song recording into a note-perfect tab is a genuinely unsolved research problem. I didn't know that going in.
Algorithms struggle to separate guitar audio from a full mix, especially in rock where distortion and effects bury everything. I tried a more simple acoustic song and got a much cleaner result.
Since accurate note representation was off the table, I settled for recognizable enough to play along to.
Vercel can’t talk to my laptop
I'd started with SQLite for storing sheet music. While processing stayed local, the sheet music itself needed to live online.
I decided to host the web app on Vercel, and only then realized that it couldn’t reach the database sitting on my laptop.
I compared Cloudflare D1 and sqlite.ai before landing on Turso (libSQL), paired with Drizzle. It was simple to deploy, and not something I'd be locked into if it stopped working out. Here my modularity principle showed up somewhere I didn't expect it to matter.
Where I had the most fun
To keep experiments small, the first notes were displayed as ASCII characters, plain text, quick to build. As a beginner I found them hard to read, so I added a fretboard view as a second way to see the same thing.
The first fretboard was a static overview of all note positions. Proof of concept but useless.
Placing notes accurately on a fretboard is a separate problem. The same note can be played in more than one spot, and tabs online differ for the same song because people transcribe by ear differently.
I rebuilt it as a sequenced strip, chained in play order, and let the model decide where to place notes for now.
Sheet music became a third view, the most compact way to actually read notes, the same way musicians do. Since it reads linearly, I added a metronome that I calibrated with the original song's bpm.
Reading music
Sometimes I'm only learning the main riff, so I added an option to select a fragment and loop it, letting me rehearse without manually restarting.
Guitar notation is written upside down from how you actually experience the instrument. I added a button that flips the string order, for an easier left-to-right, top-to-bottom read. String thickness is a small visual aid that makes the diagram read more like a real fretboard.
Principles as token burners
I noticed every prompt kept eating tokens faster, no matter how small the change. There was bloat.
A project audit with Claude revealed that my decisions file had grown to 5,452 words covering eight separate topics. Together with reference and architecture files, around 15k tokens were burned just to get the AI up to speed. And I thought documentation was a good thing.
I wanted to adapt icm-architect folder structure. Their documentation and principles offered two things that solved my problems: numbered pipeline folders and per-stage status files. I did the rest of the refactor myself: one index branching into single-topic folders, trimmed main files, scattered ideas consolidated into a real backlog, and clearer rules written in their place. That cut token usage by 85%.
No matter how good the plan looks, AI will find creative ways to create waste.
Caught it before it shipped
I tested the zero-shared-memory handoff using a free OpenCode model with no memory of this project. It oriented itself from the files alone and picked up the next task cleanly.
Then, mid-handoff, my own rule, ask before every commit, got skipped. My instruction to go ahead with the handoff had quietly been read as broader permission to commit. I caught it before anything pushed, stopped it, and wrote the rule permanently into the project's rules so it can't happen quietly again.
New superpowers, still learning
This is a project I'm actively working on. My audio pipeline needs rewriting with libraries better suited to what I'm after, and the web app runs on design system variables but is still costly to iterate on as a whole.
Fixing my own workflow first means everything after moves faster.
What's next:
Ship the first version of the Control Center, to run pipeline processes, dev servers, and git commands without touching the terminal.
Build a proper UI for the design system so it's actually usable.
Rework the folder structure and audit docs and rules files to trim the bloat.
Rewrite the web app in SvelteKit for performance.
Rewrite the audio pipeline with better libraries for notation accuracy, last, since it works well enough to leave alone for now.
But I'm now a designer who finally has tools to build, not just hand off. The problems I heard engineers bump into throughout my career are now mine to solve.
Choosing an ambitious project with limited resources turned out to be the best way to get hooked on building.