01
At a glance
AI transcription of guitar audio is still an unsolved problem. I built a full-stack app with an ML audio pipeline around that limitation, turning any song into sheet music I could play along, kinda.
Topic
Audio Processing, Music Notation
Type
Hobby Project
Stack
Next.js, TypeScript, Drizzle ORM, ML audio pipeline
Timeline
2 weeks, 2026 |
5
ML pipeline steps to turn an audio file into music notation.
65
Tests covering audio processing and the web app.
-88%
Cut in token usage by auditing and restructuring project docs after noticing an AI agent burning context on every prompt.
Directed an AI-built system, splitting work between Claude for planning and reasoning and OpenCode models for execution.
02
Why I made this
I'm a beginner guitarist
I'm really into melodic rock and jazz. Half the songs I want to learn don't have tabs, and the ones that do are written for someone way better than me. I aim to add an adjustable difficulty, like a video game difficulty slider, so I can actually play along instead of giving up.
My goal is to build a tool that turns any song into sheet music.
The real goal is learning to build with AI
I used knowledge I gathered from the past like thinking about modularity and maintainability.
I split the system into two halves: audio processing that relies on open source models to avoid ongoing costs stays local, while the web app lives online, and a shared hosted database acts as the bridge between them.
The system evolved with each day becoming what I call "The Hub" which now holds my building workflow.
This project is my long-term playground for testing ideas and tools.
03
Recognizable not accurate
Recognizable not accurate
Turning a song recording into a note-perfect tab is a genuinely unsolved research problem. I didn't know that going in.
Algorithms struggle to separate guitar audio from a full mix, especially in rock where distortion and effects bury everything. I tried a more simple acoustic song and got a much cleaner result.
Like listening to a walkie-talkie
Audio quality was bad and the extracted notes were full of noise from other instruments.
After two days and 25 commits I landed on Basic Pitch and htdemucs. I abandoned what I made and started fresh with a cleaner structure.
Since accurate note representation was off the table, I settled for recognizable enough to play along to.
Delegating work to cheap models
I use Claude for planning, reasoning, and verifying, and OpenCode models for execution. Each demands a different kind of thinking, which challenges me every day.
I set some ground rules before writing a single line: audio stays local, built with modularity and contracts in mind, and every plan gets read before it runs.
04
Design
3
Three different ways to read and interpret music notation.
To keep experiments small, the first notes were displayed as ASCII characters, plain text, quick to build.
As a beginner I found them hard to read, so I added a fretboard view as a second way to see the same thing.
Placing notes accurately on a fretboard is a separate problem.
The first fretboard was a proof of concept but useless. The same note can be played in more than one spot, and tabs online differ for the same song because people transcribe by ear differently.
I rebuilt it as a sequenced strip, chained in play order, and let the model decide where to place notes for now.
Sheet music became a third view, the most compact way to actually read notes, the same way musicians do. Since it reads linearly, I added a metronome that I calibrated with the original song's bpm.
The sheet view of Seven Nation Army, metronome calibrated to the song's 123 bpm.
Reading music notation
Guitar notation is written upside down from how you actually experience the instrument.
I get confused looking at the bottom note and playing the top string.
I added a button that flips the string order, for an easier left-to-right, top-to-bottom read. String thickness is a small visual aid that makes the diagram read more like a real fretboard.
Flipping note orientation helps me understand where the note lives on the fret board.
05
The black box
-88%
Reduction in token consumption after refactoring project rules and decision files.
I noticed every prompt kept eating tokens faster, no matter how small the change. There was bloat.
A project audit revealed that my decisions file had grown to 5,452 words and together with reference and architecture files, around 45k tokens were burned just to start a new session.
And I thought documentation was a good thing.
Folders and rules as a system
I looked into icm-architect and other repositories for principles and solutions on how to structure my projects to keep them small and predictable.
While my workflow nowadays looks different, with my early experimentation this helped to cut down on a lot of bloat.
Numbered pipeline folders and per-stage status files.
One index branching into single-topic folders.
Trimmed main files with sensible character limits.
Clear rules written in an SQL DB that is read at new session start.
Unit, e2e and playwright testing for most parts.
No matter how good the system is, AI finds creative ways to eat your tokens.
06
New superpowers
A never ending work in progress
This is a project I'm actively working on. My audio pipeline needs rewriting with libraries better suited to what I'm after, and the web app runs on design system variables but is still costly to iterate on as a whole.
What is next for this project:
Ship a simple Control Center, to run pipeline processes, dev servers, and git commands without touching the terminal.
Extend the design system with a proper UI so I can make changes visually.
Rewrite the web app in SvelteKit for performance.
Rewrite the audio pipeline with better libraries for notation accuracy, last, since it works well enough to leave alone for now.
I'm now a designer who finally has tools to build
I can ship working software, not just hand off design screens. The problems I heard engineers bump into throughout my career are now mine to solve.