How it works
Two models, one computer
Here is what happens between dropping a folder on ToneTrace and seeing a tagged library. In the current build every step runs on your own machine.
Decode
FFmpeg, bundled with the app, decodes the file to raw audio on your machine. Silence is set aside, so a two-second kick and a four-minute stem are both judged on the parts that actually contain sound.
Listen for instruments
YAMNet, an open model from Google trained on the AudioSet collection of everyday and musical sounds, listens across the whole active file rather than taking a single snapshot. Its verdict reflects what is present for most of the file, so a brief cymbal hit does not outrank the guitar underneath it.
Listen for character
CLAP, an open language-audio model from LAION, hears audio and reads text in the same space. ToneTrace uses it to turn the file into an audio fingerprint, then compares that fingerprint with a curated vocabulary of descriptions covering genre, mood, production technique and sonic quality. The closest matches become tags, and the same reading places the file on the mood and energy pad.
Settle close calls
When YAMNet is torn between close relatives, a guitar and a ukulele, say, CLAP's hearing is weighed in before the tag is chosen. Neither model can overrule the other on its own. A clear ukulele stays a ukulele; an ambiguous strum gets the guitar that CLAP hears in it.
Note where each sound happens
Knowing a file contains a guitar is less useful than knowing where. As the instrument model listens, ToneTrace keeps the timeline: which sound it heard, when it started, when it stopped and how sure it was. Those regions are what the waveform colours in, and what you drag out when you want one moment rather than the whole file.
Remember
Before any of this, the file's bytes are hashed. Tags, waveform and fingerprint are stored against that hash in a local database, so the same audio is never analysed twice, even after it has been moved or renamed.
Show your work
Each instrument tag shows the model's own probability for that sound. Each character tag shows how closely the audio fingerprint matched its description. The numbers are the models' honest opinions, not a score tuned to look confident.
Why tags carry a confidence
A tag without a number invites you to trust it blindly. A tag with a number tells you when to double-check. A kick at 90 % and a kick at 35 % are both worth showing, but you will treat them differently, and so will the export: by default only tags at or above 30 % are written into files.
The tag vocabulary for mood and genre is still being refined. Some descriptions are more useful than others, and beta testers' feedback on which tags are wrong is exactly what we are asking for.
Illustrative values from a strummed acoustic chord in the test set. Instrument confidence is a probability; genre and mood confidence is a similarity.
Why a pad and not a dropdown
Mood is not a list you pick from. A sound is somewhere between dark and bright, and somewhere between still and driving, and the useful question is usually “near this, but calmer”. A pad lets you ask that in one gesture, and widen or tighten the answer without retyping anything.
Both axes come from the language-audio model's reading of the audio itself, so they work on files nobody has ever named, sorted or tagged — which is most of anyone's library.
Built on open components
YAMNet and CLAP are published open models. They run inside the app through ONNX Runtime, FFmpeg handles decoding, and the library lives in SQLite. Each is credited with its license on the credits page, and the full license texts ship inside the app.
Hear it on your own samples
The beta is free. Pricing will be announced at launch.