AuditionMate
0%
System Architecture · by Ryan Chun

AuditionMate Architecture

An end-to-end pipeline for precise violin intonation analysis — from raw audio to teacher-grade feedback.

Scroll to explore
MP3 INPUT  →  SCORE PARSER  →  DTW ALIGNMENT  →  ONSET DETECTION  →  PYIN F0  →  HARMONIC NMF  →  CLAUDE API  →   MP3 INPUT  →  SCORE PARSER  →  DTW ALIGNMENT  →  ONSET DETECTION  →  PYIN F0  →  HARMONIC NMF  →  CLAUDE API  →  
01 / Input

MP3 & YouTube
Ingestion

AuditionMate accepts two routes: a local MP3 from data/recordings/, or a YouTube URL fetched via yt-dlp and cached by video ID so repeated runs skip re-download. Both paths converge on a single local MP3 before any processing begins — letting a student paste a link to their own practice recording or a teacher pull a reference performance.

02 / Score

MusicXML
Parsing

Scores arrive as .xml or compressed .mxl files exported by MuseScore, Sibelius, or Finale. score_parser.py extracts every note's pitch, start time, duration, measure number, and beat position. A measure-range parameter lets you isolate a single movement from a multi-movement file.

03 / Features

Audio Features
& Classification

audio_features.py computes a CQT chroma representation — a 12-dimensional time series of active pitch classes. passage_classifier.py labels each score region slow or fast, directly setting the hop length for feature extraction. Fast passages use a finer 256-sample hop; slow movements use 512.

04 / Alignment

Dynamic Time
Warping

DTW elastically aligns the score chroma to the audio chroma using cosine distance — the architectural keystone of the pipeline. The warping path maps every written note to its precise audio timestamp regardless of how much the performer rushed, dragged, or took tempo liberties.

05 / Timing

Onset
Detection

For fast passages, onset_alignment.py detects precise bow attacks via spectral flux novelty, using backtrack=True to snap each peak to the energy minimum before the attack transient. These timestamps replace DTW estimates for fast-passage notes, achieving millisecond-level precision where DTW had ±2–3 note uncertainty.

06 / Pitch

F0 Estimation
via pyin

pyin returns per-frame Hz estimates with voicing confidence scores. Frames below 0.65 confidence are discarded; survivors are octave-corrected then gated against an 80-cent plausibility threshold. Vibrato is reported independently — rate and width at 4.5–8.5 Hz — not folded into intonation error.

07 / Chords

Harmonic
NMF

For double-stops and chords pyin cannot track, harmonic_analysis.py builds score-informed spectral templates for each expected pitch and runs NMF to recover sub-bin intonation via parabolic interpolation. Coverage typically rises from ~17% to 60–70% on chorally dense movements.

08 / Summary

Intonation
Summary

summarize_intonation aggregates median pitch error, standard deviation, sharp/flat counts, the five worst-offending notes by measure and beat, and vibrato parameters. Median rather than mean prevents the Jensen's inequality bias introduced by naive Hz-averaging.

09 / LLM

Claude API
Feedback

prompt_builder.py formats the summary into a structured prompt with measure-annotated statistics and a system prompt directing Claude to respond as an experienced violin teacher — specific, physical, and limited to 2–3 actionable issues connecting numerical errors to their likely physical causes.

10 / Output

Feedback
Report

The pipeline writes a timestamped .txt to outputs/ and returns to the caller. A complete run takes 60–90 seconds — dominated by DTW alignment — producing measure-grounded, teacher-style feedback grounded in precise signal analysis.

Built by Ryan Chun, B.S. Computer Science & B.M. Violin Performance at Northwestern University. Contact: ryanchun2029@u.northwestern.edu

View on GitHub → LinkedIn →