Notalo › Learn › How optical music recognition works, and why photos are harder than PDFs

How optical music recognition works, and why photos are harder than PDFs

By Taekyung Lee · 2026-10-10

Optical music recognition, OMR, is to sheet music what OCR is to text: software that looks at a picture of a page and works out what is written on it. It has existed since the 1960s and it is still not solved, for reasons that become obvious once you try to build one. This is how Notalo's works, how the established tools work, and where each of them fails.

Why it is harder than reading text

Printed text is a line of separate symbols, each of which means one thing. Music is two-dimensional: a notehead's meaning depends on its vertical position, on a clef that may be several centimetres to the left, on a key signature at the start of the line, on an accidental earlier in the same bar, and on a stem, flag, beam and dot that may each be attached to it or to a neighbour. And everything sits on top of five lines that cut through every symbol. An OCR engine can assume the letters are separated by white space. An OMR engine cannot assume anything is separated from anything.

The classic pipeline

Almost every OMR system, from the 1990s commercial scanners to the open-source Audiveris engine that MuseScore's PDF import is built on, does the same five things in order.

1. Binarize. Turn the image into pure black and white. On a rendered PDF this is trivial. On a photo it means first correcting the lighting, because a single global threshold will lose the notes in a bright patch and merge the lines in a dark one.
2. Find and remove the staff lines. Staff lines are long, thin, horizontal, evenly spaced, and in groups of five. Finding them is a matter of looking for rows with many black pixels in a row at a regular vertical interval. Removing them is harder: a line passes through every notehead, and taking it out must not cut the notehead in half. Audiveris removes the lines and then repairs the symbols; Notalo leaves the lines in and lets the detector learn to see through them.
3. Segment and classify symbols. With the lines gone, what remains is blobs of ink. Each blob, or group of blobs, is classified: notehead, clef, sharp, rest, and so on. Audiveris uses a small neural network trained on glyph shapes for this; older systems used template matching. Notalo skips the segmentation step and uses an object detector (YOLO) that proposes boxes and classes directly on the page image, trained on DeepScoresV2, a public dataset of about 250,000 annotated symbols on engraved scores.
4. Assemble and interpret. Noteheads are attached to stems, stems to beams and flags, dots to the note on their left. Then the music rules apply: a notehead's distance from the bottom staff line plus the clef gives a letter; the key signature and any accidental earlier in the bar adjust it. Audiveris does this with a graph of candidate interpretations and prunes the ones that contradict each other, so a bar that adds up to five beats in 4/4 time loses its least confident note. Notalo's rules are simpler and never delete a detection, which is why it occasionally shows a note that is not there, but never silently drops one.
5. Output. Audiveris writes MusicXML, which notation programs can open and edit. Notalo writes the letters back onto the original image and makes a MIDI file from the same notes.

Where the two approaches differ

The classic pipeline is excellent on clean engraved music and terrible on everything else, because every stage assumes the previous stage was perfect. A staff-removal step that leaves fragments of line through a notehead sends a mangled blob to the classifier, which calls it a rest, which breaks the bar count. On a photo with uneven lighting, the fragments are everywhere.

A learned detector is more forgiving. It has seen thousands of noteheads with lines through them and classifies them anyway, and it does not care much about a mild blur. Its weakness is the opposite one: it has no model of music. It will happily find a notehead where a dot of dirt is, because nothing tells it that a bar cannot have nine beats. The sensible design, which both camps are moving towards, is a learned front end with a rule-based back end that checks the result. Notalo's back end is still thin: it reads pitches, note heads, flags, beams and dots, and it does not yet read rests, ties or tuplets. That is why the letters are almost always right and the MIDI timing sometimes is not.

The numbers

We keep a test set of our own rendered scores, with every notehead's true pitch and length recorded, and run the detector on them in three versions: the clean rendering, a copy rotated by a few degrees, and a copy degraded to look like a phone photo (blur, noise, uneven light). On the clean pages the pitches are all correct and the note lengths are right 97% of the time. On the skewed pages the same, once the page is straightened. On the photo-like pages the pitches are still 95 to 99% right, but the note lengths drop to about 60%, almost entirely because the augmentation dot after a dotted note merges with a staff line and is lost. That one failure is the next thing we are fixing, and it is a good example of how OMR fails: not on the big symbols but on a three-pixel dot that changes a note's length by half.

What is hard for everyone

• Handwritten music. No two people draw a quarter note the same way. Engines trained on engraved music fall to 50 to 70% on handwriting. There are handwritten datasets (MUSCIMA++ is the best known) but far less data than for print.
• Dense piano music. Chords of four notes a second apart, cross-staff beams, three voices on one staff. The symbols touch and overlap, and the assembly step has to guess.
• Lyrics and text. Letters under the staff look like nothing in the symbol set and can confuse a detector that was not trained on them. Notalo masks the region below each staff before detecting, which is why lyrics do not become noteheads, and also why we cannot read them.
• Old editions. Nineteenth-century engraving has different notehead shapes, different clef designs and ink bleed. Everything degrades a little.

Which tool for which job

If you want an editable score from a clean PDF, Audiveris (free, open source) followed by a notation editor is the right tool, and the commercial scanners (PhotoScore, PlayScore, ScanScore) do the same with less fiddling. If you want to play the page today and the page is a photo, a learned detector that labels the image is more useful than a MusicXML file with errors you have to find in an editor. That is the gap Notalo is built for, and it is why we output letters on the picture rather than a file.

Written by Taekyung Lee, a self-taught pianist (four years, no lessons) who built Notalo for personal practice. About · Instagram

More: all guides and articles · demo songs with letters.