MOSS-Music

Music is never just about “hearing the lyrics” or “recognizing sound events.” At the surface of a song are vocals, instruments, melody, and production texture; beneath them lie chords, harmonic progression, key, tempo, beat, section structure, and emotional tension. General-purpose audio models have often been better at recognizing “what sounds occurred.” MOSS-Music instead asks how those sounds form a song: What key is it in? Where does the chorus begin? How do the lyrics, vocals, instrumentation, and chords relate to one another?

MOSS-Music is jointly open-sourced by MOSI.AI, the OpenMOSS team, and the Shanghai Innovation Institute. It follows the modular architecture of MOSS-Audio, comprising a dedicated audio encoder, a modality adapter, and a large language model. Raw audio is first encoded into a continuous temporal representation at 12.5 Hz, then projected into the language model's embedding space. The model is then continually pretrained and instruction fine-tuned on large-scale music data, with particular emphasis on singing, lyrics, full-length songs, harmony, structure, and long-form understanding. This release includes two 8B models: MOSS-Music-8B-Instruct follows direct instructions for music captioning, Q&A, lyrics ASR, and automatic tagging; MOSS-Music-8B-Thinking supports explicit reasoning for tasks requiring extended analysis of harmony, structure, style, and long-form music.

Lyrics ASR with Timestamps

Recognizes sung Chinese and English lyrics despite accompaniment and supports sentence-level and word-level timestamps. On three singing benchmarks, MUSDB18, MIR-1K, and Opencpop, the Thinking variant achieves an average WER / CER of 15.88%.

Music Captioning and Automatic Tagging

Generates natural-language descriptions of style, mood, tempo, instrumentation, vocals, melody, harmony, structure, production texture, and use cases. It can also produce concise tags on request. Across nine dimensions scored by GPT-5.4, MOSS-Music leads on both MusicCaps and SDD, with especially strong gains in describing structure and song sections.

Harmony, Key, and Rhythm Analysis

Identifies key, beat, downbeats, BPM, and chord progressions. It supports chord transcription and timestamped chord transcription for harmonic analysis, accompaniment reference, and music education.

Structural Segmentation

Segments full-length songs into sections such as intro, verse, chorus, bridge, and outro, and provides section-boundary timestamps to capture long-range musical organization.

Instrument and Vocal Part Recognition

Identifies lead instruments, vocal types, solo or choral singing, gender, and vocal range.

Long-Form Music Q&A and Reasoning

Answers open-ended questions about complete works, with retrospective analysis of sections, lyrics, structure, emotional changes, and musical elements. The Thinking variant presents its analysis before giving a final answer.

Time-Aware Model Design

During pretraining, temporal tokens are inserted into audio-frame representations at fixed intervals, allowing the model to learn “what happened when” in a unified text-generation framework. DeepStack-style cross-layer feature injection separately projects features from early, intermediate, and final encoder layers into the first few language-model layers, preserving low-level details such as rhythmic transients and timbre. This supports timestamped lyrics ASR, beat and downbeat localization, and section-boundary detection without an additional alignment module.

Open-Source Data Pipeline

MOSS-Music-Data-Pipeline is also open-sourced. It covers the full workflow from raw audio to chat-formatted training samples: duration statistics, full-song MIR analysis and structure annotation, structure-based segmentation, segment-level lyrics ASR and key/chord analysis, metadata cleaning, and caption/query generation. JSONL sharding supports large-scale corpus construction on HPC, Kubernetes, or Ray clusters.

Music Q&A and Understanding (Accuracy ↑)

Avg is the mean across eight public music Q&A benchmarks: MMAU-music, MMAU-mini-music, MMAU-Pro-music, MMAR-music, MuChoMusic, Music-AVQA, GTZAN, and Medley-Solos-DB. The three NSynth single-note recognition tasks are listed separately and excluded from the average.

Music Captioning (GPT-5.4 Judge, 1–5)

Scores are averaged across nine dimensions: style, mood, tempo, instrumentation, vocals, melody/harmony, structure, production, and context of use.

Lyrics ASR (WER / CER ↓)

MUSDB18 contains English pop songs with accompaniment (WER), MIR-1K contains Chinese karaoke clips with accompaniment (CER), and Opencpop contains clean Mandarin studio vocals (CER).

Chord Transcription

Chord transcription and timestamped chord transcription are supported. Detailed benchmark results will be published in a future update.