Open-Sourcing SONAR: Poseidon's Multilingual ASR Evaluation Toolkit

Product
Rashmi Nagpal

SONAR, our evaluation harness for multilingual automatic speech recognition (ASR), is available on GitHub at PSDN-AI/SONAR-OSS under the MIT License. You can install it, point it at your own models and audio, and get Word Error Rate (WER), Character Error Rate (CER), semantic similarity, and the composite Poseidon Score in a single run.

In our previous blog post, we explored why traditional voice AI evaluation falls apart beyond English, and why standard metrics like WER don’t tell the full story.

Benchmarks often rely on clean, scripted audio that doesn’t reflect real-world conditions, while a single metric like WER rigidly penalizes natural variation such as local dialects, valid synonyms, and code-switching without explaining why a model fails or where demographic performance gaps may be hiding.

Today, we’re sharing a quick, hands-on guide to SONAR (psdn-sonar), our open-source evaluation toolkit by Poseidon for multilingual ASR models.

SONAR is a configurable, recipe-driven evaluation harness that provides lexical accuracy, character precision, and semantic similarity into a composite score while pinpointing specific acoustic and demographic failure modes.

What's in the open-source ASR evaluation toolkit

The open-source release includes:

  • Core audio evaluation metrics — Word Error Rate (WER), Character Error Rate (CER), semantic similarity, and the composite Poseidon Score.
  • Language processors — script normalization for Bengali, Hindi, Korean, and English so models aren't penalized for surface variation.
  • Model registry & backends — ready-to-use HuggingFace ASR models plus hosted API backends (ElevenLabs, Whisper API, AssemblyAI), and bring-your-own via -hf-model.
  • Public dataset loaders — FLEURS, Common Voice, and OpenSLR, plus a generic data loader and a discover command that fetches and prepares public datasets for a language.
  • Evaluators — single-speaker and multi-speaker evaluation with per-utterance scoring.
  • Audio preprocessing — Voice Activity Detection (VAD) and diarization (pyannote) with an automatic preprocessing-method selector for noisy / multi-speaker audio.
  • Failure-mode analysis — audio-quality metrics and demographic analysis to surface where and for whom a model degrades.
  • Reproducible reporting — psdn-sonar single | multi | custom | discover and a recipe system for configurable, reproducible runs.

Not included:

  • Proprietary and customer/campaign datasets and their loaders.
  • Deployment & infrastructure — control-plane Docker images, ArgoCD/Kubernetes build-and-promote workflows, and cloud-storage (R2/ECR) clients.

How SONAR evaluates multilingual speech recognition

SONAR standardizes the pipeline from raw audio input to composite metric scoring:

Whether you need quick WER/CER checks or composite semantic evaluation, SONAR automates script normalization, transcription, and report generation.

Quickstart Guide: how to evaluate a speech recognition model

1. Installation

Install the core library along with the ML backends for semantic scoring and HuggingFace integration:

pip install -e ".[ml]"   # core + ML models/backends

Note: Core WER/CER work on the base install; semantic similarity and running HuggingFace models need the [ml] extra.

2. Basic Usage: Single Sample Scoring

Calculate classic metrics (WER/CER) or include the composite Poseidon score for semantic accuracy:

from psdn_sonar.utils.metrics import (
    calculate_cer_wer,
    calculate_poseidon_score,
    compute_semantic_similarity,
)

ref = "the quick brown fox"
hyp = "the quick brown box"

# Standard metrics
cer, wer = calculate_cer_wer(ref, hyp)
print(f"WER={wer:.2f}  CER={cer:.2f}")

# Composite Semantic Score
sim = compute_semantic_similarity(ref, hyp)
print(f"Semantic Score={sim:.3f}")

3. Evaluate Entire Datasets

Prepare a simple tab-separated (eval.tsv) dataset file:

audio_path	transcription
clips/0001.wav	the quick brown fox
clips/0002.wav	she sells sea shells

Run evaluation across model benchmarks in a single call:

from psdn_sonar.evaluators.single_speaker import SingleSpeakerEvaluator

SingleSpeakerEvaluator.run_evaluation(
    tsv_path="eval.tsv",
    models=["whisper_small_en"],
    language="en",
    output_dir="results/demo",
)

4. Discover Built-in Models

Explore registered models and default setups for target languages (e.g., Bengali, Hindi, Korean, English):

from psdn_sonar.models.registry import list_models, get_language_defaults

print(list_models())                 # every registered model id
print(get_language_defaults("bn"))   # default models for a language: bn/hi/ko/en

Contribute a language recipe for low-resource ASR

The contribution that helps most is a new language recipe: a language processor (script normalization, loanword/number handling) plus a recipe wiring it to models and datasets. SONAR was built recipe-first so that adding a language doesn't mean touching the evaluation core.

Other things we'd welcome:

  • Dataset loaders for additional public benchmarks.
  • Model backends — new HuggingFace models or hosted ASR APIs.
  • Better normalization rules — dialect variants, code-switching, synonyms, and loanwords for existing languages.
  • New metrics or analyses that expose acoustic or demographic failure modes.
  • Docs, examples, and bug reports.

Open an issue or a pull request at PSDN-AI/SONAR-OSS. For the methodology behind the metrics, read our Evaluating Voice AI Beyond English blog.

If you find SONAR useful for your speech workflows, give the repo a ⭐ on GitHub.


Acknowledgements: The authors would like to thank Henri Viès, Krista Gambrel, and Lauren Lankford for their contributions to this release.