Hear Me Out

Interactive evaluation and bias discovery platform for speech-to-speech conversational AI

Shree Harsha Bokkahalli Satish, Gustav Eje Henter, Éva Székely

 

KTH Royal Institute of Technology, Stockholm, Sweden

🎙️ Click here to try Hear Me Out Live (Under construction for now! Reach out for a preview!)

Hear Me Out is an interactive evaluation and bias discovery platform for speech-to-speech conversational AI. Speech-to-speech models process spoken language directly from audio, without first converting it to text. They promise more natural, expressive, and emotionally aware interactions by retaining prosody, intonation, and other vocal cues throughout the conversation.

🏗️ Architecture

The backend is three services, set up and run entirely from this repo, behind self-signed SSL (browser mic capture requires HTTPS) and launched by infra/run_all.sh:

Each backend is an independent uv project under services/<name>/ with its own venv, so X-VC’s torch 2.5 / py3.10 never clashes with the others’ torch 2.4.

⚙️ Setup

infra/setup.sh is self-bootstrapping and interactive: it installs uv, clones the repo (with the seed-vc submodule) + MeanVC, uv syncs each service into its own venv, downloads all models, generates SSL, and wires up the workspace.

export HF_TOKEN=hf_xxxxx   # access to gated nvidia/personaplex-7b-v1
curl -fsSL https://raw.githubusercontent.com/shreeharsha-bs/Hear-Me-Out/main/infra/setup.sh -o setup.sh
bash setup.sh              # prompts for workspace, repo, token, etc.

• Workspace defaults to the current directory — cd into your target folder first, or set WORKSPACE=/path.
• Non-interactive (CI): pass -y with preset env, e.g. HF_TOKEN=… WORKSPACE=/workspace bash setup.sh -y.
• X-VC engine (optional, GPU): pass --xvc to also install it into its own venv; select it at run time with VC_ENGINE=xvc.

▶️ Running

cd <workspace> && bash Hear-Me-Out/infra/run_all.sh
# PersonaPlex :8000   app-api :5001   MeanVC :5002   (all SSL)

Set VC_ENGINE=meanvc|xvc to pick the voice-conversion engine on :5002. run_all.sh always serves the Vite build (frontend/dist, auto-built if missing).

🚀 Deploying a change

Edit locally, commit, push — then on the server:

cd <workspace>/Hear-Me-Out && git pull
bash infra/build-frontend.sh                 # only if the frontend changed
( cd services/<name> && uv sync )            # only if that service's deps changed
# re-run run_all.sh, or restart the affected service

Features

Hear Me Out enables users to experience interactions with conversational models in ways that aren’t typically accessible with regular benchmarking systems. Key features include:

• 🎤 Speech-to-Speech Models: Users can choose from a variety of models that retain vocal cues like prosody and intonation.
• 🔄 Real-Time Voice Conversion: Step into someone else’s voice – literally – and investigate how conversational AI systems interpret and respond to various speaker identities and expressions.
• ⚖️ Side-by-Side Comparisons: Ask a question with your own voice, then re-ask using a transformed voice. Compare the AI’s responses to observe differences in tone, phrasing, or behavior.
• 📊 Insights Through Data: Visualize metrics like speech rate, sentiment analysis, and more.

Through this immersive experience, we hope users will gain insights into identity, voice, and AI behavior. Ultimately, we aim to surface meaningful questions and inspire future research that promotes fairness and inclusivity with Hear Me Out.

🧪 Study platform

Beyond the interactive demo, the same backend runs a participant-study platform for controlled voice-conditioning experiments. Set APP_MODE=study and :5001 serves the study app (participant experiment + a token-gated admin dashboard) instead of the Chat/Convert/Metrics UI.

APP_MODE=study bash infra/build-frontend.sh
APP_MODE=study bash infra/run_all.sh

• Admin manages studies: scenarios with timed voice schedules (natural ↔ converted), target voices, and questionnaires; generates participants with counterbalanced, gender-conditional condition assignment; runs analysis and export.
• Participant flow (resumable, 1 hour): eligibility → consent → audio check → background → a practice scenario → counterbalanced analytical scenarios → questionnaires → converted-voice playback. The system prompt and voice schedule never reach the browser — the VC engine resolves them server-side.
• Voice conditions: stable-natural, stable-converted, VC-activation (natural → converted mid-call), and VC-deactivation (converted → natural).
• Analysis yields per-session technical-validity checks, a millisecond diarization timeline (overlaps, barge-ins), speech metrics, and objective VC-quality (WER, speaker similarity, UTMOS), all bundled by an export endpoint.

Demo Video

In the demo video, we explore the Moshi speech-to-speech model and its responses:

Example 1: Emotional Awareness

Notice how the model disambiguates between inputs with levity and frustration, correctly reflecting the speaker’s emotional state in its responses. This distinction adds a more human-like quality to the interaction.

Example 2: Voice Conversion - Gender Bias requesting unauthorized access

By applying voice transformations, we simulate how the model might respond to different speaker characteristics. While the differences in these responses are more subtle and inconsistent under repetition, hearing oneself in another voice opens up new perspectives.

Example 3: Voice Conversion - Gender Bias at Work