Sing Detect

A fast, native-feeling desktop app that finds singing segments in a stream recording and exports editor-friendly markers (DaVinci Resolve EDL / markers CSV / CSV / JSON). Built with Tauri 2 (Rust backend + TypeScript UI) and a small Python ML sidecar.

This is the detection-only spin-off of Stream Studio (the video-concat feature lives in its own project now).

How it works

 recording ──▶ [Rust ▸ ffmpeg ▸ 16 kHz mono WAV]
                                  │
              [Python sidecar ▸ inaSpeechSegmenter] ──json──▶ segments
                                  │
              [Rust ▸ EDL / markers / CSV / JSON exports]

Layer	Tech	Why
UI	WebView2 via Tauri 2	GPU-composited, tiny app, not Electron-sized.
Core	Rust	Audio extraction, drives the sidecar, writes exports. No console windows.
Audio	ffmpeg / ffprobe	16 kHz mono WAV extraction.
ML	Python 3.10 sidecar	inaSpeechSegmenter (TensorFlow) classifies singing as "music"; TensorFlow needs Python 3.10.

Prerequisites

• Node (18+) and Rust (rustup, MSVC toolchain) — to build.
• ffmpeg + ffprobe on PATH (or beside the exe) — bundled by pack.bat.
• Python 3.10 on the machine that runs detection. The app's built-in "Test detection engine" button installs the ML packages + downloads the model for you on first run (or run sidecar\setup_sidecar.bat manually).

Run / build

dev.bat        REM hot-reload dev (UI + Rust)
build.bat      REM release MSI installer → src-tauri\target\release\bundle\msi\
pack.bat       REM STANDALONE portable folder → dist-portable\SingDetect\

pack.bat produces a copy-paste folder with Sing Detect.exe, ffmpeg, and the Python sidecar/ bundled. Copy it to any Windows PC; the only thing the target needs is Python 3.10 (the in-app engine test handles the rest).

Detection auto-discovers py -3.10; override the interpreter with the STREAM_STUDIO_PYTHON env var, or the script path with STREAM_STUDIO_SIDECAR.

Project layout

sing_detect_v2/
├── index.html
├── src/                      # Frontend (TypeScript, no framework)
│   ├── main.ts               #   shell: single view, theme, drag-drop
│   ├── api.ts                #   typed bridge to Rust commands + events
│   ├── dom.ts / ui.ts        #   tiny DOM + toast/progress/song helpers
│   ├── styles.css            #   Fluent-ish design system (light/dark)
│   └── views/detect.ts       #   the detection view + engine setup card
├── src-tauri/
│   ├── src/
│   │   ├── lib.rs            #   commands, state, app wiring
│   │   ├── tools.rs          #   ffmpeg/ffprobe discovery + helpers
│   │   ├── detect.rs         #   drives the Python sidecar; env check/setup
│   │   └── export.rs         #   EDL / markers / CSV / JSON
│   ├── tauri.conf.json       #   window + bundles sidecar/ as a resource
│   └── capabilities/         #   dialog + opener permissions
└── sidecar/
    ├── detect.py             #   the ONLY Python — ML step only
    ├── requirements.txt
    └── setup_sidecar.bat

Detection parameters

Parameter	Default	When to change
Merge gap	10s	Raise if the singer pauses >10s mid-song
Min duration	20s	Lower for short covers/snippets
Padding	2s	Raise if songs get clipped at the edges
FPS	29.97	Match your video (25 PAL, 23.976 film) — affects timecode export only

Singing is classified as "music" by inaSpeechSegmenter; talking is "speech".