youfu
4887981a2d
Promotes the v6 detection slice into a complete, buildable Tauri 2 app and renames Stream Studio → Sing Detect (detection-only; video-concat split to its own project). Proper src/ + src-tauri/ layout, Python sidecar under sidecar/, app icons, package/Cargo manifests, and dev/build/pack scripts. Pipeline unchanged in spirit: Rust drives ffmpeg to a 16 kHz mono WAV, the Python sidecar (inaSpeechSegmenter) emits line-JSON segments, and Rust writes EDL / markers CSV / CSV / JSON exports. Long ops run on spawn_blocking; cancel kills the whole py -3.10 process tree. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
3.7 KiB
Sing Detect
A fast, native-feeling desktop app that finds singing segments in a stream recording and exports editor-friendly markers (DaVinci Resolve EDL / markers CSV / CSV / JSON). Built with Tauri 2 (Rust backend + TypeScript UI) and a small Python ML sidecar.
This is the detection-only spin-off of Stream Studio (the video-concat feature lives in its own project now).
How it works
recording ──▶ [Rust ▸ ffmpeg ▸ 16 kHz mono WAV]
│
[Python sidecar ▸ inaSpeechSegmenter] ──json──▶ segments
│
[Rust ▸ EDL / markers / CSV / JSON exports]
| Layer | Tech | Why |
|---|---|---|
| UI | WebView2 via Tauri 2 | GPU-composited, tiny app, not Electron-sized. |
| Core | Rust | Audio extraction, drives the sidecar, writes exports. No console windows. |
| Audio | ffmpeg / ffprobe | 16 kHz mono WAV extraction. |
| ML | Python 3.10 sidecar | inaSpeechSegmenter (TensorFlow) classifies singing as "music"; TensorFlow needs Python 3.10. |
Prerequisites
- Node (18+) and Rust (
rustup, MSVC toolchain) — to build. - ffmpeg + ffprobe on PATH (or beside the exe) — bundled by
pack.bat. - Python 3.10 on the machine that runs detection. The app's built-in
"Test detection engine" button installs the ML packages + downloads the
model for you on first run (or run
sidecar\setup_sidecar.batmanually).
Run / build
dev.bat REM hot-reload dev (UI + Rust)
build.bat REM release MSI installer → src-tauri\target\release\bundle\msi\
pack.bat REM STANDALONE portable folder → dist-portable\SingDetect\
pack.bat produces a copy-paste folder with Sing Detect.exe, ffmpeg, and the
Python sidecar/ bundled. Copy it to any Windows PC; the only thing the target
needs is Python 3.10 (the in-app engine test handles the rest).
Detection auto-discovers py -3.10; override the interpreter with the
STREAM_STUDIO_PYTHON env var, or the script path with STREAM_STUDIO_SIDECAR.
Project layout
sing_detect_v2/
├── index.html
├── src/ # Frontend (TypeScript, no framework)
│ ├── main.ts # shell: single view, theme, drag-drop
│ ├── api.ts # typed bridge to Rust commands + events
│ ├── dom.ts / ui.ts # tiny DOM + toast/progress/song helpers
│ ├── styles.css # Fluent-ish design system (light/dark)
│ └── views/detect.ts # the detection view + engine setup card
├── src-tauri/
│ ├── src/
│ │ ├── lib.rs # commands, state, app wiring
│ │ ├── tools.rs # ffmpeg/ffprobe discovery + helpers
│ │ ├── detect.rs # drives the Python sidecar; env check/setup
│ │ └── export.rs # EDL / markers / CSV / JSON
│ ├── tauri.conf.json # window + bundles sidecar/ as a resource
│ └── capabilities/ # dialog + opener permissions
└── sidecar/
├── detect.py # the ONLY Python — ML step only
├── requirements.txt
└── setup_sidecar.bat
Detection parameters
| Parameter | Default | When to change |
|---|---|---|
| Merge gap | 10s | Raise if the singer pauses >10s mid-song |
| Min duration | 20s | Lower for short covers/snippets |
| Padding | 2s | Raise if songs get clipped at the edges |
| FPS | 29.97 | Match your video (25 PAL, 23.976 film) — affects timecode export only |
Singing is classified as "music" by inaSpeechSegmenter; talking is "speech".