youfu
4887981a2d
Promotes the v6 detection slice into a complete, buildable Tauri 2 app and renames Stream Studio → Sing Detect (detection-only; video-concat split to its own project). Proper src/ + src-tauri/ layout, Python sidecar under sidecar/, app icons, package/Cargo manifests, and dev/build/pack scripts. Pipeline unchanged in spirit: Rust drives ffmpeg to a 16 kHz mono WAV, the Python sidecar (inaSpeechSegmenter) emits line-JSON segments, and Rust writes EDL / markers CSV / CSV / JSON exports. Long ops run on spawn_blocking; cancel kills the whole py -3.10 process tree. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
86 lines
3.7 KiB
Markdown
86 lines
3.7 KiB
Markdown
# Sing Detect
|
|
|
|
A fast, native-feeling desktop app that **finds singing segments in a stream
|
|
recording** and exports editor-friendly markers (DaVinci Resolve EDL / markers
|
|
CSV / CSV / JSON). Built with Tauri 2 (Rust backend + TypeScript UI) and a small
|
|
Python ML sidecar.
|
|
|
|
This is the detection-only spin-off of Stream Studio (the video-concat feature
|
|
lives in its own project now).
|
|
|
|
## How it works
|
|
|
|
```
|
|
recording ──▶ [Rust ▸ ffmpeg ▸ 16 kHz mono WAV]
|
|
│
|
|
[Python sidecar ▸ inaSpeechSegmenter] ──json──▶ segments
|
|
│
|
|
[Rust ▸ EDL / markers / CSV / JSON exports]
|
|
```
|
|
|
|
| Layer | Tech | Why |
|
|
|-------|------|-----|
|
|
| UI | **WebView2** via **Tauri 2** | GPU-composited, tiny app, not Electron-sized. |
|
|
| Core | **Rust** | Audio extraction, drives the sidecar, writes exports. No console windows. |
|
|
| Audio | **ffmpeg / ffprobe** | 16 kHz mono WAV extraction. |
|
|
| ML | **Python 3.10 sidecar** | `inaSpeechSegmenter` (TensorFlow) classifies singing as "music"; TensorFlow needs Python 3.10. |
|
|
|
|
## Prerequisites
|
|
|
|
- **Node** (18+) and **Rust** (`rustup`, MSVC toolchain) — to build.
|
|
- **ffmpeg + ffprobe** on PATH (or beside the exe) — bundled by `pack.bat`.
|
|
- **Python 3.10** on the machine that runs detection. The app's built-in
|
|
**"Test detection engine"** button installs the ML packages + downloads the
|
|
model for you on first run (or run `sidecar\setup_sidecar.bat` manually).
|
|
|
|
## Run / build
|
|
|
|
```bat
|
|
dev.bat REM hot-reload dev (UI + Rust)
|
|
build.bat REM release MSI installer → src-tauri\target\release\bundle\msi\
|
|
pack.bat REM STANDALONE portable folder → dist-portable\SingDetect\
|
|
```
|
|
|
|
`pack.bat` produces a copy-paste folder with `Sing Detect.exe`, ffmpeg, and the
|
|
Python `sidecar/` bundled. Copy it to any Windows PC; the only thing the target
|
|
needs is Python 3.10 (the in-app engine test handles the rest).
|
|
|
|
Detection auto-discovers `py -3.10`; override the interpreter with the
|
|
`STREAM_STUDIO_PYTHON` env var, or the script path with `STREAM_STUDIO_SIDECAR`.
|
|
|
|
## Project layout
|
|
|
|
```
|
|
sing_detect_v2/
|
|
├── index.html
|
|
├── src/ # Frontend (TypeScript, no framework)
|
|
│ ├── main.ts # shell: single view, theme, drag-drop
|
|
│ ├── api.ts # typed bridge to Rust commands + events
|
|
│ ├── dom.ts / ui.ts # tiny DOM + toast/progress/song helpers
|
|
│ ├── styles.css # Fluent-ish design system (light/dark)
|
|
│ └── views/detect.ts # the detection view + engine setup card
|
|
├── src-tauri/
|
|
│ ├── src/
|
|
│ │ ├── lib.rs # commands, state, app wiring
|
|
│ │ ├── tools.rs # ffmpeg/ffprobe discovery + helpers
|
|
│ │ ├── detect.rs # drives the Python sidecar; env check/setup
|
|
│ │ └── export.rs # EDL / markers / CSV / JSON
|
|
│ ├── tauri.conf.json # window + bundles sidecar/ as a resource
|
|
│ └── capabilities/ # dialog + opener permissions
|
|
└── sidecar/
|
|
├── detect.py # the ONLY Python — ML step only
|
|
├── requirements.txt
|
|
└── setup_sidecar.bat
|
|
```
|
|
|
|
## Detection parameters
|
|
|
|
| Parameter | Default | When to change |
|
|
|-----------|---------|----------------|
|
|
| Merge gap | 10s | Raise if the singer pauses >10s mid-song |
|
|
| Min duration | 20s | Lower for short covers/snippets |
|
|
| Padding | 2s | Raise if songs get clipped at the edges |
|
|
| FPS | 29.97 | Match your video (25 PAL, 23.976 film) — affects timecode export only |
|
|
|
|
Singing is classified as **"music"** by inaSpeechSegmenter; talking is "speech".
|