v7: standalone Sing Detect app — full Tauri project tree
Promotes the v6 detection slice into a complete, buildable Tauri 2 app and renames Stream Studio → Sing Detect (detection-only; video-concat split to its own project). Proper src/ + src-tauri/ layout, Python sidecar under sidecar/, app icons, package/Cargo manifests, and dev/build/pack scripts. Pipeline unchanged in spirit: Rust drives ffmpeg to a 16 kHz mono WAV, the Python sidecar (inaSpeechSegmenter) emits line-JSON segments, and Rust writes EDL / markers CSV / CSV / JSON exports. Long ops run on spawn_blocking; cancel kills the whole py -3.10 process tree. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
106
README.md
106
README.md
@@ -1,41 +1,85 @@
|
||||
# sing_detect
|
||||
# Sing Detect
|
||||
|
||||
Singing/music-segment detection for long live-stream recordings: find the
|
||||
segments where someone is singing and export markers (CSV / EDL / JSON) for fast
|
||||
editing in DaVinci Resolve.
|
||||
A fast, native-feeling desktop app that **finds singing segments in a stream
|
||||
recording** and exports editor-friendly markers (DaVinci Resolve EDL / markers
|
||||
CSV / CSV / JSON). Built with Tauri 2 (Rust backend + TypeScript UI) and a small
|
||||
Python ML sidecar.
|
||||
|
||||
**This branch is the latest version (v6).** A plain `git pull` gives you only the
|
||||
current code — no old versions clutter the working tree. The earlier iterations
|
||||
live in the git history and are tagged, so you can inspect or check out any of
|
||||
them:
|
||||
This is the detection-only spin-off of Stream Studio (the video-concat feature
|
||||
lives in its own project now).
|
||||
|
||||
```bash
|
||||
git checkout v3 # or v1 … v6
|
||||
git checkout - # back to latest
|
||||
## How it works
|
||||
|
||||
```
|
||||
recording ──▶ [Rust ▸ ffmpeg ▸ 16 kHz mono WAV]
|
||||
│
|
||||
[Python sidecar ▸ inaSpeechSegmenter] ──json──▶ segments
|
||||
│
|
||||
[Rust ▸ EDL / markers / CSV / JSON exports]
|
||||
```
|
||||
|
||||
## Evolution
|
||||
| Layer | Tech | Why |
|
||||
|-------|------|-----|
|
||||
| UI | **WebView2** via **Tauri 2** | GPU-composited, tiny app, not Electron-sized. |
|
||||
| Core | **Rust** | Audio extraction, drives the sidecar, writes exports. No console windows. |
|
||||
| Audio | **ffmpeg / ffprobe** | 16 kHz mono WAV extraction. |
|
||||
| ML | **Python 3.10 sidecar** | `inaSpeechSegmenter` (TensorFlow) classifies singing as "music"; TensorFlow needs Python 3.10. |
|
||||
|
||||
The history is ordered by **technical & functional evolution** (not file
|
||||
timestamp). Each tag is a complete, standalone version at the repository root.
|
||||
## Prerequisites
|
||||
|
||||
| Tag | Engine | Interface | Leap |
|
||||
|-----|--------|-----------|------|
|
||||
| `v1` | HuggingFace `matthijs/svd` (torch.jit) | tkinter file picker | First prototype — single file, proves the idea |
|
||||
| `v2` | `transformers` pipeline + pyannote | CustomTkinter GUI | Overlapping-chunk inference, CSV timecode export |
|
||||
| `v3` | inaSpeechSegmenter | CLI (argparse) | Robust pipeline: gap-merge, min-duration, EDL+CSV |
|
||||
| `v4` | inaSpeechSegmenter | CLI batch | Process whole folders, `--recursive`, `--auto-cut` |
|
||||
| `v5` | inaSpeechSegmenter | CustomTkinter multi-tab app | Modular `core/` engine + UI tab, JSON + ffmpeg-script export |
|
||||
| `v6` | inaSpeechSegmenter (Python sidecar) | Tauri (Rust + TS) | Cross-platform rewrite; Python reduced to a thin ML sidecar emitting line-JSON |
|
||||
- **Node** (18+) and **Rust** (`rustup`, MSVC toolchain) — to build.
|
||||
- **ffmpeg + ffprobe** on PATH (or beside the exe) — bundled by `pack.bat`.
|
||||
- **Python 3.10** on the machine that runs detection. The app's built-in
|
||||
**"Test detection engine"** button installs the ML packages + downloads the
|
||||
model for you on first run (or run `sidecar\setup_sidecar.bat` manually).
|
||||
|
||||
Engine evolution: `svd model` → `transformers pipeline` →
|
||||
**`inaSpeechSegmenter`** (settled from v3 on, which classifies singing voice as
|
||||
"music").
|
||||
## Run / build
|
||||
|
||||
## Notes
|
||||
- **v4** carries a copy of `detect_singing.py` because `batch_detect.py` imports it at runtime.
|
||||
- **v5** keeps the original package layout (`core/`, `ui/`, `utils/`) so the relative imports resolve; it was extracted from the larger `stream_tools` app.
|
||||
- **v6** is the detection slice only (`sidecar_detect.py`, `detect.rs`, `detect.ts`), extracted from the `stream-studio` Tauri app — `detect.ts` still references the app's `../dom`, `../ui`, `../api` modules.
|
||||
```bat
|
||||
dev.bat REM hot-reload dev (UI + Rust)
|
||||
build.bat REM release MSI installer → src-tauri\target\release\bundle\msi\
|
||||
pack.bat REM STANDALONE portable folder → dist-portable\SingDetect\
|
||||
```
|
||||
|
||||
Excluded: `song_detection_cn.py` (an ACRCloud-style fingerprint *song-identification*
|
||||
client) — it names a track rather than detecting singing segments.
|
||||
`pack.bat` produces a copy-paste folder with `Sing Detect.exe`, ffmpeg, and the
|
||||
Python `sidecar/` bundled. Copy it to any Windows PC; the only thing the target
|
||||
needs is Python 3.10 (the in-app engine test handles the rest).
|
||||
|
||||
Detection auto-discovers `py -3.10`; override the interpreter with the
|
||||
`STREAM_STUDIO_PYTHON` env var, or the script path with `STREAM_STUDIO_SIDECAR`.
|
||||
|
||||
## Project layout
|
||||
|
||||
```
|
||||
sing_detect_v2/
|
||||
├── index.html
|
||||
├── src/ # Frontend (TypeScript, no framework)
|
||||
│ ├── main.ts # shell: single view, theme, drag-drop
|
||||
│ ├── api.ts # typed bridge to Rust commands + events
|
||||
│ ├── dom.ts / ui.ts # tiny DOM + toast/progress/song helpers
|
||||
│ ├── styles.css # Fluent-ish design system (light/dark)
|
||||
│ └── views/detect.ts # the detection view + engine setup card
|
||||
├── src-tauri/
|
||||
│ ├── src/
|
||||
│ │ ├── lib.rs # commands, state, app wiring
|
||||
│ │ ├── tools.rs # ffmpeg/ffprobe discovery + helpers
|
||||
│ │ ├── detect.rs # drives the Python sidecar; env check/setup
|
||||
│ │ └── export.rs # EDL / markers / CSV / JSON
|
||||
│ ├── tauri.conf.json # window + bundles sidecar/ as a resource
|
||||
│ └── capabilities/ # dialog + opener permissions
|
||||
└── sidecar/
|
||||
├── detect.py # the ONLY Python — ML step only
|
||||
├── requirements.txt
|
||||
└── setup_sidecar.bat
|
||||
```
|
||||
|
||||
## Detection parameters
|
||||
|
||||
| Parameter | Default | When to change |
|
||||
|-----------|---------|----------------|
|
||||
| Merge gap | 10s | Raise if the singer pauses >10s mid-song |
|
||||
| Min duration | 20s | Lower for short covers/snippets |
|
||||
| Padding | 2s | Raise if songs get clipped at the edges |
|
||||
| FPS | 29.97 | Match your video (25 PAL, 23.976 film) — affects timecode export only |
|
||||
|
||||
Singing is classified as **"music"** by inaSpeechSegmenter; talking is "speech".
|
||||
|
||||
Reference in New Issue
Block a user