sing_detect/v3_inaseg_cli/README.md

# Singing Segment Detector

Automatically detect singing segments in live stream recordings and export timeline markers for DaVinci Resolve.

## What It Does

```
4hr stream recording (.mp4/.mkv/.flv)
    │
    ▼  [ffmpeg: extract 16kHz mono audio]
    │
    ▼  [inaSpeechSegmenter: classify speech/music/noise]
    │
    ▼  [merge nearby music segments, filter short ones]
    │
    ├──→ .edl        (import as DaVinci Resolve timeline)
    ├──→ _markers.csv (import as DaVinci Resolve markers)
    ├──→ .csv         (human-readable segment list)
    ├──→ .json        (programmatic use)
    └──→ .bat         (optional: auto-cut with ffmpeg, no editing needed)
```

**Key insight**: `inaSpeechSegmenter` classifies **singing voice as "music"**. So in a typical singing stream, talking = "speech", singing = "music". We filter for "music" segments and merge nearby ones (a song might have brief pauses between verses).

## Setup (Windows with conda)

### Prerequisites
- **conda** (Anaconda or Miniconda)
- **ffmpeg** in your PATH
- **NVIDIA GPU** recommended (RTX 2080 works great) but CPU also works

### Step 1: Create conda environment

```powershell
conda create -n singing-detector python=3.11 -y
conda activate singing-detector
```

### Step 2: Install PyTorch with CUDA (for GPU acceleration)

```powershell
# For RTX 2080 (CUDA 12.x)
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
```

### Step 3: Install TensorFlow (required by inaSpeechSegmenter)

```powershell
pip install tensorflow[and-cuda]
```

Or CPU-only (slower but works):

```powershell
pip install tensorflow
```

### Step 4: Install inaSpeechSegmenter and dependencies

```powershell
pip install inaSpeechSegmenter
```

### Step 5: Verify installation

```powershell
python -c "from inaSpeechSegmenter import Segmenter; print('OK!')"
```

## Usage

### Single file

```powershell
conda activate singing-detector

# Basic usage
python detect_singing.py "D:\streams\2026-04-04_stream.mp4"

# Custom settings
python detect_singing.py "D:\streams\2026-04-04_stream.mp4" --gap 15 --min-duration 30

# Output to a specific directory
python detect_singing.py "D:\streams\2026-04-04_stream.mp4" -o "D:\singing_edits"

# Also generate an auto-cut ffmpeg script (no DaVinci needed)
python detect_singing.py "D:\streams\2026-04-04_stream.mp4" --auto-cut

# For 4K F-log footage at 29.97fps (your XT3 settings)
python detect_singing.py "D:\streams\2026-04-04_stream.mp4" --fps 29.97
```

### Batch processing

```powershell
# Process all videos in a folder
python batch_detect.py "D:\streams"

# Process new files only (skip already-done ones)
python batch_detect.py "D:\streams" --skip-existing

# Process and auto-generate singing-only videos
python batch_detect.py "D:\streams" --auto-cut -o "D:\singing_edits"

# Only process .flv files
python batch_detect.py "D:\streams" --pattern "*.flv"
```

## Importing Results into DaVinci Resolve

### Option A: Import EDL as Timeline (recommended)

1. Open your project in DaVinci Resolve
2. Import the original stream video into your Media Pool
3. Go to **File → Import → Timeline**
4. Select the `_singing.edl` file
5. DaVinci will create a new timeline with only the singing segments

### Option B: Import Markers CSV

1. Create a timeline from your stream video
2. Right-click on the timeline → **Timelines → Import Markers from CSV**
3. Select the `_markers.csv` file
4. Blue markers will appear at each singing segment start/end

### Option C: Auto-Cut (no DaVinci needed)

If you used `--auto-cut`, just run the generated `.bat` script:

```powershell
D:\singing_edits\2026-04-04_stream_singing.bat
```

This uses ffmpeg to directly cut and concatenate all singing segments into a single `_singing_only.mp4` file. Fastest option, but no manual review.

## Tuning Parameters

| Parameter | Default | Description |
|-----------|---------|-------------|
| `--gap` | 10s | Max gap between music segments to merge. Increase if singer pauses >10s between verses. |
| `--min-duration` | 20s | Minimum segment length. Lower if singer does very short songs. |
| `--padding` | 2s | Extra seconds before/after each segment. Helps catch song intro/outro. |
| `--fps` | 29.97 | Frame rate for timecode. Use 25 for PAL. |

### Recommended settings for typical singing streams

- **Singer with lots of chatting between songs**: `--gap 10 --min-duration 30`
- **Singer with minimal breaks**: `--gap 5 --min-duration 20`
- **Singer who does short covers/snippets**: `--gap 8 --min-duration 15`
- **Conservative (catch everything)**: `--gap 20 --min-duration 15 --padding 5`

## Troubleshooting

### "Singing voice detected as speech"
This can happen if the singer talks over background music. Try `--gap 15` to merge nearby segments.

### "Too many false positives (non-singing music detected)"
If there's background music during chatting, increase `--min-duration 45` to only keep longer segments (full songs).

### "Segments cut off the beginning/end of songs"
Increase `--padding 5` to add more buffer.

### GPU not being used
Check that TensorFlow detects your GPU:
```python
python -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"
```

### Processing is very slow
- A 4-hour stream typically takes 5-15 minutes on GPU, 30-60 minutes on CPU.
- Make sure you're using GPU-enabled TensorFlow.
- Close other GPU-heavy applications during processing.

## Output Files

For input `stream_2026-04-04.mp4`, you get:

| File | Purpose |
|------|---------|
| `stream_2026-04-04_singing.edl` | DaVinci Resolve timeline import |
| `stream_2026-04-04_markers.csv` | DaVinci Resolve marker import |
| `stream_2026-04-04_singing.csv` | Human-readable segment list |
| `stream_2026-04-04_singing.json` | Programmatic use |
| `stream_2026-04-04_singing.bat` | Auto-cut script (with `--auto-cut`) |
| `stream_2026-04-04_raw_segments.csv` | All segments for debugging (with `--raw-segments`) |