KTV/MTV video synthesis service using FFmpeg. Creates karaoke videos with dual audio tracks (accompaniment + original) and synchronized ASS subtitles.

Overview

This service processes video clips, audio tracks, and subtitles to produce:

MTV (Music Television): Single audio track with original vocals and subtitles
KTV (Karaoke Television): Dual audio tracks - accompaniment (default) and original vocals

Architecture

Framework: ahserver + longtasks + Redis
Port: 9084
Queue: ktv_synth
Worker: FFmpeg subprocess (no GPU required)

Features

Two-step FFmpeg synthesis pipeline
ASS subtitle rendering with karaoke effects
Dual audio track support with proper metadata
Configurable video looping for scene clips
1920x1080 output resolution with Lanczos scaling
Automatic duration calculation

Installation

Prerequisites

Python 3.8+
FFmpeg with libx264 and AAC support
Redis server

Setup

# Clone repository
cd /data/ymq/ktv-synth-service

# Ensure FFmpeg is installed
ffmpeg -version

# Ensure Redis is running
redis-cli ping

Usage

Starting the Service

./start.sh

The service will start on port 9084 and begin processing tasks from the Redis queue.

Stopping the Service

./stop.sh

Health Check

Visit http://localhost:9084/app/health.dspy or check the service status.

API

Task Payload

Submit tasks to the Redis queue ktv_synth:

{
    "task_type": "synthesize",
    "video_files": [
        "/path/to/scene1.mp4",
        "/path/to/scene2.mp4",
        "/path/to/scene3.mp4"
    ],
    "original_audio": "/path/to/original.wav",
    "accompaniment": "/path/to/no_vocals.wav",
    "subtitle_path": "/path/to/subtitles.ass",
    "output_dir": "/tmp/ktv-synth-outputs",
    "title": "SongName",
    "duration": 240.5,
    "loops": 3,
    "output_modes": ["mtv", "ktv"]
}

Parameters

video_files (required): List of video file paths (scene clips to loop)
original_audio (required): Path to original full audio with vocals
accompaniment (required for KTV): Path to accompaniment track (no vocals)
subtitle_path (required): Path to ASS subtitle file
output_dir (optional): Output directory (default: /tmp/ktv-synth-outputs)
title (optional): Song title for output naming (default: output)
duration (optional): Target duration in seconds (auto-calculated if not provided)
loops (optional): Number of video loops (auto-calculated if not provided)
output_modes (optional): List of outputs to generate: ["mtv"], ["ktv"], or ["mtv", "ktv"]

Response

{
    "mtv_path": "/tmp/ktv-synth-outputs/SongName_MTV.mp4",
    "ktv_path": "/tmp/ktv-synth-outputs/SongName_KTV.mp4",
    "mtv_size_mb": 125.45,
    "ktv_size_mb": 145.67,
    "duration": 240.5
}

Technical Details

Two-Step Synthesis Process

Step 1: Create Silent Looped Video Track

Concatenates and loops scene clips to match target duration:

ffmpeg -y -f concat -safe 0 -stream_loop {loops} -i {concat_list} \
  -t {duration} -an -c:v libx264 -preset fast -crf 23 {temp_video}

Step 2a: MTV Synthesis (Single Track)

Combines video with original audio and ASS subtitles:

ffmpeg -y -i {temp_video} -i {original_audio} \
  -map 0:v -map 1:a \
  -vf "ass={subtitle_path},scale=1920:1080:flags=lanczos" \
  -c:v libx264 -preset fast -crf 23 \
  -c:a aac -b:a 192k \
  {mtv_output}

Step 2b: KTV Synthesis (Dual Track)

Creates dual audio tracks with accompaniment as default:

ffmpeg -y -i {temp_video} -i {accompaniment} -i {original_audio} \
  -map 0:v -map 1:a -map 2:a \
  -vf "ass={subtitle_path},scale=1920:1080:flags=lanczos" \
  -c:v libx264 -preset fast -crf 23 \
  -c:a:0 aac -b:a:0 192k -metadata:s:a:0 handler_name="伴奏(Accompaniment)" \
  -c:a:1 aac -b:a:1 192k -metadata:s:a:1 handler_name="原唱(Original)" \
  -disposition:a:0 default -disposition:a:1 0 \
  {ktv_output}

Video Encoding Settings

Codec: H.264 (libx264)
Preset: fast
CRF: 23 (balanced quality/size)
Resolution: 1920x1080
Scaling: Lanczos (high quality)

Audio Encoding Settings

Codec: AAC
Bitrate: 192 kbps
Tracks: 1 (MTV) or 2 (KTV)

KTV Audio Track Metadata

Track 0: Accompaniment (default playback)
- Handler: "伴奏(Accompaniment)"
- Disposition: default
Track 1: Original with vocals
- Handler: "原唱(Original)"
- Disposition: 0 (not default)

Configuration

Edit conf/config.json:

{
    "port": 9084,
    "queue": "ktv_synth",
    "filesroot": "/tmp/ktv-synth-outputs",
    "redis_url": "redis://127.0.0.1:6379",
    "worker_cnt": 1,
    "stuck_seconds": 1800,
    "max_age_hours": 24
}

Troubleshooting

FFmpeg Errors

Check FFmpeg installation and codec support:

ffmpeg -codecs | grep libx264
ffmpeg -codecs | grep aac

Redis Connection

Verify Redis is running:

redis-cli ping

Permission Issues

Ensure the service has write access to output directories:

chmod 755 /tmp/ktv-synth-outputs

High Memory Usage

Reduce worker count in conf/config.json:

{
    "worker_cnt": 1
}

Performance

MTV Generation: ~2-3x real-time (240s video in ~80-120s)
KTV Generation: ~2-3x real-time
Concurrent Tasks: Limited by worker_cnt (default: 1)
Memory: ~500MB-1GB per worker (depends on video resolution)

Integration

This service integrates with:

demucs-service: Audio source separation (provides accompaniment tracks)
whisper-service: Subtitle generation (provides ASS files)
wan22-service: Video generation (provides scene clips)

License

Internal use only.

Support

For issues or questions, contact the development team.