FFmpeg NVENC transcoding setup — hardware video encoding on Ubuntu

Software x264 transcoding melts CPUs — a single 1080p H.264 stream at medium preset eats 4-6 cores. NVENC, the hardware encoder built into NVIDIA GPUs, does the same job in a fraction of the power and frees your CPUs for ingest and packaging. This guide sets up FFmpeg with NVENC on Ubuntu for live IPTV/streaming transcoding.

We target a GPU server with an RTX PRO 4500 or 6000 Blackwell — see our video transcoding server page for the hardware.

Prerequisites

• Ubuntu 22.04 with an NVIDIA GPU (RTX PRO, RTX 4090, L4, L40S)
• Root access
• Basic familiarity with FFmpeg command-line

Step 1 — Install the NVIDIA driver

On a Hostfory GPU box the driver is usually pre-installed. Verify:

nvidia-smi

If it's missing:

apt update
apt install -y nvidia-driver-550 nvidia-utils-550
reboot

After reboot, nvidia-smi should show your GPU, driver version and CUDA version.

Step 2 — Install FFmpeg with NVENC support

The Ubuntu-repo FFmpeg often lacks NVENC. Install a build that includes it, or compile. The quickest path is the static build from BtbN:

cd /opt
wget https://github.com/BtbN/FFmpeg-Builds/releases/download/latest/ffmpeg-master-latest-linux64-gpl.tar.xz
tar xf ffmpeg-master-latest-linux64-gpl.tar.xz
mv ffmpeg-master-latest-linux64-gpl ffmpeg
ln -sf /opt/ffmpeg/bin/ffmpeg /usr/local/bin/ffmpeg
ln -sf /opt/ffmpeg/bin/ffprobe /usr/local/bin/ffprobe

Verify NVENC is available:

ffmpeg -hide_banner -encoders | grep nvenc

You should see h264_nvenc, hevc_nvenc, and on Blackwell/Ada, av1_nvenc.

Step 3 — A basic live transcode

Pull an input stream, transcode to H.264 NVENC, push to an output:

ffmpeg -hwaccel cuda -hwaccel_output_format cuda \
  -i "http://source/stream.ts" \
  -c:v h264_nvenc -preset p4 -tune ll -b:v 5M -maxrate 6M -bufsize 10M \
  -c:a aac -b:a 128k \
  -f mpegts "udp://127.0.0.1:1234"

Key flags:

• -hwaccel cuda -hwaccel_output_format cuda — keep frames on the GPU, avoid CPU round-trips
• -preset p4 — NVENC preset (p1 fastest → p7 slowest/best quality). p4 is a good live balance
• -tune ll — low-latency tuning for live
• -b:v 5M -maxrate 6M -bufsize 10M — target 5 Mbps, cap bursts at 6 Mbps

Step 4 — Build an ABR ladder

For adaptive streaming you output multiple rungs. This produces a 3-rung HLS ladder (1080p/720p/480p) from one input, all on the GPU:

ffmpeg -hwaccel cuda -hwaccel_output_format cuda \
  -i "http://source/stream.ts" \
  -filter_complex "[0:v]split=3[v1][v2][v3]; \
    [v1]scale_cuda=1920:1080[v1out]; \
    [v2]scale_cuda=1280:720[v2out]; \
    [v3]scale_cuda=854:480[v3out]" \
  -map "[v1out]" -c:v:0 h264_nvenc -preset p4 -b:v:0 5M \
  -map "[v2out]" -c:v:1 h264_nvenc -preset p4 -b:v:1 3M \
  -map "[v3out]" -c:v:2 h264_nvenc -preset p4 -b:v:2 1500k \
  -map a:0 -c:a aac -b:a 128k \
  -f hls -var_stream_map "v:0,a:0 v:1,a:0 v:2,a:0" \
  -master_pl_name master.m3u8 \
  -hls_time 6 -hls_list_size 6 \
  /var/www/hls/stream_%v.m3u8

scale_cuda keeps the scaling on the GPU too — critical for density. A single RTX PRO 4500 handles 8-12 of these 1080p ladders concurrently.

Step 5 — Switch to HEVC or AV1 for bandwidth savings

HEVC (hevc_nvenc) cuts bitrate ~35-40% for the same quality. AV1 (av1_nvenc, Ada/Blackwell only) saves even more:

# HEVC
-c:v hevc_nvenc -preset p4 -b:v 3M -maxrate 4M -bufsize 8M

# AV1 (RTX 40-series / Blackwell)
-c:v av1_nvenc -preset p4 -b:v 2.5M

Watch your STB fleet — older MAG boxes and many Smart TVs don't decode HEVC/AV1 reliably. Test before switching your whole panel, or you'll spend more in support tickets than you save in bandwidth.

Step 6 — Monitor GPU utilization

Watch encoder load in real time:

nvidia-smi dmon -s u

The enc column shows NVENC utilization. If it's pegged at 100%, you've hit the GPU's session limit — add a second GPU or move some streams to another box.

Density reference

Rough concurrent ladder counts we see in production:

The RTX PRO series has no NVENC session limit — that's why we recommend it for serious transcode farms over the consumer RTX 4090.

Where to go from here

Wrap this in a systemd service or a stream manager (Nimble, Flussonic). For a full panel setup see our Xtream UI guide. To size the GPU for your channel count, check how much GPU you actually need or talk to engineering.