Building a durable generation queue for Wan2GP on RunPod

I’ve been using Wan2GP to run Wan 2.1 I2V 14B on RunPod, and it’s great — except for one thing: the generation queue evaporates when the pod stops. Stop the pod, start it again, and your pending queue is gone. Browser refresh? Queue gone. Client disconnect? Queue gone.

I forked Wan2GP, added a durable queue that writes queue.zip on every mutation and on SIGTERM, auto-restores it on startup, and hardened two RunPod deployment paths around it: a pre-built Docker image for repeatable deployments, and a clone-and-run template that needs no image build. This post covers what I built, how it works, and all the gotchas I hit along the way.

The problem

Wan2GP’s queue lives in the browser session. Stop the RunPod pod (which sends SIGTERM), and the Python process dies. Start a new pod, open a new browser tab, and the queue is empty — all those pending jobs are gone.

This is especially painful on RunPod, where you pay by the minute. Stopping a pod to save money kills your queue. Crashing your browser kills your queue. It’s a terrible UX for something that should just… persist.

The solution: durable queue.zip

The core idea is simple: write the queue to a file on a persistent volume after every mutation, and restore it on startup.

How it works

Every mutation writes — every enqueue, complete, or abort calls autosave_queue(), which writes the current queue state to /workspace/queue.zip (an atomic write: temp file + os.replace). This is the same pattern databases use for crash safety.
SIGTERM flush — a signal handler (_graceful_shutdown) is registered before Gradio launches. When RunPod stops the pod, it sends SIGTERM. tini -g forwards it to the process group, gosu passes it through, and the handler flushes queue.zip before exit.
Autoload on startup — when Gradio starts, it reads /workspace/queue.zip and restores the queue into the new browser session. Completed tasks are pruned (they’re already done), so only pending/running tasks reappear.

What’s on the volume

/workspace is a RunPod Network Volume (200 GB) that persists across pod stops. Everything important lives there:

/workspace/
├── queue.zip          # durable queue (auto-saved, auto-restored)
├── wgp_config.json    # Gradio config (save_path → /workspace/outputs)
├── ckpts/             # model weights (auto-downloaded, ~15 GB, persists)
├── outputs/            # generated videos
├── hf-cache/           # Hugging Face cache
└── wan2gp.log          # server log

Two RunPod templates

I built two deployment paths, both published as public RunPod templates:

1. Hardened pre-built image (`template.json`)

A two-stage Docker build: stage 1 compiles torch 2.10+cu128 + SageAttention for your target CUDA architectures; stage 2 layers the RunPod hardening (tini, gosu, sshd, env config). The image is pushed to ghcr.io/riogesulgon/wan2gp:v3. Here’s what’s in the box:

tini -g + TINI_SUBREAPER=1 — forwards SIGTERM to the whole process group, so the durable-queue handler actually fires on pod stop. RunPod injects its own init as PID 1, so tini can’t be PID 1 — setting TINI_SUBREAPER registers it as a child subreaper instead, which is functionally equivalent for signal forwarding and zombie reaping.
gosu for privilege drop — runs wgp.py as a non-root user while keeping signal propagation intact. The original Wan2GP used su -p, which swallows signals. gosu doesn’t.
16 GB swap file on the volume — best-effort, for memory-hungry generations.
SSH on port 22 — key-only root login, host keys persisted on the volume, for rsync-over-SSH.
Pre-configured environment — PYTORCH_CUDA_ALLOC_CONF, HF_HUB_ENABLE_HF_TRANSFER, hf_transfer pip-installed for fast model downloads, NVENC ffmpeg defaults, single-worker Gradio, all baked into the image.
SageAttention compiled for Ampere SM 8.0/8.6 + Ada SM 8.9 (RTX A5000/A4500 and RTX 4060) — fast attention kernels baked in, with automatic fallback to flash/SDPA on other GPUs. The CUDA_ARCHITECTURES and MAX_JOBS build-args let you target other GPUs without touching the Dockerfile.

2. Simple clone-and-run (`template-simple.json`)

For when you don’t want to build or push a custom image. Uses the public pytorch/pytorch:2.7.0-cuda12.8-cudnn9-devel base image — no custom image build, no registry, no Docker-in-Docker. On pod start, the start command clones the fork onto the persistent /workspace volume and runs runpod/bootstrap.sh, which installs torch + requirements, then starts the Gradio UI.

First boot takes ~10–15 min (pip install of torch + requirements; subsequent boots are fast — pip cache lives on the volume).
SageAttention is not installed by default (Wan2GP falls back to flash/SDPA — works, slightly slower). To enable it later, run inside the pod: cd /workspace/Wan2GP && pip install --no-build-isolation git+https://github.com/thu-ml/SageAttention.git (the devel base image has nvcc, so it compiles for the pod’s GPU).
Same durable queue, same /workspace volume persistence, same wgp_config.json seeding — just without the baked-in compile step.
Use Connect → Web Terminal for shell access (no sshd needed).

The build gotchas

Building the image and publishing the template was a journey. Here are the problems I hit, in case you’re building something similar:

Docker-in-Docker doesn’t work on RunPod

RunPod pods are containers. Running Docker inside a container needs privileged mode, which RunPod’s standard pods don’t enable. So you can’t just spin up a RunPod pod and docker build. Instead, I built on a plain Linode VM with Docker installed natively — no DinD, no rootless buildah, just docker build. This is the simplest and most reliable path.

`git clone --depth 1` then `git fetch <sha>` fails on GitHub

The Dockerfile cloned the fork with --depth 1, then tried to git fetch --depth 1 origin <commit-sha>. GitHub rejects fetching an arbitrary SHA into a shallow clone with fatal: couldn't find remote ref. The fix: use a full clone + git checkout for pinned SHAs, and shallow clone only for main (the default).

SageAttention compile needs `CUDA_ARCHITECTURES` as a build-arg

SageAttention’s setup.py detects your GPU at build time via torch.cuda.get_device_capability(). But on a build host with no GPU (or the wrong GPU), this fails or picks the wrong arch. The upstream Dockerfile patched this to use the CUDA_ARCHITECTURES env var, but MAX_JOBS was hardcoded at 8. I made both configurable via --build-arg so you can do MAX_JOBS=4 to avoid OOM on smaller build hosts.

`PermissionError: [Errno 13] Permission denied: 'settings'`

The Docker image runs wgp.py as a non-root user (via gosu), but the application directory /opt/Wan2GP was owned by root. When wgp.py tried to os.mkdir("settings") (relative to cwd), it failed. Fix: chown -R user:user /opt/Wan2GP in the Dockerfile after the clone.

`KeyError: 'attention_mode'`

The start script seeded a minimal wgp_config.json with only save_path keys, but wgp.py expects many more required keys (attention_mode, video_profile, etc.) and crashes with KeyError. Fix: seed a complete config with all hard-required keys, or don’t seed at all and let wgp.py create its own defaults.

Gradio 307 redirects to internal pod IPs

Behind RunPod’s reverse proxy, accessing the Gradio root URL (/) returns a 307 redirect to the pod’s internal IP (http://100.65.24.95:60015//), which is unreachable from the browser. This happens because Gradio constructs redirect URLs from the server’s own address (0.0.0.0:7862), which resolves to the internal IP.

This took three attempts to fix:

ProxyHeadersMiddleware (attempt 1) — tried Starlette’s ProxyHeadersMiddleware. First the import path was wrong (proxyheaders vs proxy_headers), then it turned out Starlette’s version doesn’t do what we need. Abandoned.
ProxyHeadersMiddleware from uvicorn (attempt 2) — switched to uvicorn.middleware.proxy_headers. It imported fine but didn’t actually fix the redirect — it only fixes X-Forwarded-* headers, not the Location header on redirects.
_RunPodProxyFix custom ASGI middleware (final fix) — a hand-written ASGI middleware in wgp.py that intercepts 301/302/307/308 redirects pointing to internal IPs (10.x, 100.x, 127.x, 172.16–31.x, 192.168.x) and rewrites the Location header using the Host or X-Forwarded-Host from the original request. This catches any edge cases that GRADIO_ROOT_PATH doesn’t cover.

Combined with GRADIO_ROOT_PATH — set dynamically in start.sh from RUNPOD_POD_ID to https://{POD_ID}-7862.proxy.runpod.net — Gradio now constructs correct redirect URLs and the middleware catches any stragglers.

`hf_transfer` package missing

The Dockerfile set HF_HUB_ENABLE_HF_TRANSFER=1 but never installed the hf_transfer Python package. Hugging Face’s download code checks the env var first and errors out if the package isn’t available. Fix: pip install --no-cache-dir hf_transfer in stage 2 of the Dockerfile.

`tini` isn’t PID 1 on RunPod

The initial approach was to use tini -g as PID 1 via the Dockerfile ENTRYPOINT. But RunPod injects its own init process as PID 1, so tini can’t be PID 1 — it prints a warning (“Tini is not running as PID 1 and isn’t registered as a child subreaper”) and fails to reap zombies or forward signals. The fix: export TINI_SUBREAPER=1 in start.sh, which registers tini as a child subreaper — functionally equivalent to being PID 1 for signal forwarding and zombie reaping.

Bash function order: `log` called before definition

Bash requires functions to be defined before they’re called. When I added the GRADIO_ROOT_PATH block at the top of start.sh, it called log before the log() function was defined further down. The fix: move the log() definition to line 1 (right after set -Eeuo pipefail).

The repo

The fork is at github.com/riogesulgon/Wan2GP, branched from upstream deepbeepmeep/Wan2GP. Key files:

wgp.py — the durable queue implementation (autosave on every mutation, SIGTERM handler, autoload on startup) + the _RunPodProxyFix ASGI middleware
runpod/Dockerfile — stage 2 hardened RunPod image (clones fork, tini/gosu/sshd, env hardening)
Dockerfile — stage 1 deps image (torch 2.10+cu128, SageAttention patched for CUDA_ARCHITECTURES)
runpod/start.sh — boot script (swap, SSH, config seeding, symlinks, gosu, tini, UI readiness probe)
runpod/bootstrap.sh — simple template bootstrapper (clone + pip install + exec wgp.py)
runpod/build.sh — build helper for both stages with optional push
runpod/template.json — hardened image RunPod template API body
runpod/template-simple.json — clone-and-run RunPod template (no image build)
runpod/PLAN.md, README.md, PUBLISH.md, RUNBOOK.md — design docs and ops runbook
QUEUE_PERSISTENCE_PLAN.md — durable queue design document

Using the templates

Option A: Pre-built hardened image

Deploy from the template on RunPod with image ghcr.io/riogesulgon/wan2gp:v3 (RTX A5000, A4500, or RTX 4060, 200 GB disk).
Wait for the braille-art startup banner and ✅ Wan2GP UI READY on port 7862 in the logs.
Open Connect → HTTP [7862] → Gradio. (The GRADIO_ROOT_PATH fix ensures the proxy URL works without redirecting to an internal IP.)
Select Wan 2.1 I2V 14B, queue jobs, generate. First run downloads the model (~15 GB, one-time, persists on the volume). hf_transfer is installed for fast downloads.
Durable queue test: with jobs pending, Stop the pod (RunPod sends SIGTERM → tini forwards it → queue.zip flushes) → Start again → open a fresh browser tab → the pending queue autoloads. Completed jobs are not re-run.

Option B: Simple clone-and-run

Deploy from the simple template on RunPod (uses pytorch/pytorch:2.7.0-cuda12.8-cudnn9-devel base image, 200 GB disk).
First boot takes ~10–15 min (pip install torch + requirements). Subsequent boots are fast — pip cache lives on the volume.
Open Connect → HTTP [7862] → Gradio, or use Connect → Web Terminal for shell access.
Same durable queue, same persistence — just without SageAttention compiled in. Enable it later by running pip install --no-build-isolation git+https://github.com/thu-ml/SageAttention.git in the pod terminal.

Upstream status

The fork branches from upstream at 6b92c54 (Krea2 Edit, MSR 2.0, PiD 1.5, ConvROT LoRA). Upstream has since added 8 more commits — text encoder GGUF support, a Krea2 lora fix, a plugins catalog update, a prompt relay epsilon feature (PR #2019), and joy edit surgical improvements. These haven’t been merged into the fork yet; a git merge origin/main should be clean since the fork’s changes are isolated to wgp.py (queue + proxy fix) and the runpod/ directory (all new files).

Acknowledgments

deepbeepmeep/Wan2GP — the original Wan 2.1 I2V implementation
ProbeAI’s community RunPod templates — inspiration for the hardened template pattern (tini, gosu, swap, SSH)
The RunPod community for debugging Docker-in-Docker, build-arg gotchas, and template publishing
The Wan2GP community for the Krea2, PiD, and LTX2 work that the fork builds on top of