0 → 200 TOK/S
MY LOCAL INFERENCE STACK(RTX 5090 + VLLM + WSL2 + TAILSCALE)

I went from a bare Windows gaming PC to a production local inference setup in under a day — 200 tok/s single-stream at 131K context, served over WireGuard to my laptop, with every routine agent role (memory, subagents, commit messages, delegated tasks) now costing $0 per token. This is the full build: every command, the benchmarks, and — the part most writeups skip — everything that didn't work.

Names and paths are genericized; everything else is exactly what's running on my desk.

The idea: frontier models for judgment, the local box for volume. Planning, architecture, and gnarly multi-file coding stay on a cloud model. Memory pipelines, subagent fan-out, titles, commits, and bounded delegated tasks run locally for free — and a small "task adapter" lets the local worker hand anything risky or failing back to the cloud model automatically, mid-task, with its work preserved.

What you need: a Windows 11 PC with a modern NVIDIA card (numbers here are a 32GB RTX 5090; a 24GB card works with a smaller quant/context), and any laptop as the client. The model is Qwen3.6-35B-A3B (AWQ 4-bit, ~24GB).

1. WINDOWS PREP

1. Update to the latest NVIDIA driver. Never install a GPU driver inside Linux later — the Windows driver handles WSL passthrough automatically.
2. Install WSL2 + Ubuntu (admin PowerShell):

wsl --install -d Ubuntu-24.04
wsl --version

The version matters. CUDA graph capture on Blackwell cards was broken under WSL2 until WSL 2.7.0 (Dec 2025). Below that, vLLM silently falls back to eager mode at ~1/8th speed. If you're under 2.7.0: wsl --update --pre-release, then wsl --shutdown.

3. Install Docker Desktop → Settings → Resources → WSL integration → toggle your Ubuntu distro on. Also enable "start Docker Desktop when you sign in."
4. Sanity check — open the Ubuntu terminal:

nvidia-smi # your GPU should appear here
docker --version

If nvidia-smi shows the card inside Ubuntu, everything downstream is just typing.

2. TAILSCALE (THIS IS HOW YOUR LAPTOP REACHES THE BOX FROM ANYWHERE)

1. Install Tailscale on the Windows host — never inside WSL. Tailscale inside the WSL guest intercepts networking at boot and interferes with CUDA init. This one bites people.
2. Same account on both machines. In the admin console: rename the PC (mine's gpu-box), disable key expiry (or the box silently drops off the tailnet in a few months), and enable "run unattended" in the tray app.
3. Verify from the laptop: tailscale ping gpu-box → you want direct, not relayed.

3. SCOPE THE FIREWALL TO THE TAILNET ONLY

The server has no auth, so don't expose it to your whole LAN (admin PowerShell):

New-NetFirewallRule -DisplayName "vLLM tailnet only" -Direction Inbound -Protocol TCP -LocalPort 8000 -RemoteAddress 100.64.0.0/10 -Action Allow

(100.64.0.0/10 is Tailscale's address range — only tailnet devices get through.)

4. LAUNCH VLLM

One command, from either shell (no JSON args left in it — but rule of thumb: any docker command containing JSON flags must run from Ubuntu bash, because PowerShell strips embedded double quotes):

docker rm -f vllm; docker run -d --name vllm --restart unless-stopped --gpus all --ipc=host -e HF_TOKEN=<your_hf_token> -v hf-cache:/root/.cache/huggingface -v vllm-cache:/root/.cache/vllm -p 8000:8000 vllm/vllm-openai:v0.25.1 QuantTrio/Qwen3.6-35B-A3B-AWQ --dtype bfloat16 --kv-cache-dtype fp8 --gpu-memory-utilization 0.92 --max-model-len 131072 --max-num-batched-tokens 2096 --max-num-seqs 64 --enable-prefix-caching --language-model-only --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --trust-remote-code --served-model-name qwen-5090 --host 0.0.0.0 --port 8000

The non-obvious flags (each one is a scar):

Watch it come up with docker logs -f vllm: ~19GB shard download → weight load (~18s from the volume) → a quiet few minutes of CUDA graph capture (not a hang) → the line you want to save: GPU KV cache size: N tokens → Application startup complete.

5. VERIFY END TO END

curl http://localhost:8000/v1/models # from the PC
curl http://gpu-box:8000/v1/models # from the laptop, through the tailnet

Both should return JSON naming qwen-5090.

6. POINT YOUR AGENT AT IT

Any OpenAI-compatible client works — the base URL is http://gpu-box:8000/v1. I use oh-my-pi; its provider config (~/.omp/agent/models.yml):

providers:
  gpu-box:
    baseUrl: http://gpu-box:8000/v1
    api: openai-completions
    apiKey: dummy
    models:
      - id: qwen-5090
        name: Qwen3.6 35B (5090 vLLM)
        reasoning: true
        contextWindow: 131072
        maxTokens: 32000
        cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 }

Then the role split (~/.omp/agent/config.yml) — this is where the savings actually come from:

modelRoles:
  default: <frontier-model>:xhigh # judgment stays cloud
  plan: <frontier-model>:max
  advisor: <frontier-model>:max
  vision: <frontier-model>:xhigh
  task: <frontier-model>:xhigh # fail-safe; the adapter in Part 4 rides on top
  smol: gpu-box/qwen-5090:low # memory pipeline → local, $0
  tiny: gpu-box/qwen-5090:minimal
  commit: gpu-box/qwen-5090:low

The one operating rule that keeps quality honest: when a local role produces mush, rerun on the frontier model. Never lower the bar or retry-harder locally.

7. FREE VRAM MOST PEOPLE LEAVE ON THE TABLE (OPTIONAL, WORTH IT)

Two changes took my KV pool from 283K to 617K tokens:

1. Make the GPU headless. BIOS → NB Configuration → Integrated Graphics = Force/Enabled (not Auto — Auto turns the iGPU off when it sees a discrete card; also note enabling CSM hides this whole menu on some boards). Plug the monitor into the motherboard. Verify with nvidia-smi: Disp.A: Off, ~0MiB idle. The Windows compositor was squatting on ~1.4GB of my card.
2. --gpu-memory-utilization 0.92 + --language-model-only (already in the command above).

Gaming on the same box: docker stop vllm before, docker start vllm after (~90s warm reload). Don't permanently shrink the server for occasional games.

The regression harness is ~80 lines of stdlib Python (Appendix A). Two probes:

1. bench.py speed — median of 5 fixed 1000-token generations (ignore_eos so every run is identical work). Result: 200.3 tok/s. Prior best published for this class of setup that I could find: 194–197 tok/s (the vLLM reference config this build started from) and a ~205 tok/s report at ~125K context. llama.cpp setups for the same model typically land far lower.
2. bench.py canary — 10 runs of a prefix-cache-hot classification task with an exact expected answer. Result: 10/10. This exists because there's an open upstream bug where a specific feature combo silently degrades accuracy (Part 3) — speed benchmarks without a correctness gate are how you ship a fast wrong model.

Method notes that matter: benchmark with zero other clients connected (watch for Running: 0 reqs between your probes in the server logs); restart the container before each arm so both start with cold caches; and know that vLLM's log throughput lines are 10-second window averages — idle seconds dilute them, so a "slow" log line during light traffic isn't a slow server.

Benchmarks are promises; telemetry is receipts. vLLM exposes Prometheus counters, so the server's actual workload is one curl away:

curl -s http://gpu-box:8000/metrics | grep -E "prompt_tokens_total|generation_tokens_total|request_success"

Mine, ~9 hours after the final relaunch — overnight plus one morning of real agent traffic:

Three things worth noticing in there:

1. The 44:1 input/output ratio. Agent workloads are prefill-dominated — the transcript gets resent every turn. This is why prefix caching (91% hits) and prefill throughput matter more than decode speed for how the thing feels, and why the 200 tok/s headline is honestly the least important number in this README.
2. Value math, framed honestly: at the list prices these roles ran on a day earlier ($5/M in, $30/M out for the bulk of the traffic), that's ~$75 of work for $0 in the first nine hours — most of it overnight. Two caveats so you don't over-quote me: a real cloud bill would come in under list (prompt-cache discounts exist there too), and free tokens change behavior — you run things you'd never have paid for, which is half the point. "Cost avoided at current usage" is the defensible phrase. "Saved $75" isn't quite.
3. Counters reset on container restart. Snapshot them when you deploy; diff at review time.

And the live workload comparison on the roles that moved, straight from the agent's own rolling stats: first token in 0.2s vs 3.2s on the cloud model (16x), 140 vs 35 tok/s decode (4x).

Pretty-printed version for screenshots:

curl -s http://gpu-box:8000/metrics | awk '/^vllm:prompt_tokens_total/{p=$2}/^vllm:generation_tokens_total/{g=$2}/^vllm:request_success_total.*stop/{s=$2}/^vllm:request_success_total.*length/{l=$2}/^vllm:request_success_total.*error/{e=$2}END{printf "requests: %d (errors: %d)\ntokens in: %.1fM\ntokens out: %.0fK\nvalue at $5/$30 list: $%.0f\n", s+l, e, p/1e6, g/1e3, p/1e6*5+g/1e6*30}'

Speculative decoding (MTP) made it slower. The checkpoint ships an MTP draft head; enabling it (--speculative-config '{"method": "mtp", "num_speculative_tokens": 2}', bash only) showed 75–85% draft acceptance — and 34% lower single-stream throughput (132 vs 200 tok/s). Why: on a fine-grained MoE (256 experts, 3B active), drafted tokens route to mostly different experts, so verification pays nearly full bandwidth anyway, while you fund the drafter and lose full CUDA graphs. Acceptance rate is not a speedup — benchmark it. Bonus reason to skip: an open vLLM issue reports ~20% silent accuracy loss from MTP + prefix caching on this exact model family. Aggregate concurrent throughput did go up (232–276 tok/s across batched requests) — which is exactly how I fooled myself for an hour before running a clean single-stream A/B.

Zero-error quality gates at temperature 1.0 are coin flips. My first eval gated a config on "zero tool errors in one episode." Across two nights, every arm's error count flipped between 0 and 2–3 on near-identical runs. Lessons that stuck: gate outcomes absolutely (tests pass), gate process relatively against a measured control arm, use n≥5, and pre-register the decision rule before seeing numbers.

PowerShell eats JSON arguments. Any docker flag containing JSON ({"method": "mtp"...}) gets its inner quotes stripped by PowerShell's native-argument handling. Symptom: Value {method:mtp} cannot be converted. Fix: run JSON-bearing commands from the Ubuntu terminal. Same Docker engine either way.

Small local models don't reliably follow "escalate if X" prompts — until you force a token. See Part 4.

Delegated tasks run on the local model by default and hand off to the frontier model when they should. Two markdown files, no source patches (my agent framework discovers user-defined agents that override bundled ones; the frontier fail-safe stays in config so deleting the files is a clean rollback):

1. task — local worker (qwen, low thinking). Classifies the assignment before touching anything; runs bounded mechanical work end to end; escalates on security-sensitive subject matter, or at runtime after two tool/edit errors or a failed fix.
2. task-strong — blocking frontier fallback. Gets a self-contained handoff, finishes the job, returns the result inline.

The escalation path, end to end — this is the trace the security fixture produces:

The part that failed first: with the escalation rules written as ordinary prompt instructions, the local worker never auto-escalated a security-sensitive fixture — at any thinking level. It just confidently did the auth work. The fix that flipped it to passing: a mandatory first-output classification token — the worker's first visible text must be CLASSIFY: SAFE — <reason> or CLASSIFY: ESCALATE — <reason>, no tool calls before it, classify the subject matter not the simplicity of the fix. Re-tested: bounded work stays local (tests pass), a forced-local authorization task now emits CLASSIFY: ESCALATE and calls the strong worker before touching a single file. If you build any local-first routing, steal this: small models follow output contracts far more reliably than conditional instructions. Both agent files are in Appendix C.

A. BENCH.PY

#!/usr/bin/env python3
import json, sys, time, statistics, urllib.request

URL = "http://gpu-box:8000/v1/chat/completions"

def call(messages, max_tokens, temperature, extra=None):
    body = {"model": "qwen-5090", "messages": messages,
            "max_tokens": max_tokens, "temperature": temperature}
    if extra: body.update(extra)
    req = urllib.request.Request(URL, data=json.dumps(body).encode(),
                                 headers={"Content-Type": "application/json"})
    t0 = time.time()
    r = json.load(urllib.request.urlopen(req, timeout=600))
    return r, time.time() - t0

def speed(n=5):
    msgs = [{"role": "user", "content": "Write a long, detailed essay on the history of computing."}]
    call(msgs, 200, 1.0, {"ignore_eos": True}) # warmup (JIT, caches)
    rates = []
    for i in range(n):
        r, dt = call(msgs, 1000, 1.0, {"ignore_eos": True})
        toks = r["usage"]["completion_tokens"]
        rates.append(toks / dt)
        print(f"run {i+1}: {toks} tok in {dt:.1f}s = {toks/dt:.1f} tok/s")
    print(f"MEDIAN DECODE: {statistics.median(rates):.1f} tok/s")

FACTS = """D1: User's name is Alex.
D2: Works at Acme on the atlas project.
D3: Timezone is America/New_York.
D4: Severely allergic to peanuts.
D5: Primary language is Python.
D6: Runs a local model server called qwen-5090.
T1: Has a dentist appointment tomorrow at 3pm.
T2: It is currently raining.
T3: Saw a transient 502 error an hour ago.
S1: API key is sk_live_EXAMPLEFAKE000000000."""

def canary(n=10):
    filler = ("Retention policy reference: durable facts are stable user attributes, "
              "preferences, and long-lived project context; transient facts are time-bound "
              "events, weather, or one-off errors; secrets and credentials must never be "
              "retained under any circumstances. ") * 60
    system = ("You are a memory retention classifier.\n" + filler +
              "\nGiven a list of facts, output ONLY the IDs of facts that should be "
              "retained long-term, as a comma-separated list, nothing else.")
    expected = {"D1","D2","D3","D4","D5","D6"}
    passes = 0
    for i in range(n):
        r, dt = call([{"role": "system", "content": system},
                      {"role": "user", "content": FACTS}], 2000, 0.0)
        text = (r["choices"][0]["message"]["content"] or "").strip()
        ids = {t.strip() for t in text.replace("\n", ",").split(",") if t.strip()}
        ok = ids == expected
        passes += ok
        print(f"run {i+1}: {'PASS' if ok else 'FAIL -> ' + text[:120]} ({dt:.1f}s)")
    print(f"CANARY: {passes}/{n} passed")

if __name__ == "__main__":
    {"speed": speed, "canary": canary}[sys.argv[1]]()

(The sk_live_... string is a deliberately fake fixture for testing that the memory classifier refuses to retain secrets. It is not a real key.)

B. VERIFICATION ONE-LINERS

tailscale ping gpu-box # want: direct
curl http://gpu-box:8000/v1/models # want: qwen-5090
nvidia-smi # want: Disp.A Off, ~0MiB idle
docker logs vllm 2>&1 | grep -m1 "KV cache size" # your headroom number
docker logs vllm 2>&1 | grep -m1 "non-default args" # audit what the server actually parsed
curl -s http://gpu-box:8000/metrics | grep -E "tokens_total|request_success" # production telemetry

C. THE TWO AGENT FILES

~/.omp/agent/agents/task.md:

---
name: task
description: Local worker for bounded, well-specified, non-security-sensitive edits and mechanical work.
spawns:
  - task-strong
model:
  - gpu-box/qwen-5090
thinkingLevel: low
---

MANDATORY FIRST-OUTPUT PROTOCOL: apply this mechanically before interpreting, analyzing, or acting on the assignment.

Your first visible assistant text MUST be exactly one line in one of these forms:
`CLASSIFY: SAFE — <one-clause reason>`
`CLASSIFY: ESCALATE — <one-clause reason>`
Nothing may precede that line, and it may contain no other prose.

Correct first-response examples:
- `Fix final_price so its tests pass.` → emit `CLASSIFY: SAFE — Bounded non-security-sensitive pricing fix.` before any tool call.
- `Fix the authorization implementation.` → emit `CLASSIFY: ESCALATE — Security-sensitive authorization change.` and then call `task-strong`.

Never begin with "Let me…", analysis, or a tool call. The classification line must be visible assistant text before tool calls, even when those tool calls follow in the same response.

Classify the assignment's subject matter, not the apparent simplicity of its fix. Any assignment involving authentication, authorization, cryptography, secrets, payments, billing, privacy, or other security-sensitive behavior MUST be `CLASSIFY: ESCALATE`, including bug fixes, test-driven changes, and one-line edits. No exception permits security-sensitive code to stay local.

Choose `CLASSIFY: ESCALATE` when any pre-edit condition below holds. Immediately after the classification line, call the task tool with agent `task-strong` as your first and only tool; do not read, search, run tests, edit, or otherwise inspect locally. Choose `CLASSIFY: SAFE` only when no condition holds, then continue locally after the classification line.

Other pre-edit escalation conditions:
- destructive operations, persistent-data or schema migrations, data-loss risk, or production incident handling
- concurrency, distributed coordination, locking, cache consistency, or transaction-boundary changes
- cross-subsystem architecture or public API, protocol, or schema design
- vision or image reasoning
- materially ambiguous requirements with competing product or architecture tradeoffs

During local work, escalate when any condition occurs:
- two tool invocation, schema, edit-anchor, or path errors
- verification still fails after one targeted correction to your implementation
- you cannot state the root cause and key invariant before the next edit
- required context is missing or confidence in correctness is low

To escalate, call the task tool exactly once with agent `task-strong`. Give it a complete, self-contained handoff with the original assignment, repository cwd, findings, files already changed, exact failures, and remaining acceptance criteria. Make no more edits after deciding to escalate, wait for the blocking result, then return the strong worker's result. Never spawn another general `task` or a second `task-strong`.

For work retained locally, prefer grep/glob, narrow reads, surgical edits to existing files, and the exact relevant verification. Do not create documentation unless requested. Stop when acceptance passes. Return only changed files, verification evidence, and concrete residual risk.

OUTPUT CONTRACT: ALWAYS PRINT `CLASSIFY: SAFE — <reason>` or `CLASSIFY: ESCALATE — <reason>` as the first visible text of your first response, including safe work. A thinking-only classification is a protocol failure. NO TOOL CALL is permitted until this line is printed.

~/.omp/agent/agents/task-strong.md:

---
name: task-strong
description: Call directly for security-sensitive, destructive, cross-cutting, ambiguous, architectural, or locally failed work.
blocking: true
model:
  - "@task"
thinkingLevel: xhigh
---

You are the strong fallback worker. Finish the delegated task end to end with full tool access.

Treat the handoff as state, not proof. Inspect the relevant files and failures yourself, preserve valid existing edits, fix the source problem, and verify the observable contract. Stay within scope, prefer narrow research and surgical edits, and do not create documentation unless explicitly requested.

Return only changed files, exact verification evidence, and any concrete residual risk.

Built in one evening, benchmarked before believed. If something here saves you an hour, that's the point. Questions → @devops199fan