The usual instinct is to check whether the output file exists and skip if so. That fails in the direction that hurts: a stale file with the right name silently satisfies the check and you ship the wrong content.
Hash what went in instead.
import hashlib, json
def stage_key(stage_name: str, inputs: dict) -> str:
payload = json.dumps(inputs, sort_keys=True, ensure_ascii=False)
digest = hashlib.sha256(payload.encode()).hexdigest()[:16]
return f"{stage_name}:{digest}"
sort_keys=True matters. Without it, two identical dicts serialise differently depending on insertion order and you get cache misses that look like cache bugs.
This is where it goes wrong. Miss a field and you serve stale results after a change that should have invalidated them.
key = stage_key("image", {
"prompt": scene.prompt,
"model": model_id,
"style_preset": config.style,
"aspect_ratio": config.aspect_ratio,
"seed": scene.seed,
"prompt_template_version": TEMPLATE_VERSION, # bump when you edit it
})
prompt_template_version is the one people forget. Edit the base template, and every prompt changes even though no scene did. Without a version in the key, the whole library goes stale invisibly.
def cached(key: str, produce, store: Path):
marker = store / f"{key.replace(':', '_')}.json"
if marker.exists():
meta = json.loads(marker.read_text())
if Path(meta["path"]).exists():
return meta["path"]
marker.unlink() # artifact gone, drop the marker
path = produce()
marker.write_text(json.dumps({
"path": str(path),
"created": datetime.now(timezone.utc).isoformat(),
}))
return path
Checking that the artifact still exists is what keeps this honest. A marker pointing at a deleted file is worse than no marker.
The second benefit is bigger than the caching. A stage keyed on its input is idempotent, so a retry after a transient failure re-does only the work that did not complete.
for attempt in range(max_attempts):
try:
return cached(key, lambda: call_api(scene), store)
except TransientError:
if attempt == max_attempts - 1:
raise
time.sleep(base_delay * (2 ** attempt) + random.uniform(0, 0.5))
Without idempotency, a retry policy on a paid API is a policy for spending more money.
One that bit me: a failed-items database left over from testing contained fabricated entries. On the next real run they fed straight into the retry path and produced work nobody asked for.
Keep scratch output out of the directories the pipeline reads. If a test writes into the live tree, it is not a test any more — it is an input.
At my measured rate, image generation is 96% of per-video cost. A single avoided rerun pays for the afternoon it took to build this. Across a library where many scenes are semantically near-identical, the cache stops being a retry optimisation and becomes a margin.
]]>“Should this scene be a close-up or a wide shot?”
That is not a language problem. It is a weighted choice constrained by a few rules: vary the framing, avoid jarring transitions, tighten during emotional peaks, open up at the start and end.
SHOT_WEIGHTS = {
"extreme_closeup": 0.10,
"closeup": 0.25,
"medium": 0.30,
"wide": 0.25,
"overhead": 0.05,
"insert": 0.05,
}
INVALID_TRANSITIONS = {
("extreme_closeup", "wide"),
("wide", "extreme_closeup"),
("overhead", "extreme_closeup"),
}
A hundred lines of Python covers it, and covers it better.
Telling a model “do not use the same shot more than twice in a row” is a request. In code it is a guarantee.
def next_shot(self, index, total, intensity=0.5):
weights = self._adjust_for_arc(index, total, intensity)
if self.recent:
last = self.recent[-1]
if self._run_length(last) >= self.max_consecutive:
weights[last] = 0.0 # hard block, not a nudge
shot = self._weighted_choice(weights)
if self.recent and (self.recent[-1], shot) in INVALID_TRANSITIONS:
shot = self._pick_valid(self.recent[-1], weights)
self.recent.append(shot)
return shot
One detail there cost me real output quality. My first version damped the repeated shot’s weight to 0.1 instead of zeroing it. Runs of three still slipped through, and a limit that is usually enforced is not a limit. Set it to zero.
planner = ShotPlanner(style="mixed", seed=42)
Same seed, same plan, every time. That means you can regenerate a job after fixing something downstream and get an identical structure, and you can write tests against the planner:
def test_no_long_runs():
p = ShotPlanner(style="mixed", seed=42)
seq = [p.next_shot(i, 149).value for i in range(149)]
run = worst = 1
for a, b in zip(seq, seq[1:]):
run = run + 1 if a == b else 1
worst = max(worst, run)
assert worst <= 2, f"run of {worst}"
You cannot write that test against a model call.
| model call | weighted table | |
|---|---|---|
| latency | seconds | microseconds |
| cost per job | non-zero | zero |
| reproducible | no | with a seed |
| testable | not really | yes |
| enforces hard limits | asks | guarantees |
Interpreting the actual content of a scene. “This passage is about betrayal in a rain-soaked alley” is a language task, and no lookup table produces it.
Split the two. Let the model read meaning; let code decide structure. In my pipeline that removed an entire class of API calls, made output reproducible, and — the part I did not expect — improved consistency, because rules that are enforced beat rules that are requested.
One warning from my own experience: the deterministic planner sat in my codebase for months, disabled by an argument-name mismatch, silently falling back to the model on every run. Log your fallbacks loudly. A silent fallback is a feature you are paying for and not receiving.
]]>A hardcoded rate table goes stale in every direction at once. Providers change pricing. Your model string silently falls back to a more expensive one when the primary is unavailable. Token counting differs from the provider’s own accounting. Requests carry overhead you never counted.
None of these announce themselves. Your total stays plausible while being consistently wrong.
Most gateways will return the billed amount if you ask. On an OpenAI-compatible endpoint it is one field:
payload = {
"model": model,
"messages": messages,
"usage": {"include": True}, # <- this line
}
resp = requests.post(endpoint, json=payload, headers=headers, timeout=120)
data = resp.json()
usage = data.get("usage", {})
if "cost" in usage:
cost, estimated = float(usage["cost"]), False
else:
cost, estimated = estimate_from_table(model, usage), True
Fall back to the table rather than replacing it. Not every provider returns cost, and you still want a number for the ones that do not.
provider model (purpose) calls cost
---------------------------------------------------------------
openrouter gemini-flash-image (image) 149 5.0735
elevenlabs scribe (stt) 6 0.1492 EST
openrouter gemini-flash-lite (director) 1 0.0181
---------------------------------------------------------------
TOTAL 5.2408
That EST flag is not decoration. It tells you which rows you may price against and which you may not. When somebody asks what a unit costs, you can answer precisely for the measured portion and honestly about the rest.
One call proves the field parses. It does not prove the provider returns it reliably.
events = [json.loads(l) for l in open(log_path)]
actual = sum(1 for e in events if not e["estimated"])
print(f"{actual}/{len(events)} actual")
costs = [e["cost_usd"] for e in events if not e["estimated"]]
print(f"min {min(costs):.5f} mean {sum(costs)/len(costs):.5f} max {max(costs):.5f}")
Twenty of twenty came back actual, clustered between $0.03377 and $0.03452. Tight clustering is itself a signal — wide spread would have meant a fallback model was being used more often than I thought.
PRICES = {
"gemini-flash-image": 0.034, # measured 2026-08, was 0.030 (est.)
}
Even with real cost available, keep the table honest. It is what runs when the provider omits the field, and a stale fallback reintroduces the same drift through a smaller door.
At the measured rate, a 149-image job is $5.07 rather than the $4.47 I had been reporting. On one video that is 60 cents. On a price list built for thousands of them, it is the difference between a margin and a slow loss.
Measure the thing you sell. Estimate only what you cannot measure, and label it.
]]>ufw status, and saw no rule for port 8000. Then I opened http://server-ip:8000 from another network and my admin dashboard loaded.
UFW writes rules into the INPUT chain. Traffic to a published container port does not traverse INPUT — Docker DNATs it and it passes through FORWARD, where Docker inserts its own accept rules during startup.
So ufw status is accurate about what it manages. It just does not manage this.
# UFW's view — nothing on 8000
sudo ufw status numbered
# Reality
sudo iptables -t nat -L DOCKER -n | grep 8000
Every port you publish with -p or a compose ports: entry is reachable from anywhere the network can reach the host, regardless of what UFW says.
Bind to loopback. The cleanest option when the service is only for you.
services:
panel:
ports:
- "127.0.0.1:8000:8000" # not "8000:8000"
Reach it over an SSH tunnel or a private mesh network. A port that is not listening on a public interface cannot be scanned.
Filter in DOCKER-USER. Docker provides this chain specifically so your rules survive its own rule generation, and it is evaluated before Docker’s accepts.
sudo iptables -I DOCKER-USER -p tcp --dport 8000 \
! -s 203.0.113.10 -j DROP
sudo iptables -L DOCKER-USER -n --line-numbers
Persist it, or you lose the rule on the next reboot:
sudo apt install -y iptables-persistent
sudo netfilter-persistent save
When something is unreachable and you cannot tell why, the block log names the exact packet.
sudo grep 'BLOCK' /var/log/ufw.log | tail -5
[UFW BLOCK] IN=br-82f64cb60043 SRC=10.0.1.5 DST=10.0.0.1 DPT=9631 SYN
That single line told me something a dozen assumptions had not: my container network was on 10.0.0.0/8, not the 172.16.0.0/12 default I had written a rule for. The rule was correct and matched nothing.
Do not write rules against veth names. They are regenerated every time a container restarts.
# Fragile — gone after the next restart
sudo ufw allow in on veth617b319 to any port 9631
# Durable — survives container and network recreation
sudo ufw allow from 10.0.0.0/8 to 10.0.0.1 port 9631 proto tcp
Bridge names like br-82f64cb60043 are more stable but still tied to a network that can be recreated. Address-based rules outlive both.
sudo ss -tulpn | grep LISTEN
Anything bound to 0.0.0.0 is public unless something outside UFW is stopping it. Run this on a server you believe is locked down — the results are usually educational.
A module that creates a VPC, a database, a cache, and a monitoring stack is not a module. It’s a novel. Split it.
modules/
vpc/
main.tf
variables.tf
outputs.tf
database/
main.tf
variables.tf
outputs.tf
monitoring/
main.tf
variables.tf
outputs.tf
Each module should be understandable in one read. If you need to scroll to understand what a module creates, it’s doing too much.
Every variable should have a type, a description, and a default when a sensible one exists.
variable "instance_type" {
type = string
description = "EC2 instance class for the app servers"
default = "t3.medium"
validation {
condition = can(regex("^t3\\.", var.instance_type))
error_message = "Only t3 instance types are supported for this workload."
}
}
variable "enable_monitoring" {
type = bool
description = "Whether to attach CloudWatch agent and alarms"
default = true
}
The validation block is not optional. It’s the difference between a typo caught at plan time and a production outage at apply time.
A module’s outputs are its public interface. Export what consumers need, nothing more.
output "vpc_id" {
description = "The ID of the created VPC"
value = aws_vpc.main.id
}
output "private_subnet_ids" {
description = "IDs of the private subnets, for use by app modules"
value = aws_subnet.private[*].id
}
Don’t export every attribute of every resource. That couples consumers to implementation details you might want to change later.
Remote state, state locking, and a clear separation between environments. Non-negotiable.
terraform {
backend "s3" {
bucket = "terraform-state-prod"
key = "app/terraform.tfstate"
region = "us-east-1"
dynamodb_table = "terraform-locks"
encrypt = true
}
}
One state file per environment per component. Never share state between dev and prod. Never run terraform apply from a laptop against production state.
The best Terraform test is terraform plan in CI on every pull request. If the plan output is clean, the change is probably safe. If it’s a wall of red, something is wrong.
Readable infrastructure is a team sport. Write it for the person who has to change it at 2am, because eventually that person is you.
]]>sys.path.insert(0, PROJECT_ROOT)
import cost_tracker
It worked. It also introduced an ImportError in a module I had not touched, which only appeared under one import order.
sys.path is searched in order. Putting the project root at position zero means it is searched before the directory of the module doing the importing.
My layout had a config_manager.py at the root and another inside a stage directory. They were different files with different contents, and the stage relied on its local one.
project/
config_manager.py <- generic
cost_tracker.py
03_image-generator/
config_manager.py <- stage-specific
generator.py <- expects the local one
runpod_provider.py
After insert(0, PROJECT_ROOT), from config_manager import config inside the stage resolved to the root file. Same name, different module, missing attribute.
Import order decided the outcome. Python caches modules in sys.modules by name, so whichever import ran first won and the second silently received it.
# Order A — generator imports its config first, caches the local one
import generator # config_manager -> stage-local
import runpod_provider # gets the cached stage-local one, fine
# Order B — runpod_provider goes first, and the root wins
import runpod_provider # config_manager -> root
import generator # gets the cached root one, ImportError
In practice order A happened to be the common path, so the bug existed for a while without firing. Those are the ones that surface in production.
sys.path.append(PROJECT_ROOT) # not insert(0, ...)
append puts the root last. Stage-local modules keep priority; the shared helper is still reachable because nothing else defines that name.
import config_manager
print(config_manager.__file__)
# .../03_image-generator/config_manager.py <- correct
Test both orders explicitly. It takes one extra line and it is the only thing that catches this class of bug:
# tests/test_import_order.py
import subprocess, sys
for first, second in [("generator", "runpod_provider"),
("runpod_provider", "generator")]:
code = f"import {first}, {second}; import config_manager; print(config_manager.__file__)"
out = subprocess.run([sys.executable, "-c", code], capture_output=True, text=True)
assert "03_image-generator" in out.stdout, f"{first} first -> {out.stdout}"
Path manipulation is a workaround. The real answer is packaging: make the shared code a proper package, install it with pip install -e ., and let normal resolution do its job.
project/
pyproject.toml
src/pipeline_common/
__init__.py
cost_tracker.py
from pipeline_common import cost_tracker # unambiguous, order-independent
If you are past a couple of stages, do this instead of reaching for sys.path. Duplicate module names across a repository are a slow-burning fuse, and insert(0, ...) is what lights it.
The transcription step produced 416 subtitle segments with word-level timings. Those are semantic boundaries — real ones, derived from where the speaker actually paused. The planner ignored all of them and used a stopwatch instead.
# What it did
scene_count = int(total_duration / 10)
scenes = [
Scene(start=i * 10, end=(i + 1) * 10)
for i in range(scene_count)
]
Every boundary here is arbitrary. Some land mid-clause. Some split a sentence whose two halves need the same image.
Keep the duration bounds — they exist so no image sits on screen too long — but let content decide where inside them the cut falls.
def plan_scenes(segments, min_len=10.0, max_len=15.0):
scenes, current = [], []
for seg in segments:
current.append(seg)
span = current[-1].end - current[0].start
if span < min_len:
continue
# Past the minimum: cut at a real boundary, or at the ceiling
if seg.ends_sentence or span >= max_len:
scenes.append(Scene(
start=current[0].start,
end=current[-1].end,
text=" ".join(s.text for s in current),
))
current = []
if current:
scenes.append(Scene(
start=current[0].start,
end=current[-1].end,
text=" ".join(s.text for s in current),
))
return scenes
The scene now carries its own text, which means the prompt for its image can be built from what is actually being said during it — not from whatever words happened to fall inside a ten-second window.
At my measured rate the arithmetic is blunt:
| images | image spend | |
|---|---|---|
| fixed 10s | 149 | $5.07 |
| semantic, 10-15s | ~60 | $2.04 |
Image generation was 96% of my per-video cost. Cutting image count by 60% cuts the whole unit cost by nearly the same proportion. No other optimisation in the pipeline comes close.
Longer scenes with static images look worse, not better. A still frame held for fifteen seconds reads as a stall. You have to spend some of what you saved on motion.
const progress = frame / durationInFrames;
const scale = interpolate(progress, [0, 1], [1.0, 1.08]);
const x = interpolate(progress, [0, 1], [0, -20]);
<img style={{ transform: `scale(${scale}) translateX(${x}px)` }} />
Slow, restrained, and varied in direction between scenes so it does not become its own pattern. Roughly 8% scale over the full duration is enough — anything more reads as an effect rather than as life.
Ship the two changes together. Semantic grouping without motion makes the output cheaper and worse, which is the wrong trade to make on the thing you sell.
]]>Your CLI’s output is its UI. Treat it that way.
use std::io::{self, Write};
fn main() {
// Wrong: dumping raw data
println!("{:?}", results);
// Right: structured, scannable output
for item in &results {
println!(" {} {:<24} {}", item.status_icon(), item.name, item.detail);
}
// Always flush before exit on interactive output
io::stdout().flush().unwrap();
}
Align columns. Use color to communicate state, not to decorate. Red means failure, green means success, yellow means attention. Never use color as the only signal, and always respect NO_COLOR.
A good error message tells the user what went wrong and what to do next.
// Wrong
eprintln!("Error: connection failed");
// Right
eprintln!("error: could not reach registry.example.com:443");
eprintln!(" hint: check your network connection or VPN status");
eprintln!(" hint: run with --offline to use the local cache");
Exit codes matter too. Zero for success, non-zero for failure, and different codes for different failure classes so scripts can branch on them.
Follow the conventions users already know:
-h / --help for help-v / --verbose for more output-q / --quiet for less output--version for version info--dry-run when the tool changes stateUse clap with derive macros. The help output it generates is better than anything you’ll write by hand.
use clap::Parser;
#[derive(Parser)]
#[command(name = "wiretap", about = "Inspect HTTP traffic from the terminal")]
struct Args {
/// Port to listen on
#[arg(short, long, default_value_t = 8080)]
port: u16,
/// Filter by route pattern
#[arg(short, long)]
filter: Option<String>,
/// Output raw JSON instead of formatted text
#[arg(long)]
json: bool,
}
Users notice startup time. If your tool takes 400ms to print help, it feels broken. Keep the binary small, avoid lazy network calls at startup, and profile the cold path.
The best CLI tools feel instant. That’s not an accident. It’s a design decision.
]]>Hardware accelerated encoding disabled - "crf" option is not
supported with hardware acceleration
The obvious reading: my CRF value is too aggressive, lower it and hardware encoding comes back. I built a task around that assumption and wrote it into project documentation. Both were wrong.
Before spending a day on it, I opened the file that emits the warning.
// @remotion/renderer/dist/get-codec-name.js
if (crf !== null && typeof crf !== 'undefined') {
return 'crf'; // -> disables hardware acceleration
}
It is not the value. Specifying crf at all trips this branch. Any number, including the “safe” one I was about to switch to.
Then, a few lines down:
if (codec === 'h264') {
if (preferred && process.platform === 'darwin' && !unsupportedQualityOption) {
return { encoderName: 'h264_videotoolbox', hardwareAccelerated: true };
}
warnAboutDisabledHardwareAcceleration();
return { encoderName: 'libx264', hardwareAccelerated: false };
}
The only h264 hardware path is Apple VideoToolbox. On Linux and Windows it returns libx264 unconditionally. Dropping crf entirely would silence the warning and change nothing, and my deployment target has no GPU regardless.
Three assumptions, all wrong, all disproved by reading forty lines.
crf 10: exit=0 elapsed=95s size=53,458,834 warning: YES
crf 18: exit=0 elapsed=94s size=27,976,310 warning: YES
The warning stays either way. But the change was still worth making for a reason unrelated to the warning: 1.91x smaller files at identical render time. I kept it, and documented that hardware encoding is unavailable on this platform so nobody chases it again.
Warning text is written by someone describing a condition, not prescribing your fix. It tells you what the code observed. It does not tell you what branch you are in, what platform gates apply, or whether the suggested remedy exists in your environment.
The tools are unglamorous and fast:
grep -rn "hardware acceleration" node_modules/@remotion/renderer/dist/ | head
python -c "import somelib, os; print(os.path.dirname(somelib.__file__))"
Installed source is right there. It is the actual behaviour rather than a description of it, it matches your exact version rather than the docs for a different one, and reading it usually takes less time than one wrong experiment.
I had written the wrong conclusion into a project brief, where it shaped a task list. A confident wrong note in shared documentation costs more than no note, because it stops other people from checking.
The fix was to correct it in place and add a line saying the premise had been tested and disproved — so the next person reads the finding rather than repeating the investigation.
]]>Most pipelines fail in the same way: a change passes CI, lands in production, and breaks something that CI never tested. The fix is not more tests. The fix is a pipeline that assumes failure and routes around it.
1. Every deploy is reversible in one step. If rolling back requires a meeting, your pipeline is broken. Rollback should be a button, not a procedure.
2. Health checks gate the rollout. A deploy that reports success while the app is crash-looping is a liar. The pipeline should verify the service is actually healthy before it calls the job done.
3. Small batches beat big bangs. Ten small deploys a day is safer than one large deploy a week. The blast radius of each change stays small, and the cause of any failure is obvious.
stages:
- test
- build
- deploy
- verify
test:
stage: test
script:
- bundle exec rspec
build:
stage: build
script:
- docker build -t app:$CI_COMMIT_SHORT_SHA .
- docker push registry.example.com/app:$CI_COMMIT_SHORT_SHA
deploy:
stage: deploy
script:
- ./scripts/deploy.sh $CI_COMMIT_SHORT_SHA
environment:
name: production
verify:
stage: verify
script:
- ./scripts/healthcheck.sh || ./scripts/rollback.sh
The verify stage is the part most teams skip. It’s also the part that matters most. If the health check fails, the pipeline rolls back automatically and alerts the team. No human required.
A health check is not a 200 response from /. It’s a check that the service can actually do its job: connect to the database, reach its dependencies, and respond within a latency budget.
#!/usr/bin/env bash
# healthcheck.sh
set -euo pipefail
for i in {1..12}; do
if curl -sf --max-time 5 https://app.example.com/health/deep > /dev/null; then
echo "healthy"
exit 0
fi
sleep 10
done
echo "unhealthy after 2 minutes"
exit 1
The /health/deep endpoint checks the database connection, the cache, and any critical downstream services. A shallow ping tells you the process is alive. A deep check tells you the service is working.
Pipelines are not about automation. They’re about trust. When the team trusts the pipeline, they deploy often, they take smaller risks, and they sleep through the night. When they don’t, they batch changes, delay releases, and get woken up anyway.
Build the pipeline you’d trust at 3am. Then never get woken up by it.
]]>