A render farm that needs a human at 3am isn't a farm, it's a hobby. Here's the smallest autopilot I'd trust with an office of Macs and a shot list.
Rule 1: never lose work you can requeue
Workers touch a heartbeat next to the job they claimed. If it goes quiet past a threshold, the job goes back to the queue — the same rule the manual reaper used, just without the human:
let age = if hb.exists() { mtime_age(&hb) } else { mtime_age(&job) };
if age < stale_min * 60 { continue } // still alive, leave it alone
std::fs::rename(&job, queue.join(format!("REQUEUED_{}.job", orig)))?;Rule 2: a memory kill is not a normal failure
Retrying an OOM at the same size on the same machines just dies again — that's a loop, not a retry. So rc=137 branches: ask for a bigger Mac if the farm has one, otherwise re-queue the job smaller.
if is_memory_rc(&card.rc) {
let big = biggest_ram_gb(root);
if big > card.min_ram_gb { job.min_ram_gb = big; } // wait for the 64GB box
else { job.width = card.width * 2 / 3; job.perf = "light".into(); }
}Why memory gets its own path is the whole of the MLX memory story.
Rule 3: stop, don't thrash
The rule I'd fight for: after N failures in a row, pause the entire queue. A broken model path or a full disk will otherwise burn six hours failing two hundred times, and you'll wake up to a folder of nothing.
Pausing moves waiting jobs into queue/hold/ — the workers' glob can't see it, in-flight renders finish normally, and nothing is deleted. Resuming is a rename back.
Rule 4: exactly one babysitter
Every Mac runs the same app, so two of them would requeue the same failure twice and double the night. A heartbeat lock file on the share settles it:
pub fn claim_supervisor(root: &str, host: &str, now: u64) -> bool {
// someone else holds it and is still alive? stay out of the way
if held_by_other(&p, host, 120) { return false }
std::fs::write(&p, json!({ "host": host, "ts": now }).to_string()).is_ok()
}The bit that makes it trustworthy
Everything autopilot does goes into logs/autopilot.log on the share, and it only ever requeues — it never deletes a job and never touches a file a worker holds. In the morning you get a notification with the tally and a report: what landed, what failed as cards you can act on, who rendered what, and what's already approved.
Plan the night by pasting a shot list — prompts × sizes × takes, priced before you commit it. Choose proofs instead of heroes and you wake up to a contact sheet to cherry-pick from, which is the cheap way to spend a farm.
The verdict
Autonomy earns trust by being boring and reversible. Requeue, retry once, back off memory, hard-stop on a streak, write it all down — that's it. No ML-driven scheduler, no self-healing anything. Ship the pause button first; the clever bits can wait.