Your crontab is a production system

Ask a team where their production system lives and they’ll point at the API, the database and maybe a queue. Ask where the job that sends invoices lives, and the answer is often a pause, then “on one of the app servers, I think”.

Cron is the most reliable piece of software most of us run, and that’s exactly the problem. It does what it’s told, quietly, for years. When something around it changes, it keeps doing what it’s told, quietly, and nobody notices until a customer asks why they didn’t get their monthly report.

Four ways cron fails without telling you

The job runs on a server that’s gone. Someone resized the fleet, moved the app to a new box and forgot the crontab on the old one. The old server was decommissioned last spring. So was your weekly export.

The job runs twice. A deploy script copies the same crontab to every app server, so the cleanup job that was meant to run once now runs on four machines at the same minute, racing each other.

The job fails into the void. MAILTO sends errors to root@localhost, which forwards them nowhere. Or the output is appended to a log file that logrotate deleted three weeks ago.

The job overlaps itself. The nightly sync usually takes ten minutes. One night the upstream API is slow, the run takes ninety minutes, and the next scheduled run starts on top of it. Now there are two, and they both write to the same table.

None of these are exotic. If you have more than a handful of servers and more than a handful of jobs, at least one of them is probably happening right now.

Treat jobs like services

The fix isn’t to stop using cron syntax; it’s a perfectly good way to describe a schedule. The fix is to give scheduled work the same things we expect from a service:

• One place where jobs are defined, so “what runs where” has a single answer.
• An owner for every job, so a failure has someone to go to.
• A record of every run, with output and exit code, kept long enough to debug last Tuesday.
• Retries with backoff for the failures that are just a network blip.
• Timeouts and an overlap policy, so a slow run can’t pile up.
• Alerts for missed runs, not just failed ones. A job that never started is the quietest failure of all.

Start with an inventory

You don’t need new tooling to take the first step. Run this on each server and paste the output into a document:

$ for u in $(cut -f1 -d: /etc/passwd); do sudo crontab -l -u "$u" 2>/dev/null | sed "s/^/$u: /"; done
$ cat /etc/crontab /etc/cron.d/*

Then, for each line, write down what it does, who would notice if it stopped, and what should happen when it fails. You’ll find jobs nobody remembers adding, jobs for products you no longer sell, and at least one job that’s been failing for months.

That inventory is the hard part. Once you have it, moving the jobs somewhere with history and alerts is mostly typing. (In Fwoop, fwoop jobs import does the typing for you, but the thinking is still yours.)

The point

Your crontab sends invoices, renews certificates, prunes databases and backs things up. That’s production. It should be treated like it.