Laravel Horizon in Production: Queues That Fail Loudly

Horizon makes Laravel queues pleasant to run — a dashboard, sane defaults, tidy configuration. It also makes it easy to assume everything is fine because the dashboard is green. The failures that actually hurt are the ones Horizon shows you only if you configured it to look.

The silent-death problem

A worker crashes, or the box reboots, or a deploy does not restart the process. Requests still succeed — they just enqueue jobs that no one runs. Emails do not send, webhooks do not fire, exports never finish. There is no error page, so nothing alerts. You find out when a customer does. Every other decision here exists to prevent that.

Uptime of the worker process is not the metric. Queue wait time is. A running worker that is hopelessly behind is still an outage.

Supervisor keeps workers alive; you keep Supervisor honest

Horizon relies on a process supervisor to restart it. That is necessary, not sufficient — Supervisor will happily restart a worker into a crash loop and report it as running. Configure it to restart on exit, but also monitor the restart count and the actual throughput, or a flapping worker looks healthy from a distance.

[program:horizon]
command=php /app/artisan horizon
autostart=true
autorestart=true
stopwaitsecs=3600          ; let long jobs finish on deploy
user=www-data
redirect_stderr=true
stdout_logfile=/var/log/horizon.log

Memory and process sizing

Long-running PHP workers accumulate memory. Set a per-worker memory limit and let Horizon recycle a worker after N jobs, so a slow leak never grows into an OOM kill mid-job. Size the number of processes against the machine the same way you would PHP-FPM — arithmetic, not a copied default — and give different queues different worker pools so a flood of low-priority jobs cannot starve the ones a user is waiting on.

• Set memory per worker and recycle after a bounded job count
• Separate queues by priority; give the critical queue its own workers
• Set timeout below retry_after or a job can run twice at once
• Make jobs idempotent so a retry is safe, because retries will happen

Failed jobs are a workflow, not a log line

By default a failed job lands in a table and is forgotten. Treat that table as a work queue: alert when it grows, keep enough context on each entry to understand the failure, and make retrying deliberate and safe rather than a blind bulk button. A failed payment-reconciliation job you can replay after fixing the cause is recoverable; one you only discover in a log three days later is a data problem.

Alert on the thing that matters

Wire alerts to queue wait time and failed-job rate, not just “is the process up.” Horizon exposes wait time per queue — page someone when the critical queue’s oldest job crosses a threshold. That single alert catches the silent-death case, the crash-loop case, and the fell-behind-under-load case, because all three show up as jobs waiting too long.

What good looks like

A production Horizon setup we are happy with: Supervisor restarts workers, workers recycle on a memory and job-count budget, critical work has its own pool, every job is idempotent, the failed-jobs table is monitored and replayable, and an alert fires on wait time before any human notices. None of it is exotic. It is the difference between a queue you trust and one you keep discovering.