# DevOps & Performance

**The incident was a default nobody had revisited.**

- Category: Infrastructure · Performance
- Scope: Nginx · PHP-FPM · queues · Redis
- Engagement: Audit, then ongoing
- Status: In production

Canonical: https://technologiesninja.com/project-devops-performance.html

## The problem

Response times degraded predictably with concurrency. Under load, requests queued at the PHP-FPM pool, which had been sized for a smaller server two migrations earlier.

A queue worker had been down for two days without anyone noticing, so a category of background job had silently stopped running. A nightly cron was failing the same way.

There was no monitoring on queue depth, error rates or slow queries, so every incident was discovered by a customer.

## The hard parts

- The instinct was to rewrite; the evidence pointed at configuration and queries
- Load had to be characterised realistically — average concurrency hides the peaks that matter
- Several slow queries were slow only against production data volume
- Fixes had to land without a maintenance window
- Silent failure had to be made loud, which meant monitoring before optimisation

## Architecture

1. **measure** — Real concurrency, queue depth and slow queries captured before any change
2. **size** — PHP-FPM pools resized to the actual machine and workload profile
3. **supervise** — PM2 and queue workers put under supervision so a crash restarts
4. **cache** — Redis introduced for cache, sessions and locks with a deliberate eviction policy
5. **index** — Slow queries addressed with indexes and query restructuring
6. **schedule** — Cron jobs made to report success and failure explicitly
7. **monitor** — Queue depth, error rates and slow queries surfaced continuously

**Stack:** Ubuntu · Nginx · PHP-FPM · PM2 · Redis · MySQL · Cron · Docker · Horizon

## What shipped

- Server hardening and SSL brought to a maintained standard
- Nginx and PHP-FPM tuned against measured concurrency rather than defaults
- Queue workers and PM2 processes under supervision with automatic restart
- Redis caching for sessions, cache and locks
- Index and query work on the slowest production queries
- Cron jobs that report outcome instead of failing quietly
- Monitoring on queue depth, error rate and slow queries

## Result

**No rewrite** — Headroom without an architectural change.

The afternoon degradation resolved with configuration and query work. No rewrite was required, which is the common outcome when a performance problem is measured before it is diagnosed.

The silent failures were the more serious finding. Supervision and reporting mean a dead worker or failed cron is now visible in minutes rather than days.

This began as an audit and continued as ongoing infrastructure support.

## Read next

- **Google Ads Analytics Dashboard** (Case study) → /project-google-ads-dashboard
- **DevOps & Performance** (Service) → /service-devops-performance
- **Laravel Web Applications** (Service) → /service-laravel-development
- **PHP-FPM Tuning for Laravel Under Load** (Article) → /blog-php-fpm-tuning

---
© 2026 TechnologiesNinja · Built and maintained in-house