The incident was a default nobody had revisited.
The client’s platform was fast in the morning and unresponsive by mid-afternoon. The application code was not the problem.
The problem
Fast when idle.
Response times degraded predictably with concurrency. Under load, requests queued at the PHP-FPM pool, which had been sized for a smaller server two migrations earlier.
A queue worker had been down for two days without anyone noticing, so a category of background job had silently stopped running. A nightly cron was failing the same way.
There was no monitoring on queue depth, error rates or slow queries, so every incident was discovered by a customer.
The hard parts
Finding it before rebuilding it.
- The instinct was to rewrite; the evidence pointed at configuration and queries
- Load had to be characterised realistically — average concurrency hides the peaks that matter
- Several slow queries were slow only against production data volume
- Fixes had to land without a maintenance window
- Silent failure had to be made loud, which meant monitoring before optimisation
Architecture
What was changed.
- 01measureReal concurrency, queue depth and slow queries captured before any change
- 02sizePHP-FPM pools resized to the actual machine and workload profile
- 03supervisePM2 and queue workers put under supervision so a crash restarts
- 04cacheRedis introduced for cache, sessions and locks with a deliberate eviction policy
- 05indexSlow queries addressed with indexes and query restructuring
- 06scheduleCron jobs made to report success and failure explicitly
- 07monitorQueue depth, error rates and slow queries surfaced continuously
Stack
- Ubuntu
- Nginx
- PHP-FPM
- PM2
- Redis
- MySQL
- Cron
- Docker
- Horizon
What shipped
What was delivered.
- Server hardening and SSL brought to a maintained standard
- Nginx and PHP-FPM tuned against measured concurrency rather than defaults
- Queue workers and PM2 processes under supervision with automatic restart
- Redis caching for sessions, cache and locks
- Index and query work on the slowest production queries
- Cron jobs that report outcome instead of failing quietly
- Monitoring on queue depth, error rate and slow queries
Result
No rewrite
Configuration and query work only
Headroom without an architectural change.
The afternoon degradation resolved with configuration and query work. No rewrite was required, which is the common outcome when a performance problem is measured before it is diagnosed.
The silent failures were the more serious finding. Supervision and reporting mean a dead worker or failed cron is now visible in minutes rather than days.
This began as an audit and continued as ongoing infrastructure support.
Related
Read next.
Next step