When Production Breaks at 2AM: My Incident Response Process

It happens to everyone eventually. Sentry fires at 2AM. Slack from the client: "the site is down." You have been asleep for 40 minutes. Your brain is not fully online.

This is the exact process I follow. Having a process is the difference between resolving it in 15 minutes and spending three hours while making it worse.

Step 1: Understand the Blast Radius Before Touching Anything

Five questions that take two minutes and determine everything else:

1. Is the entire app down or only specific features?
2. Is it affecting all users or a subset?
3. When did it start? (Sentry's "first seen" timestamp)
4. Did anything deploy in the last 12 hours?
5. Did anything external change? (Third-party API, DNS, database hosting, CDN)

Step 2: Communicate Before You Have the Fix

"Investigating an issue affecting X, will update in 15 minutes" buys you time and builds trust. Radio silence for an hour while debugging is worse than not knowing the cause. If you use a status page, post an incident immediately. Users are significantly more forgiving when they can see you are actively working on it.

Step 3: Roll Back First, Debug Second

If something deployed in the last few hours and something is now broken, roll back:

# Docker Swarm
docker service update --rollback app_name

# Deployer / Capistrano
dep rollback production

# Manual with git
git revert HEAD~2..HEAD
git push origin main && deploy

Rolling back takes 2-5 minutes and restores service immediately. You can then debug in staging with proper time and tools. Exception: if the deployment included database migrations that cannot be reversed cleanly. In that case, fix-forward.

Step 4: Look at the Right Logs

In order of usefulness:

1. Sentry — exact exception, stack trace, affected users
2. Application logs — recent errors
3. Nginx access logs — which URLs are returning 500s
4. PHP-FPM logs — worker crashes, timeout errors
5. MySQL slow query log — if database-related

# 5xx errors in the last 5 minutes
sudo tail -f /var/log/nginx/access.log | grep " 5[0-9][0-9] "

# Application errors
sudo tail -500 /var/www/app/storage/logs/laravel.log | grep "ERROR\|CRITICAL"

# PHP-FPM errors
sudo journalctl -u php8.4-fpm --since "10 minutes ago"

Step 5: Quick Production Health Checks

php artisan about              # Config cache status, drivers
php artisan queue:monitor      # Is the queue backed up?
mysql -e "SHOW PROCESSLIST"    # Locked or long-running queries

Step 6: Blameless Post-Mortem

Do it within 24 hours while memory is fresh. Answer:

• What happened (timeline)
• Why it happened (root cause — not "someone deployed broken code")
• How we detected it — and how we could detect it faster next time
• What we are changing to prevent recurrence

"The developer deployed broken code" is not a root cause. "We have no automated integration tests and our rollback process is manual" is a root cause. The goal is to fix systems, not punish people.

Prevention Is Cheaper Than Firefighting

• Feature flags — deploy code before enabling it
• Canary deployments — roll out to 5% of users first
• Load testing before high-traffic events
• Sentry alerts with sensible thresholds (not "every error")
• Automated rollbacks triggered by error rate spikes