Essential Workflow for Software Troubleshooting

1. Before Problems Occur (Preparation Stage)

Skipping preparation can significantly delay resolution and may prevent you from determining the RCA (Root Cause Analysis).
It’s crucial to establish the following as part of routine operations:

• Understand baseline performance under normal conditions (CPU usage, response times, etc.)
• Know where investigation logs are stored
• Set up continuous logging for key metrics (e.g., CPU usage, I/O stats)
• Have a staging/test environment that matches production
• Review common incident scenarios in advance (FAQ or internal knowledge base)

[Poor Preparation]
    ↓
Slower cause identification → Prolonged recovery
    ↓
No RCA determined → Higher chance of recurrence

2. When Problems Occur (Initial Response)

The key to an effective initial response is accurate situation assessment, determining the scope of impact, and preventing escalation.

2.1 Situation Check

• Identify any system changes before and after the issue (deployment, config changes, hardware replacement, etc.)
• Consider rollback if necessary

2.2 Isolation

• Separate potential causes (application, network, infrastructure, database, OS)
• Define clear reproduction steps (exact actions, exact outcomes)
• Record exact error codes, date, messages, and stack traces

2.3 Impact Scope and Priority

• Determine whether the impact is system-wide or limited to specific users
• If high impact, consider limiting or suspending services to prevent escalation

2.4 Temporary Measures

• Apply pre-prepared workarounds
• Follow your escalation path and inform stakeholders immediately

[Issue Detected]
    ↓
Determine scope
    ↓
Isolate cause (App / Network / DB / OS)
    ↓
Apply workaround or rollback

3. After the Incident (Post-Response)

Once services are restored, focus on preventing recurrence and capturing knowledge.

3.1 Reproduction Test

• Confirm whether the same issue can be reproduced in a test environment
• If reproducible, RCA becomes significantly easier

3.2 Cause Analysis & Permanent Fix

• Use log analysis and configuration comparison to find the root cause
• Implement permanent measures (patching, configuration changes, process improvements)

3.3 Knowledge Sharing

• Document the cause, steps taken, and preventive measures
• Register findings in the company’s knowledge base or wiki

3.4 Monitoring Improvements

• If the issue wasn’t detected in time, add or improve monitoring
• Review alert thresholds and monitored items

[Recovery Complete]
    ↓
Reproduction test
    ↓
RCA (Root Cause Analysis)
    ↓
Permanent Fix
    ↓
Knowledge sharing & monitoring improvements

4. Summary

• Skipping preparation delays recovery and may prevent RCA
• In the initial phase, isolation and scope determination are critical
• Post-incident, focus on cause analysis, prevention, and knowledge sharing
• Following this cycle improves both speed and quality of incident handling