Performance Baselines on Linux: sar, iostat & Deviations
Performance Baselines on Linux: sar, iostat & Deviations
System performance baselines monitoring is essential for understanding normal system behavior and detecting performance anomalies. This comprehensive guide covers everything you need to know about establishing performance baselines, tracking performance trends, detecting deviations, and setting up automated baseline monitoring on Linux servers.
For related performance topics, see Server Performance Monitoring. For troubleshooting performance issues, see High CPU Usage Server.
Why Performance Baselines Matter
Performance baselines help you understand normal system behavior, detect performance anomalies, track performance trends, plan capacity upgrades, and optimize system performance. Without proper baseline monitoring, performance issues can go undetected, capacity planning becomes guesswork, and optimization efforts lack direction.
Effective baseline monitoring enables you to:
- Detect performance anomalies immediately
- Track performance trends over time
- Plan capacity based on actual usage
- Optimize system performance effectively
- Maintain performance standards
- Respond quickly to performance issues
Understanding Performance Baselines
Before diving into monitoring methods, it's important to understand performance baselines:
Baseline Components
- CPU Usage: Normal CPU utilization patterns
- Memory Usage: Typical memory consumption
- Disk I/O: Normal disk activity levels
- Network Usage: Typical network traffic patterns
Baseline Metrics
- Average Values: Mean performance metrics
- Peak Values: Maximum performance metrics
- Minimum Values: Minimum performance metrics
- Standard Deviation: Performance variability
Method 1: Establish Performance Baselines
Establishing baselines provides reference points for comparison:
Collect Baseline Metrics
# Collect CPU baseline
top -bn1 | grep "Cpu(s)" | awk '{print $2}'
# Collect memory baseline
free -m | awk 'NR==2{printf "%.2f%%", $3*100/$2}'
# Collect disk I/O baseline
iostat -x 1 5 | awk '/Device/ {getline; print $10}'
# Collect network baseline
ifstat -t 1 5 | tail -1
Store Baseline Data
# Save CPU baseline
top -bn1 | grep "Cpu(s)" > /tmp/cpu-baseline.txt
# Save memory baseline
free -m > /tmp/memory-baseline.txt
# Save disk baseline
iostat -x 1 5 > /tmp/disk-baseline.txt
# Save network baseline
ifstat -t 1 5 > /tmp/network-baseline.txt
Calculate Baseline Statistics
# Calculate average CPU usage
sar -u 1 60 | awk '/Average/ {print 100-$8}'
# Calculate average memory usage
sar -r 1 60 | awk '/Average/ {print $4}'
# Calculate average disk I/O
sar -d 1 60 | awk '/Average/ {print $10}'
# Calculate average network usage
sar -n DEV 1 60 | awk '/Average/ {print $5, $6}'
Method 2: Monitor Performance Trends
Monitoring trends helps identify performance changes:
Track Performance Over Time
# Monitor CPU trends
sar -u 1 10
# Monitor memory trends
sar -r 1 10
# Monitor disk I/O trends
sar -d 1 10
# Monitor network trends
sar -n DEV 1 10
Compare Current vs Baseline
# Compare CPU usage
current_cpu=$(top -bn1 | grep "Cpu(s)" | awk '{print $2}')
baseline_cpu=$(cat /tmp/cpu-baseline.txt | awk '{print $2}')
echo "Current: $current_cpu, Baseline: $baseline_cpu"
# Compare memory usage
current_mem=$(free -m | awk 'NR==2{printf "%.2f", $3*100/$2}')
baseline_mem=$(cat /tmp/memory-baseline.txt | awk 'NR==2{printf "%.2f", $3*100/$2}')
echo "Current: $current_mem, Baseline: $baseline_mem"
Method 3: Detect Performance Deviations
Detecting deviations helps identify performance issues:
Identify Performance Anomalies
# Check for CPU spikes
sar -u 1 60 | awk '$3 > 80 {print "High CPU:", $3}'
# Check for memory pressure
free -m | awk 'NR==2{if ($3/$2 > 0.9) print "High memory usage"}'
# Check for disk I/O issues
iostat -x 1 5 | awk '$10 > 80 {print "High disk I/O:", $10}'
# Check for network congestion
ifstat -t 1 5 | awk '$2 > 1000000 {print "High network usage"}'
Calculate Deviation Thresholds
# Calculate CPU deviation
current_cpu=$(sar -u 1 1 | awk '/Average/ {print 100-$8}')
baseline_cpu=50
deviation=$((current_cpu - baseline_cpu))
if [ $deviation -gt 20 ]; then
echo "CPU deviation detected: $deviation%"
fi
Method 4: Analyze Performance Patterns
Analyzing patterns helps understand performance behavior:
Identify Performance Patterns
# Analyze CPU patterns by hour
sar -u | awk '{print $1, $3}' | sort | uniq -c
# Analyze memory patterns
sar -r | awk '{print $1, $4}' | sort | uniq -c
# Analyze disk I/O patterns
sar -d | awk '{print $1, $10}' | sort | uniq -c
# Analyze network patterns
sar -n DEV | awk '{print $1, $5, $6}' | sort | uniq -c
Detect Seasonal Patterns
# Analyze daily patterns
sar -u | awk '{print $1}' | sort | uniq -c
# Analyze weekly patterns
sar -u | awk '{print $1, $2}' | sort | uniq -c
# Identify peak usage times
sar -u | awk '{if ($3 > 80) print $1, $2, $3}' | sort | uniq -c
Method 5: Automated Baseline Monitoring with Zuzia.app
Manual checks work while you are looking. To keep running them without you, Zuzia uses its agent and scheduled tasks.
How it works in Zuzia
- Install the Zuzia agent on the server with one command. CPU, RAM, disk, load and network metrics start reporting on their own, with alerts on a threshold, on drift from the server's normal, or on a disk fill-up forecast.
- Save the commands from this guide as a scheduled task and choose how often it runs, from every minute to once a year. The agent runs it on every host you pick.
- If a command fails, you get an alert right away by email, Slack, Discord, push (through your Pushover account) or SMS (through your Vonage account).
- The output of every run is kept, so you can look back at what changed. On paid plans, AI can read the output and alert you only when it spots a problem.
Setting it up
- Add the server in the Zuzia dashboard and install the agent.
- Create a scheduled task with the check commands below, then choose the hosts and how often it runs.
- Choose your notification channels. Turn on maintenance mode during planned work so it doesn't trigger alerts.
Custom Baseline Monitoring Commands
Add these commands as scheduled tasks for comprehensive baseline monitoring:
# Collect performance metrics
sar -u 1 1 | awk '/Average/ {print 100-$8}'
# Compare with baseline
current=$(sar -u 1 1 | awk '/Average/ {print 100-$8}'); baseline=50; echo "Current: $current, Baseline: $baseline"
# Detect deviations
sar -u 1 60 | awk '$3 > 80 {print "High CPU:", $3}'
# Analyze trends
sar -u | tail -20
Best Practices for Baseline Monitoring
1. Establish Baselines During Normal Operations
Collect baseline data during normal operations:
- Monitor performance during typical workload
- Collect baseline data over multiple days
- Establish baselines for different time periods
- Update baselines when workload changes
2. Monitor Baselines Continuously
Don't set and forget baselines:
- Use Zuzia.app for continuous baseline monitoring
- Compare current performance with baselines regularly
- Update baselines when workload patterns change
- Review baseline effectiveness periodically
3. Set Appropriate Deviation Thresholds
Configure thresholds based on your environment:
- Warning: 20% deviation from baseline
- Critical: 50% deviation from baseline
- Emergency: 100% deviation from baseline
Adjust thresholds based on your server's performance characteristics.
4. Track Performance Trends
Monitor performance over time:
- Track performance trends daily
- Identify seasonal patterns
- Plan capacity based on trends
- On paid plans, let AI read the task output and alert you only when it spots a problem
5. Respond Quickly to Deviations
Have response procedures ready:
- Define escalation procedures for performance deviations
- Prepare performance optimization procedures
- Test performance recovery procedures regularly
- Document performance incident responses
Troubleshooting Performance Issues
Step 1: Identify Performance Deviations
When performance deviations occur:
Compare with Baseline:
- Check current performance metrics
- Compare with established baselines
- Calculate deviation percentage
Investigate Deviations:
- Review performance trends
- Check for recent changes
- Identify root causes
Step 2: Analyze Performance Patterns
When performance issues are detected:
Review Performance Data:
- Analyze performance trends
- Identify performance patterns
- Check for seasonal variations
Investigate Root Causes:
- Review system logs
- Check application activity
- Verify system resources
Step 3: Optimize Performance
When performance needs optimization:
Immediate Actions:
- Optimize resource usage
- Scale infrastructure if needed
- Fix performance bottlenecks
Long-Term Solutions:
- Update performance baselines
- Improve monitoring
- Plan capacity upgrades
FAQ: Common Questions About Baseline Monitoring
How often should I update performance baselines on my Linux server?
Update baselines when workload patterns change significantly, after major system changes, or quarterly. Zuzia.app can track performance continuously and automatically adjust baselines based on trends. Review baseline effectiveness regularly.
What performance metrics should I baseline?
Baseline CPU usage, memory usage, disk I/O, network usage, and application-specific metrics. Focus on metrics that affect system performance and user experience.
Can Zuzia.app detect performance deviations automatically?
Yes, Zuzia.app can detect performance deviations by comparing current performance with baselines, calculating deviation percentages, tracking performance trends, and alerting when deviations exceed thresholds. Use commands that compare current metrics with baselines.
How do I respond to performance deviation alerts?
When performance deviation alerts occur, immediately check current performance, compare with baselines, investigate root causes, optimize performance if needed, and update baselines if workload patterns changed. Document all performance incidents for future reference.
Should I monitor baselines on all servers?
Yes, monitor baselines on all production servers. Performance issues can occur on any server, and comprehensive baseline monitoring helps maintain performance standards across your entire infrastructure.
Related guides, recipes, and problems
Related guides
- Server Performance Monitoring
- Monitoring System Performance Baselines
- Server Performance Optimization
Related recipes
Related problems