Monitor Log File Sizes and logrotate

Monitor Log File Sizes and logrotate

Monitor log file sizes and rotation to track log growth, detect rotation failures, and prevent disk space issues. This guide shows you how to monitor log files and set up automated monitoring.

For comprehensive log monitoring, see Centralized Log Monitoring. For troubleshooting log issues, see Log Files Consuming Disk Space.

Why Monitoring Log Files Matters

Log files can grow indefinitely and consume disk space, causing disk full errors and service failures. Monitoring log file sizes and rotation ensures logs are managed properly and disk space is maintained.

Method 1: Check Log File Sizes

Find Large Log Files

# Find large log files in /var/log
find /var/log -type f -size +100M -exec ls -lh {} \;

# Find largest log files
find /var/log -type f -exec du -h {} + | sort -rh | head -10

# Check specific log file size
ls -lh /var/log/syslog
ls -lh /var/log/nginx/access.log

Monitor Log File Growth

# Monitor log file size over time
while true; do
  SIZE=$(du -h /var/log/syslog | awk '{print $1}')
  echo "$(date): $SIZE"
  sleep 60
done

# Check log file sizes in directory
du -sh /var/log/* | sort -rh

Method 2: Check Log Rotation Status

Check Logrotate Configuration

# View logrotate configuration
cat /etc/logrotate.conf

# Check logrotate status
ls -la /var/lib/logrotate/status

# Test logrotate configuration
sudo logrotate -d /etc/logrotate.conf

# Force logrotate execution
sudo logrotate -f /etc/logrotate.conf

Verify Log Rotation

# Check rotated log files
ls -lh /var/log/*.log.*.gz

# Check log rotation dates
ls -lth /var/log/*.log.* | head -10

# Count rotated log files
find /var/log -name "*.log.*.gz" | wc -l

Method 3: Automated Log File Monitoring with Zuzia.app

Running these checks by hand works once. Zuzia's agent lets you repeat them on every server without logging in to each one.

How it works in Zuzia

• Install the Zuzia agent on the server with one command.
• Zuzia runs a basic security audit on the host and lists what it found. On paid plans, AI summarises the findings in plain language.
• The agent collects system logs, so you can read them in the panel instead of over SSH.
• Save the commands from this guide as a scheduled task and choose how often they run. If a command fails, you get an alert right away by email, Slack, Discord, push (through your Pushover account) or SMS (through your Vonage account). On paid plans, AI can read the output and alert you only when it spots a problem.

Setting it up

1. Add the server in the Zuzia dashboard and install the agent.
2. Open the host's security audit and logs.
3. Create a scheduled task with the check commands below, choose the hosts and frequency, and pick your notification channels.

Best Practices for Log File Monitoring

1. Monitor Log File Sizes Continuously

• Track log file sizes regularly
• Alert when log files exceed thresholds
• Monitor log growth rates
• Prevent disk space issues

2. Verify Log Rotation

• Check log rotation configuration
• Monitor rotation execution
• Verify rotated logs are compressed
• Ensure old logs are deleted

3. Set Appropriate Rotation Policies

• Configure rotation based on log volume
• Set appropriate retention periods
• Compress old logs to save space
• Delete logs after retention period

4. Monitor Log Growth Trends

• Track log growth over time
• Identify log growth patterns
• Plan log storage capacity
• Optimize log generation

Troubleshooting Log File Issues

Step 1: Identify Log Problems

When log files are consuming disk space:

# Find largest log files
find /var/log -type f -exec du -h {} + | sort -rh | head -10

# Check log file sizes
du -sh /var/log/*

# Check disk space
df -h /var/log

Step 2: Resolve Log Issues

Based on investigation:

1. Rotate Log Files:

   # Force log rotation
   sudo logrotate -f /etc/logrotate.conf
   
   # Manually rotate specific log
   sudo logrotate -f /etc/logrotate.d/nginx
   

2. Clean Up Old Logs:

   # Remove old rotated logs
   find /var/log -name "*.log.*.gz" -mtime +30 -delete
   
   # Clean up old logs
   sudo journalctl --vacuum-time=30d
   

3. Optimize Log Configuration:

   • Adjust log rotation frequency
   • Reduce log retention period
   • Optimize log verbosity

FAQ: Common Questions About Log File Monitoring

How often should I check log file sizes?

For production servers, continuous automated monitoring is essential. Zuzia.app can check log file sizes every 30-60 minutes, alerting you when log files exceed thresholds.

What is considered a large log file?

Large log file depends on your disk capacity and log volume. Generally, log files over 100MB should be rotated, files over 500MB require immediate attention, and files over 1GB indicate rotation failures.

How do I prevent log files from consuming disk space?

Prevent log file issues by configuring proper log rotation, setting appropriate retention periods, monitoring log file sizes continuously, and cleaning up old logs regularly.

Can log monitoring impact server performance?

Log monitoring commands have minimal impact on server performance when done correctly. Use appropriate monitoring frequency and avoid monitoring during peak log generation periods.

Related guides, recipes, and problems

• Related guides

  • guides--monitoring-log-files-centralized-analysis.md
  • guides--monitoring-disk.md

• Related recipes

  • recipes--monitor-recent-system-logs-last-minutes.md
  • recipes--monitor-system-log-entries-linux.md
  • recipes--find-large-files-linux-server.md

• Related problems

  • problems--log-files-consuming-disk-space.md
  • problems--disk-space-full-server.md