Day 43:  logrotate – sidecar container to rotate your container logfiles – 7 Days of Docker

Docker log rotation is the automatic process of managing container log files by limiting their size and quantity on the host filesystem. Without log rotation, Docker containers can continuously write logs until disk space is exhausted, potentially causing system failure. By default, Docker uses the json-file logging driver which does not automatically rotate logs, making manual configuration essential for production environments. Log rotation prevents disk space issues by cycling logs at predetermined intervals and removing old log files when storage limits are reached.

Docker container logs are stored in different locations depending on your operating system. On Linux systems, logs are located at /var/lib/docker/containers/[container-id]/[container-id]-json.log. On Windows, they are stored at C:\ProgramData\docker\containers\[container-id]\[container-id]-json.log. For Docker Desktop on macOS, logs are stored within a Docker VM and can be accessed via screen ~/Library/Containers/com.docker.docker/Data/vms/0/tty. Each container has its own dedicated log file named after its full container ID. You can find the exact log path for any container using the command docker inspect –format='{{.LogPath}}’ container-name.

The json-file driver is Docker’s default logging driver that stores logs in JSON format but does not perform log rotation by default. It has significant overhead due to JSON marshaling and unmarshaling, and writes are non-atomic which can cause partial messages. The local logging driver uses a more efficient storage format, performs automatic log rotation by default, and is recommended by Docker for production environments to prevent disk exhaustion. However, both json-file and local drivers support the docker logs command for viewing logs, while other logging drivers may not. The local driver is specifically designed to be more storage-efficient and includes built-in rotation capabilities without additional configuration.

To configure log rotation globally for all new containers, edit or create the daemon.json file located at /etc/docker/daemon.json on Linux or C:\ProgramData\docker\config\daemon.json on Windows. Add the following configuration:

After saving the file, restart the Docker daemon with sudo systemctl restart docker on Linux or by restarting the Docker service on Windows. Note that this configuration only affects newly created containers; existing containers must be recreated to use the new settings.

In Docker Compose, you can specify logging options per service in your docker-compose.yml file using the logging key:

The max-size option sets the maximum file size before rotation (e.g., 10m for 10 megabytes), max-file specifies how many rotated log files to retain, and compress enables gzip compression of rotated logs to save disk space. After modifying the docker-compose.yml file, recreate the containers using docker-compose up -d –force-recreate.

You can configure log rotation when creating a container with the docker run command using log options:

This approach provides flexibility for tuning log handling on a deployment-by-deployment basis without affecting other containers. The settings specified at container creation are immutable and cannot be changed without recreating the container.

Changes to daemon.json only affect newly created containers, not existing ones. This is because container configurations are static from the point of creation and logging driver settings are immutable container properties. To apply new log rotation settings to existing containers, you must stop, remove, and recreate them. Use docker-compose up -d –force-recreate for Compose-managed containers, or manually stop and recreate containers with docker stop, docker rm, and then docker run or docker create with the desired settings. Make sure to backup any important data stored in containers before recreating them, though data in named volumes or bind mounts will be preserved.

Recommended values depend on your application’s logging volume and retention requirements. Common configurations include max-size of 10m to 50m (megabytes) and max-file of 3 to 10 files. For high-traffic applications, consider 50m with 5-7 files for approximately one week of retention. For development environments, 10m with 3 files is typically sufficient. Docker containers using default settings with max-file set to 3 and max-size of 10MB will keep approximately 30MB of logs per container. Calculate total storage by multiplying max-size × max-file × number of containers. Monitor your disk usage and adjust these values based on your actual logging patterns and storage capacity.

You can inspect the logging driver of any container using the docker inspect command:

To see the full logging configuration including all options, use:

You can also check the system-wide default logging driver with docker info | grep “Logging Driver” on Linux or docker info and look for the Logging Driver field.

To identify containers with large log files on Linux, use the following command:

For a more detailed analysis including container names:

This helps identify problematic containers that need immediate attention or log rotation configuration.

Yes, you can manually truncate log files without stopping containers using:

However, this is not recommended for production environments as it’s a temporary solution. The proper approach is to implement automated log rotation through daemon.json or container-specific logging configurations. Manual deletion can also cause issues with external tools that monitor or scrape these log files. Additionally, directly accessing or modifying files in /var/lib/docker/containers/ can interfere with Docker’s logging system and cause unexpected behavior. Always prefer Docker’s built-in rotation mechanisms over manual intervention.

Blocking mode (default) means the application must wait for the Docker logging driver to process and accept each log message before continuing execution, ensuring all logs are delivered but potentially causing latency. Non-blocking mode places log messages in an in-memory ring buffer immediately, allowing the application to continue without delay, but risks losing logs if the buffer fills up. Non-blocking mode is configured with:

Use blocking mode with fast local drivers like json-file or local. Use non-blocking mode with network-based drivers like fluentd, gelf, or awslogs to prevent application performance degradation. The max-buffer-size option controls the ring buffer size, with default of 1MB.

Yes, enabling compression for rotated logs can significantly reduce disk space usage, typically shrinking files to 10-20% of their original size. Enable compression with:

However, be aware that reading compressed log information requires decompression, which causes temporary disk usage increase and increased CPU usage. The docker logs command will automatically decompress rotated files when reading them. For systems with limited CPU resources, you may want to skip compression and instead reduce max-file count or move logs to external storage more quickly.

You can integrate logrotate directly into your container for application-specific log management. Add to your Dockerfile:

Create a crontab file with:

This approach is useful for applications that write logs to files inside the container rather than stdout/stderr, though the recommended practice is to log to stdout/stderr and let Docker handle rotation.

When you restart a container (docker restart), the log file continues to grow in the same file, and the container retains its logging configuration. When you recreate a container (docker rm followed by docker run or docker-compose up –force-recreate), a new container with a new ID is created, resulting in a new log file in a new directory. The old container’s log files remain on disk until you manually delete them or prune old container data with docker system prune. Log rotation settings from the previous container are not automatically carried over unless specified again in the docker run command or docker-compose.yml file. For production environments, use named volumes or external logging solutions to persist important log data across container lifecycle changes.

Docker’s built-in logging drivers (json-file and local) only support size-based rotation, not time-based rotation. To implement time-based rotation, you have several options: use the journald logging driver which supports time-based rotation through systemd configuration, use external logging solutions like Fluentd or Logstash that support time-based policies, or implement a host-level logrotate configuration. For host-level logrotate, create /etc/logrotate.d/docker-containers:

However, be cautious with this approach as external tools modifying Docker log files can interfere with Docker’s logging system. The recommended approach for time-based retention is to use centralized logging systems that can archive and manage logs based on time policies.

Docker supports multiple logging drivers for different use cases: json-file (default) stores logs locally as JSON, suitable for development and debugging; local driver uses efficient storage format with automatic rotation, recommended for production; journald integrates with systemd’s journal, good for systems using systemd; syslog sends logs to syslog server for centralized management; fluentd forwards logs to Fluentd collector for aggregation; gelf sends logs to Graylog Extended Log Format endpoints; awslogs sends logs to Amazon CloudWatch; gcplogs sends to Google Cloud Logging; and splunk sends to Splunk logging server. For production environments without external logging infrastructure, use the local driver. For containerized environments requiring centralized logging, use fluentd or gelf. Only json-file and local drivers support the docker logs command. Choose based on your infrastructure, monitoring strategy, and whether you need local log access or centralized aggregation.

First, verify the logging configuration is applied to your container with docker inspect -f ‘{{.HostConfig.LogConfig}}’ container-name. If the configuration looks correct but rotation isn’t happening, ensure the container was created after the daemon.json changes, as existing containers don’t inherit new settings. Check that you restarted the Docker daemon after modifying daemon.json using sudo systemctl restart docker. Verify that max-size values are formatted correctly with units (e.g., “10m” not 10). For Docker Compose, ensure you recreated containers with docker-compose up -d –force-recreate. Check the actual log file size with docker inspect –format='{{.LogPath}}’ container-name followed by ls -lh on that path. If using Docker installed via Snap, check for daemon.json in /var/snap/docker/current/config/daemon.json instead of /etc/docker/. Look for Docker daemon errors in system logs with journalctl -u docker.service. Common issues include JSON syntax errors in daemon.json, permission problems, and insufficient disk space preventing rotation.

Yes, you can override daemon.json defaults on a per-container basis. The daemon.json settings serve as defaults for containers that don’t specify logging options. To use different settings for specific containers, configure logging in Docker Compose:

Or with docker run commands using –log-opt flags. This flexibility allows you to tune storage usage based on each application’s logging characteristics. High-traffic applications generating many logs can have larger retention, while low-traffic applications can use minimal settings to conserve disk space.

Production Docker log management best practices include: always configure log rotation either globally via daemon.json or per-container to prevent disk exhaustion; use the local logging driver as the default for its efficiency and built-in rotation; implement centralized logging using tools like Fluentd, ELK Stack, or commercial solutions to aggregate logs from multiple containers; add metadata labels and tags to logs for better organization and filtering; configure appropriate max-size and max-file values based on your logging volume and retention requirements; monitor disk space proactively with automated alerts; use non-blocking delivery mode for network-based logging drivers to prevent application latency; enable compression for rotated logs to save disk space; ensure containers log to stdout and stderr rather than internal files; implement proper timestamp handling for accurate log correlation; use structured logging formats like JSON for easier parsing and analysis; regularly test log rotation by simulating high-volume logging; document your logging configuration in infrastructure-as-code; and establish clear log retention policies that comply with regulatory requirements. Consider implementing log sampling for extremely high-volume applications to reduce storage costs while maintaining visibility.

To migrate to the local logging driver, update your daemon.json configuration:

Restart the Docker daemon with sudo systemctl restart docker. For existing containers, you must recreate them to use the new driver. Plan a maintenance window to stop and recreate containers. For Docker Compose applications, use docker-compose down followed by docker-compose up -d. Note that you cannot access old json-file logs through docker logs after switching drivers, so export any important logs before migration using docker logs container-name > container-logs.txt. The local driver uses a different storage format optimized for efficiency, and logs will start accumulating in the new format. Verify the change with docker info | grep “Logging Driver” and docker inspect for individual containers. The local driver performs similarly to json-file but with better storage efficiency and automatic rotation, making it ideal for production workloads.

Calculate storage requirements based on the formula: (max-size × max-file × number of containers) + overhead. For example, with 20 containers, max-size of 10MB, and max-file of 5, you need approximately 1GB (20 × 10MB × 5). Add 20-30% overhead for temporary space during rotation and compression. For production environments, monitor actual usage patterns over time and adjust accordingly. High-traffic applications may generate logs faster than expected, requiring larger allocations. Consider implementing disk space monitoring with alerts when usage exceeds 70-80% of allocated space. In cloud environments, ensure your Docker data volume (/var/lib/docker) has sufficient space separate from the OS volume. For Kubernetes, allocate sufficient node storage for container logs based on pod density. If disk space is limited, reduce max-file count or implement more aggressive log forwarding to external systems. Use docker system df to check current Docker disk usage including logs, and docker system prune to clean up old data. Regular capacity planning reviews should include log storage analysis to prevent unexpected disk full scenarios.

Short-lived containers present unique logging challenges since they may complete before logs are fully processed. For batch jobs and ephemeral containers, use synchronous logging with blocking mode to ensure all logs are captured before container termination. Configure logging to forward to external systems that persist after the container exits. Use labels and tags to identify short-lived container logs for easier tracking:

Implement docker system prune regularly to clean up log files from stopped containers. Consider using a sidecar logging container pattern for critical short-lived jobs to ensure log collection completes. For Kubernetes CronJobs, configure TTL for finished jobs to automatically clean up resources including logs. Use centralized logging solutions with buffering to handle bursts of short-lived containers without losing data. Monitor log ingestion rates to ensure your logging infrastructure can handle container churn. For containers that complete in seconds, ensure your logging driver’s timeout settings are appropriate to capture all output before the container exits.

Log rotation settings impact both I/O performance and application latency. Smaller max-size values cause more frequent rotations, increasing I/O overhead but maintaining smaller individual files. Larger values reduce rotation frequency but risk longer interruptions when rotation occurs. The compression option trades CPU usage for disk space savings—compression typically uses 5-10% additional CPU during rotation but reduces storage by 80-90%. More retained files (higher max-file) increase disk I/O during log reading operations. Blocking delivery mode (default) ensures reliability but may add 1-5ms latency per log line with local drivers, more with network drivers. Non-blocking mode eliminates this latency but requires sufficient buffer size to prevent log loss. The json-file driver has 10-30% more overhead than the local driver due to JSON formatting. For high-performance applications logging thousands of messages per second, use non-blocking mode with the local driver, larger buffer sizes (4-8MB), and disable compression. For typical applications, default settings provide good balance. Always test performance impact in staging environments before applying to production. Monitor application latency, disk I/O wait times, and CPU usage to identify logging bottlenecks.