Set Logging Limit for Docker Jun 1st 2021 Words: 235

Background

Late in the night, I received a phone call from my colleague said that our SQL server is unresponsive. I thought it was strange because our SQL server was running in a docker container, which should restart in case of an exception. After logging in the server, I noticed the disk was full, but the docker told me that the database volume only used hundreds of megabytes. So I tried to find the culprit by sorting the big files:

1
sudo find / -type d -size +100M | sort -n -r | head -n 20

I found a log file in the /var/lib/docker taking up over 50 GB of space!

Solution

Clean the log file

1
sudo find /var/lib/docker/ -type f -name "*.log" -delete

The space would be released when docker daemon restarts.

Set limit for the log

Edit (create if not exists) /etc/docker/daemon.json:

1
2
3
4
5
6
7
8
9
{
    "log-driver": "local",
    "log-opts": {
        "cache-max-size": "10m",
        "cache-max-file": "3",
        "max-size": "10m",
        "max-file": "3"
    }
}

More info in the document: Configure the default logging driver

IMPORTANT NOTICE: change to daemon.json only applies to the newly created containers

For docker compose deployed service, executing the following commands:

1
2
3
4
docker-compose down
sudo systemctl daemon-reload
sudo systemctl restart docker
docker-compose up -d