Late in the night, I received a phone call from my colleague said that our SQL server is unresponsive. I thought it was strange because our SQL server was running in a docker container, which should restart in case of an exception. After logging in the server, I noticed the disk was full, but the docker told me that the database volume only used hundreds of megabytes. So I tried to find the culprit by sorting the big files:
I found a log file in the
/var/lib/docker taking up over 50 GB of space!
The space would be released when docker daemon restarts.
Edit (create if not exists)
More info in the document: Configure the default logging driver
IMPORTANT NOTICE: change to
daemon.json only applies to the newly created containers
For docker compose deployed service, executing the following commands: