Summary of some situations when Docker container disk is full

Preface

This article describes two situations I have encountered recently. I will add more if there are new discoveries later.

Reference documentation: https://docs.docker.com/engine/reference/commandline/dockerd/

Too many application logs

By default, each Docker container has 10G of storage space. When this size is exceeded, the container will have problems.

You can refer to the official documentation for the dm.basesize parameter to modify the default container size:

Specifies the size to use when creating the base device, which limits the size of images and containers. The default value is 10G. Note that thin devices are "sparse" in nature, so a mostly empty 10G device will not use 10 GB of space on the pool. However, the file system will use more space for empty boxes the larger the device.

The base device size can be increased on daemon restart, which will allow all future images and containers (based on those new images) to have the new base device size.

example

$ sudo dockerd --storage-opt dm.basesize=50G

This will increase the base device size to 50G. If the existing base device size is larger than 50G, the Docker daemon will throw an error. This option allows the user to expand the base device size, but does not allow shrinking.

This value affects the system-wide "base" empty filesystem that may have been initialized and inherited by pulled images. Typically, changing this value requires additional steps:

$ sudo service docker stop

$ sudo rm -rf /var/lib/docker

$ sudo service docker start

The problem I encountered here was that a certain module would output 1G of logs within 10 minutes. This problem could be solved by modifying the log policy.

Too many Dockerd logs

There is a docker service of GitLab. After running for a few months, it becomes impossible to push and pull. The reason is that the /var/lib/docker disk space is full. The reason why the disk space is full is that the dockerd log takes up more than ten GB of space.

The path of the log file is as follows: /var/lib/docker/containers/CONTAINER_ID/CONTAINER_ID-json.log

Reference: https://stackoverflow.com/questions/31829587/docker-container-logs-taking-all-my-disk-space

You can refer to the following solutions:

1. Start container parameters

Reference: https://docs.docker.com/engine/reference/commandline/run/

Use --log-opt Log driver options to configure the logging policy.

For example: --log-opt max-size=50m.

2. Global default configuration

Configured in daemon.json, the default location in Linux is /etc/docker.

Configuration example:

{
 "log-driver": "json-file",
 "log-opts": {
  "max-size": "10m",
  "max-file": "3",
  "labels": "production_status",
  "env": "os,customer"
 }
}

Log configuration reference: https://docs.docker.com/config/containers/logging/configure/

For a complete daemon.json example, see: https://docs.docker.com/engine/reference/commandline/dockerd/

{
 "authorization-plugins": [],
 "data-root": "",
 "dns": [],
 "dns-opts": [],
 "dns-search": [],
 "exec-opts": [],
 "exec-root": "",
 "experimental": false,
 "features": {},
 "storage-driver": "",
 "storage-opts": [],
 "labels": [],
 "live-restore": true,
 "log-driver": "json-file",
 "log-opts": {
 "max-size": "10m",
 "max-file":"5",
 "labels": "somelabel",
 "env": "os,customer"
 },
 "mtu": 0,
 "pidfile": "",
 "cluster-store": "",
 "cluster-store-opts": {},
 "cluster-advertise": "",
 "max-concurrent-downloads": 3,
 "max-concurrent-uploads": 5,
 "default-shm-size": "64M",
 "shutdown-timeout": 15,
 "debug": true,
 "hosts": [],
 "log-level": "",
 "tls": true,
 "tlsverify": true,
 "tlscacert": "",
 "tlscert": "",
 "tlskey": "",
 "swarm-default-advertise-addr": "",
 "api-cors-header": "",
 "selinux-enabled": false,
 "userns-remap": "",
 "group": "",
 "cgroup-parent": "",
 "default-ulimits": {
 "nofile": {
  "Name": "nofile",
  "Hard": 64000,
  "Soft": 64000
 }
 },
 "init": false,
 "init-path": "/usr/libexec/docker-init",
 "ipv6": false,
 "iptables": false,
 "ip-forward": false,
 "ip-masq": false,
 "userland-proxy": false,
 "userland-proxy-path": "/usr/libexec/docker-proxy",
 "ip": "0.0.0.0",
 "bridge": "",
 "bip": "",
 "fixed-cidr": "",
 "fixed-cidr-v6": "",
 "default-gateway": "",
 "default-gateway-v6": "",
 "icc": false,
 "raw-logs": false,
 "allow-nondistributable-artifacts": [],
 "registry-mirrors": [],
 "seccomp-profile": "",
 "insecure-registries": [],
 "no-new-privileges": false,
 "default-runtime": "runc",
 "oom-score-adjust": -500,
 "node-generic-resources": ["NVIDIA-GPU=UUID1", "NVIDIA-GPU=UUID2"],
 "runtimes": {
 "cc-runtime": {
  "path": "/usr/bin/cc-runtime"
 },
 "custom": {
  "path": "/usr/local/bin/my-runc-replacement",
  "runtimeArgs": [
  "--debug"
  ]
 }
 },
 "default-address-pools":[{"base":"172.80.0.0/16","size":24},
 {"base":"172.90.0.0/16","size":24}]
}

After configuring the parameters, you need to restart the Docker service.

docker-compose configuration

Reference: https://docs.docker.com/compose/compose-file/compose-file-v2/

Configuration example:

logging:
 options:
  max-size: '12m'
  max-file: '5'
 driver:json-file

Replenish

I wrote this in a hurry. The links I posted contain very complete information, which should be able to solve this type of problem.

Summarize

The above is the full content of this article. I hope that the content of this article will have certain reference learning value for your study or work. Thank you for your support of 123WORDPRESS.COM.