Thanks for an excellent bug report, and doing the research!

Good question, I know memory-limit is affected by the availability of these options, but not sure if they (should) affect reading memory use here.

ping @crosbymichael perhaps you know if this is expected behavior if these options are missing, or is this a bug?

I also get this issue, on my local debian box which currently runs docker 1.9.0. No problem on my Amazon Linux AMI instance (docker 1.8.2) which does not show the following docker info lines, while the debian one does;

WARNING: No memory limit support
WARNING: No swap limit support

Same issue. I looked on the host and noted that the memory stats are not showing in /sys/fs/cgroup/memory.

ls /sys/fs/cgroup/

blkio
cpu
cpuacct
cpu,cpuacct
cpuset
devices
freezer
net_cls
net_cls,net_prio
net_prio
perf_event

Docker Info:

Server Version: 1.9.1
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 391
Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.16.0-4-amd64
Operating System: Debian GNU/Linux 8 (jessie)
CPUs: 1
Total Memory: 494.5 MiB
Name: swarm-node-1
Registry: https://index.docker.io/v1/
WARNING: No memory limit support
WARNING: No swap limit support
Labels:
provider=digitalocean

Yes, it should effect reading memory usage since cgroup memory accounting is not going to work when missing.

Not sure what to do here as printing a warning isn't really going to work since the stats take up the screen... maybe instead of 0/0 we do something like unsupported

In my case, there is a separate issue documenting the enabling of the memory controller for viewing memory stats: #251.

@cpuguy83 Documenting the zero stats may indicate that the proper cgroup manager is likely disabled, and how to manually verify via checking for the directory would be helpful. Probably here: https://docs.docker.com/engine/reference/commandline/stats/ or https://docs.docker.com/engine/articles/runmetrics/. While the article explains the connection to cgroups, it does not specify that docker will print zero if the statistic is not present.

Excuse my ignorance, but is the lack of support due to software or hardware?

Perhaps the memory usage, and the memory % columns just not be present when they are unsupported considering that without support the values are inaccurate, and, the admin may have deliberately chosen not to support that feature for whatever reason.

In my situation, docker stats sometimes reports 0 memory usage for a container and sometimes not, even when the container's state is "not changing" (it is not being restarted or servicing any requests). I see this on Docker 1.8.2 and 1.9.1 on RHEL 7.2.

If I do while true; do docker stats --no-stream $(docker ps -q) | tail -n +2 | awk '{print $3}'; sleep 1; done, I should see no zeroes, but sometimes stats will return 0 for all the containers, sometimes for just a few, more often for none of the containers. There happen to be twenty-five containers running -- a couple PostgreSQL containers and the rest some custom web services.

I have the same issue using Docker 1.10.3 on Debian 3.16.0-4-amd64 x86_64 GNU/Linux. I am also receiving a warning that my kernal does not have correct modules when I try setting memory limit for my containers.

@marc0der I have exactly the same problem with the same kernel and docker version.

In my case I'm running docker over a Xen VM, and I really thought it was that until I saw that all you have the same problem!

It seems like everybody reporting the problem here has the default Debian Jessie install.

Possible fix: http://awhitehatter.me/debian-jessie-wdocker/

I tried it on my instance on Digital Ocean and it worked like a charm!
Thanks!
On Tue, 5 Apr 2016 at 17:39, Petr Pridal notifications@github.com wrote:

It seems like everybody reporting the problem here has the default Debian
Jessie.

Possible fix: http://awhitehatter.me/debian-jessie-wdocker/

—
You are receiving this because you were mentioned.

Reply to this email directly or view it on GitHub
#18420 (comment)

I'm on Ubuntu 14.04 with 4.2.0-18-generic kernel and have mem/swap accounting enabled (docker info doesn't complain). Similar to @murphyke case some containers show statistics, some do not (zeroes only). Those that show have barely correct values, such as 1.5 or 2 Mb, some even few Kbs. I know for sure that it can't be true. What can I do to help troubleshoot this issue?

I tried it on my instance on Digital Ocean and it worked like a charm! Thanks!

yes, it works for me too (Debian Jessie)

@klokan Thanks, this fixes also the deal on my VMWare with debian Jessie.

TL;DR:

Your kernel is probably not configured not to have cgroup activated by default. To turn it on, you need a simple modification to the GRUB_CMDLINE_LINUX_DEFAULT line in /etc/default/grub with the following:

GRUB_CMDLINE_LINUX_DEFAULT="quiet cgroup_enable=memory swapaccount=1"

Then run sudo update-grub and reboot.

This should bring back memory stats.
More info: http://awhitehatter.me/debian-jessie-wdocker/

This should be definitively part of check-config script.

@vaab are you interested in opening a PR to add that?

Hi,
I followed the steps mentioned to resolve the issue but I still can't get the desired result. I followed the steps as here to install docker on Ubuntu 20.04 https://docs.docker.com/engine/install/ubuntu/
I didn't have the grub file present already in /etc/default, therefore I manually created one and added the line:

GRUB_CMDLINE_LINUX_DEFAULT="init=/bin/systemd console=hvc0 console=ttyS0 cgroup_enable=memory swapaccount=1"

Any idea on how to diagnose/fix this issue?

If anyone is looking for this issue on RaspberryPi OS, you will have to edit /boot/cmdline.txt and add cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory to get the memory stats.

If anyone is looking for this issue on RaspberryPi OS, you will have to edit /boot/cmdline.txt and add cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory to get the memory stats.

I am using Ubuntu 20.04(LTS) on Raspberry pi4. In my case, the 'cmdline.txt' file was present in '/boot/firmware/' folder. I followed the solution mentioned above and rebooted the pi. After that docker stats started showing the memory usage for the running containers.

If anyone is looking for this issue on RaspberryPi OS, you will have to edit /boot/cmdline.txt and add cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory to get the memory stats.

Thank you for sharing this. It works on Raspberry Pi 4

Hello, I'm having the same issue... But unfortunately, the GRUB_CMDLINE_LINUX_DEFAULT setting did not fix it. I am running on Ubuntu 20.04 server, not on raspberry pi. The flag is active, as shown by cat /proc/cmdline:

BOOT_IMAGE=/vmlinuz-5.4.0-81-generic root=UUID=******* ro net.ifnames=0 biosdevname=0 nomodeset cgroup_enable=memory swapaccount=1

Does anyone have a suggestion? Maybe things have changed since 2016. Thanks!

If anyone is looking for this issue on RaspberryPi OS, you will have to edit /boot/cmdline.txt and add cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory to get the memory stats.

Thank you for sharing this. It works on Raspberry Pi 4

I'm also on a Raspberry Pi 4 and this did not solve the problem for me. 😢 I still show 0 memory usage and 0 memory limit for every container.

Thank you for sharing this. It works on Raspberry Pi 4

I'm also on a Raspberry Pi 4 and this did not solve the problem for me. 😢 I still show 0 memory usage and 0 memory limit for every container.

What OS are you running? My comment was specific to RaspberryPi OS.

Thank you for sharing this. It works on Raspberry Pi 4

I'm also on a Raspberry Pi 4 and this did not solve the problem for me. 😢 I still show 0 memory usage and 0 memory limit for every container.

What OS are you running? My comment was specific to RaspberryPi OS.

Found the problem -- an unintended line break that looked like a word-wrap in the cmdline.txt file. Got rid of the line break and all's working correctly.

i have the same error
docker stats:

docker version:

 Version:           20.10.14
 API version:       1.41
 Go version:        go1.16.15
 Git commit:        a224086
 Built:             Thu Mar 24 01:48:02 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.14
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.15
  Git commit:       87a90dc
  Built:            Thu Mar 24 01:45:53 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.5.11
  GitCommit:        3df54a852345ae127d1fa3092b95168e4a88e2f8
 runc:
  Version:          1.0.3
  GitCommit:        v1.0.3-0-gf46b6ba
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Context:    default
Debug Mode: false
Plugins:
app: Docker App (Docker Inc., v0.9.1-beta3)
buildx: Docker Buildx (Docker Inc., v0.8.1-docker)
scan: Docker Scan (Docker Inc., v0.17.0)

Server:
Containers: 9
Running: 1
Paused: 0
Stopped: 8
Images: 31
Server Version: 20.10.14
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 3df54a852345ae127d1fa3092b95168e4a88e2f8
runc version: v1.0.3-0-gf46b6ba
init version: de40ad0
Security Options:
apparmor
seccomp
 Profile: default
Kernel Version: 5.17.1-051701-generic
Operating System: Ubuntu 20.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 36
Total Memory: 31.25GiB
Name: xeon
ID: SKJW:OYWJ:QIKA:YH6S:T5DC:UNAC:LD5A:WEME:V7VX:B5ET:UPGQ:SBTT
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

uname -r
5.17.1-051701-generic

GRUB File /etc/default/grub

GRUB_DEFAULT=0
GRUB_TIMEOUT_STYLE=hidden
GRUB_TIMEOUT=0
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet cpuset cgroup_enable=memory cgroup_memory=1 splash swapaccount=1"
GRUB_CMDLINE_LINUX="cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1 swapaccount=1"

i think this happen after i upgrade the kernel version

I have exactly the same symptoms and reconfiguring grub does not seems to help.

Maybe you can recommend some intermediate steps - like what to check and from where, were are my system information:

root@server:/home/futuraura# uname -a
Linux server 5.4.0-109-generic #123-Ubuntu SMP Fri Apr 8 09:10:54 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
root@server:/home/futuraura# docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Docker Buildx (Docker Inc., v0.8.1-docker)
  scan: Docker Scan (Docker Inc., v0.17.0)

Server:
 Containers: 3
  Running: 2
  Paused: 0
  Stopped: 1
 Images: 24
 Server Version: 20.10.14
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3df54a852345ae127d1fa3092b95168e4a88e2f8
 runc version: v1.0.3-0-gf46b6ba
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.4.0-109-generic
 Operating System: Ubuntu 20.04.4 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 7.562GiB
 Name: server
 ID: QDCE:7MLS:M25W:5TT2:AKDL:D76D:H72P:YVRX:JFSK:3ACY:BJUR:TKTB
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

root@server:/home/futuraura# docker version
Client: Docker Engine - Community
 Version:           20.10.14
 API version:       1.41
 Go version:        go1.16.15
 Git commit:        a224086
 Built:             Thu Mar 24 01:48:02 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.14
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.15
  Git commit:       87a90dc
  Built:            Thu Mar 24 01:45:53 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.5.11
  GitCommit:        3df54a852345ae127d1fa3092b95168e4a88e2f8
 runc:
  Version:          1.0.3
  GitCommit:        v1.0.3-0-gf46b6ba
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

If anyone is looking for this issue on RaspberryPi OS, you will have to edit /boot/cmdline.txt and add cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory to get the memory stats.

I am using Ubuntu 20.04(LTS) on Raspberry pi4. In my case, the 'cmdline.txt' file was present in '/boot/firmware/' folder. I followed the solution mentioned above and rebooted the pi. After that docker stats started showing the memory usage for the running containers.

Thank GOD for you. I have looked for hours.