We had to clean up some left over cgroups after another set of
experiments with LXC.The guys doing it encountered problems, as the
logic is the opposite of what you expect from your experience on a
normal unix filesystem.
Specifically the problem happens when you have a nested cgroup – for example /cgroup/foo/bar/
The problem happens because this is a virtual filesystem with its own rules.
If you create a cgroup (in this case we’ll just mount the net_cls cgroup because its smaller)
# mount -t cgroup /dev/cgroup /cgroup -o net_cls # cd /cgroup # ls cgroup.procs net_cls.classid notify_on_release release_agent tasks # mkdir foo
# ls
cgroup.procs foo net_cls.classid notify_on_release release_agent tasks
# ls foo
cgroup.procs net_cls.classid notify_on_release tasks
There are a set of files created to manipulate the cgroup. If the tasks entry is empty, then the cgroup is unused.
However the files can’t be removed.
# rm -f *
rm: cannot remove `cgroup.procs': Operation not permitted
rm: cannot remove `foo': Is a directory
rm: cannot remove `net_cls.classid': Operation not permitted
rm: cannot remove `notify_on_release': Operation not permitted
rm: cannot remove `release_agent': Operation not permitted
rm: cannot remove `tasks': Operation not permitted
So here’s where the confusion arose. To remove a directory hierarchy
on a normal file system you should first remove all files from each
subdirectory and then remove the parent directories.
This is what “rm -fr directory” will usually do for you.
However, as the files cannot be removed, that approach will fail in
this virtual filesystem. “rm -rf” is the wrong approach.
Instead, you need to remove just the cgroup directories. If you want
to do this recursively here’s the magic incantation (we first make a
sub-cgroup to give it more than one level of depth);
# mkdir -p foo/bar
# ls foo/
bar/ cgroup.procs net_cls.classid notify_on_release tasks
# find foo -depth -type d -print -exec rmdir {} ;
foo/bar
foo
# ls
cgroup.procs net_cls.classid notify_on_release release_agent tasks
As the files aren’t real, they don’t need to be removed – we
only need to remove the cgroup directories. The “-depth” argument tells
find to do a depth first recursion, so we remove sub cgroups
first.
If that fails, then you’ll most likely have a task occupying one of the cgroups somewhere.
You can find it like this;
# mkdir -p foo/bar
# echo $$ > foo/bar/tasks
# find foo -depth -type d -print -exec cat {}/tasks ;
foo/bar
21413
21956
21957
foo
And to remove those tasks from the subcgroup add them into the top level.
# echo $$ > tasks
# find foo -depth -type d -print -exec cat {}/tasks ;
foo/bar
foo
#
The last gotcha with cgroups relates to mounting subsets of functionality.
If you have created a cgroup, and then attempt to mount or
remount /dev/cgroup with different cgroup subsystems active, the mount
will fail with a rather unhelpful error message “mount: /dev/cgroup
already mounted or /cgroup busy”
The answer is to remove the cgroups you have created, so that
there are no cgroups on the system, and then you can mount with
different subsystems active.
As in the following example;
# umount /cgroup
# mount -t cgroup /dev/cgroup /cgroup -o net_cls,cpuacct
mount: /dev/cgroup already mounted or /cgroup busy
# mount -t cgroup /dev/cgroup /cgroup -o net_cls
# find /cgroup/foo -depth -type d -print -exec rmdir {} ;
/cgroup/foo/bar
/cgroup/foo
# umount /cgroup
# mount -t cgroup /dev/cgroup /cgroup -o net_cls,cpuacct
# ls /cgroup
cgroup.procs cpuacct.usage net_cls.classid release_agent
cpuacct.stat cpuacct.usage_percpu notify_on_release tasks
Which is probably why fedora mounts them all separately. Even though that looks messy.