Some Zombies just can't be killed - Zombie Processes

We've been told by so many movies and TV series that no matter how smelly or ugly Zombies can be, you can easily kill them by either blowing up their heads with a shotgun or gently detaching it from the rest of the body (this doesn't really kill the zombie but renders it almost harmless, fun fact, head can't run while detached from the legs). Unfortunately Zombie processes don't really follow the previous "Zombie expected" behavior and they just can't be killed (kill -9 PID).

What in the world is a Zombie process?

Wikipedia has a pretty nice description here, but essentially we are talking about a process that has finished its execution through either exit or exit_group syscalls but remains present in the system due to some unfinished housekeeping tasks. You can easily identify a zombie process from its Z state in for example top or ps. A nice description can be found here the wait man page:

       A child that terminates, but has not been waited for becomes a
       "zombie".  The kernel maintains a minimal set of information about
       the zombie process (PID, termination status, resource usage
       information) in order to allow the parent to later perform a wait to
       obtain information about the child.  As long as a zombie is not
       removed from the system via a wait, it will consume a slot in the
       kernel process table, and if this table fills, it will not be
       possible to create further processes.  If a parent process
       terminates, then its "zombie" children (if any) are adopted by
       init(1), (or by the nearest "subreaper" process as defined through
       the use of the prctl(2) PR_SET_CHILD_SUBREAPER operation); init(1)
       automatically performs a wait to remove the zombies.

For some reason there's a lot of confusion around this Z state, for example:

• Orphan processes aren't zombie processes, in fact for a process to become zombie its parent process has to be still alive. If the parent dies it will be adopted by init and waitED properly.
•  A Zombie process isn't consuming memory (or CPU), to be more precise, isn't consuming as much memory as it was consuming while it was running. However it still occupies a slot in the process table and this could be a problem.

 What happens when exit or exit_group are called?

As I mentioned before these are the syscalls used to terminate a process gracefully. You can find the details of these two for kernel 4.14 here and here. I'm not going to describe the whole code because first I don't understand all of it and second I would probably do it wrong :), but... I'm going to point out some interesting parts:

• Things begin in do_group_exit which takes care of killing all the threads in the current process thread group then it calls do_exit to finish the process itself.
• do_exit takes care of a few things as well, for example releasing some of the resources associated to the process like files and shared memory (lines 858-866). Later on in line 885, calls to exit_notify, which is in charge of sending to the relative processes the bad news, kind of "Hey!, I'm dying" thing. Now within this function a few things will happen being the most important ones the following:
• do_notify_parent will be called to signal the parent process with a SIGCHLD signal. This function will return either true or false depending on whether the dying process should become a Zombie or just go regular dead.
• Have a look at the comment in here, long story short, IF the father decides to ignore (by setting the handler to SIG_IGN) the SIGCHLD signal (or SA_NOCLDWAIT flag is set) the dying process can reap itself and doesn't become a zombie and in that case do_notify_parent will return true. Otherwise it will return false.
• Back to exit_notify, you can see in here how the process decides whether or not to become a zombie depending on the autoreap variable.

Something worth mentioning is the fact that Zombie is an exit state (tsk ->exit_state) while the scheduling state of the process is actually Dead, you can see this is defined by calling here do_task_dead function. So a Zombie process isn't consuming CPU resources at all, for the scheduling perspective this process is gone, and so it is all the other resources that had been allocated to it.

Why can't it be killed?

Well, you can still send SIGKILL signal to a Zombie process and as you may know, you can't escape from that signal. However technically speaking even though the process is still in the process table, there's nothing it can do, there's no code attached to it anymore, no stack, no files, nothing.

Example

Lets have a look at all this zombie stuff with a simple example. You can get the code here, it's a simple C program that will fork 2 child processes and allocate some memory in each. The parent will sleep for 30 seconds, while the child processes will wait for 15 seconds, hence dying while the father is still running (sleeping) becoming spoooky zombiesssss:

juan@test:~$gcc -o zombie_maker zombie_maker.c
juan@test:~$ ./zombie_maker &
juan@test:~$ Parent PID = 2050, PGRP = 2050, PPID = 1934, PSID = 1934
Child 0 -> PID = 2051, PGRP = 2050, PPID = 2050, PSID = 1934
Child 1 -> PID = 2052, PGRP = 2050, PPID = 2050, PSID = 1934

juan@test:~$ ps aux|grep zombie
juan      2050  0.0  0.0   4216   732 pts/1    S    21:34   0:00 ./zombie_maker
juan      2051  0.0  0.0   6268    92 pts/1    S    21:34   0:00 ./zombie_maker
juan      2052  0.0  0.0   6268    92 pts/1    S    21:34   0:00 ./zombie_maker
juan      2054  0.0  0.0  15960  2268 pts/1    R+   21:34   0:00 grep --color=auto zombie
juan@test:~$ ps aux|grep zombie
juan      2050  0.0  0.0   4216   732 pts/1    S    21:34   0:00 ./zombie_maker
juan      2051  0.0  0.0      0     0 pts/1    Z    21:34   0:00 [zombie_maker] 
juan      2052  0.0  0.0      0     0 pts/1    Z    21:34   0:00 [zombie_maker] 
juan      2056  0.0  0.0  15960  2268 pts/1    S+   21:34   0:00 grep --color=auto zombie
juan@test:~$ ps aux|grep zombie
juan      2058  0.0  0.0  15960  2288 pts/1    S+   21:35   0:00 grep --color=auto zombie
[2]+  Done                    ./zombie_maker
juan@test:~$

After the 15 second, we can see how both child processes not only became Zombies but they also show no memory footprint whatsoever on the system (VSZ is 0 and RSS is 0 as well). After the 30 seconds we can see that none of the 3 processes exist anymore, but who reaped the Zombie children from the system???

Yeahp, init did! Attaching to init with strace we can see the waitid calls returning with the details from the Zombie processes:

root@test:/home/juan# strace -p 1
Process 1 attached
select(54, [3 5 6 7 10 11 15 16 17 19 20 24 26 30 34 53], [], [7 10 11 15 16 17], NULL) = ? ERESTARTNOHAND (To be restarted if no handler)
...
waitid(P_ALL, 0, {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=2051, si_status=3, si_utime=0, si_stime=0}, WNOHANG|WEXITED|WSTOPPED|WCONTINUED, NULL) = 0
waitid(P_ALL, 0, {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=2052, si_status=3, si_utime=0, si_stime=0}, WNOHANG|WEXITED|WSTOPPED|WCONTINUED, NULL) = 0
select(54, [3 5 6 7 10 11 15 16 17 19 20 24 26 30 34 53], [], [7 10 11 15 16 17], NULL^CProcess 1 detached
 
root@test:/home/juan#

Running zombie_maker with strace will show you how the parent process gets indeed interrupted by the SIGCHLD signals but since the handler isn't defined it is ignored. PLEASE, don't confuse that ignore with the one that happens when the handler is set to SIG_IGN, literally this is what the documentation says:

       POSIX.1-2001 specifies that if the disposition of SIGCHLD is set to
       SIG_IGN or the SA_NOCLDWAIT flag is set for SIGCHLD (see
       sigaction(2)), then children that terminate do not become zombies and
       a call to wait() or waitpid() will block until all children have
       terminated, and then fail with errno set to ECHILD.  (The original
       POSIX standard left the behavior of setting SIGCHLD to SIG_IGN
       unspecified.  Note that even though the default disposition of
       SIGCHLD is "ignore", explicitly setting the disposition to SIG_IGN
       results in different treatment of zombie process children.)

Summary

No matter how brave or root you are, you can't kill a Zombie process :D, killing its parent is usually the way to get rid of them. When the parent dies, init will adopt the little zombie bastards and reap them off the process table, as we saw on the example. Also as mentioned, a Zombie process doesn't consume much system resources (actually almost none), so unless you have hundreds of them you shouldn't worry much.

The kernel code is going crazy, it would be nice if we had more comments in place :P.

Process mating season 101 - Fork and Clone

Yeah, I know, the title is awesome, isn't it? Come on! at least it should be if you understand what fork() and clone() do in the context of Linux syscalls.

These two calls are the ones in charge of creating life on your system, basically spawning new processes. We'll have a look at the fundamental differences, some examples to understand them and some interesting facts

That said, before moving forward I'd like to outline the following:

• Syscalls are the interface the kernel exposes to the user space processes, or if you want, they are the way processes access to the kernel services.
• Usually processes don't use syscalls directly but they use them through some wrapper functions provided by glibc (in GNU/Linux at least). This wrapper functions provide sort of an abstraction layer, handling syscall parameters, return values and other situations.

Note: btw I've already talked a bit about syscalls here maybe you want to have a look.

Fork, the processes mitosis process

[off topic] I'm getting extremely good at writing titles![/off topic]

Pretty much like the eukaryotic cells replication process, the result of a fork call is a new almost identical process known as child process. The "almost" is key here, there are some properties that will be different, like (for an exhaustive list, please have a look at this):

• PID will be different, the kernel will assign a new/unused PID to the child process. The parent PID of the child process will be its parent's PID (kind of makes sense, doesn't it?)
• The child process doesn't inherit timers or memory locks.
• Others

On the other side of the "almost" we have:

• The child process will have an exact copy of its parent entire virtual address space (fork() is implemented using copy-on-write pages, so the only penalty that it incurs is the time and memory required to duplicate the parent's page tables, and to create a unique task structure for the child.). 
• The child process inherits copies of structures like open file descriptors, open directory streams, etc.

Right after the fork() call the processes will be, although they are still sharing some resources,  two different entities, and they could be running two different code paths. This will be easier to understand with a simple example.

Simple fork() example

The code is self explanatory (or at least I tried) so I won't explain it in details

#include <stdio.h>
#include <sys/types.h>
#include <unistd.h>
#include <sys/syscall.h>
#include <errno.h>
#include <string.h>

int main()
{
        int pid,ppid,childpid,keep_it;
        pid=getpid();//Get process ID
        ppid=getppid();//Get parent process ID

        childpid=fork();//fork() returns the child PID on the parent's code path and 0 on the child's. On failure returns -1
        //From this point on, there are 2 processes running, unless fork failed of course :D.
        keep_it=errno;
        if(childpid==-1)//Check if fork() failed
        {
                printf("Fork failed due to \"%s\"\n",strerror(keep_it));//print the system error that corresponds to errno
                return -1;
        }

        if(childpid==0)//Here is where the paths change, it could be done differently.
        {//Child code path here
                printf("Child process: \nPID\tPPID\n%d\t%d\n",pid,ppid);
                pid=getpid();
                ppid=getppid();
                printf("Child process: \nPID\tPPID\n%d\t%d\n",pid,ppid);
                sleep(5);
        }
        else
        {//Parent code path here
                sleep(10);
                printf("Parent process: \nPID\tPPID\n%d\t%d\n",pid,ppid);
                printf("Parent process: child PID was %d\n",childpid);
        }
        return 1;
}

lets run it to see what happens:

juan@test:~/clone_fork$ gcc -o fork_simple_example fork_simple_example.c
juan@test:~/clone_fork$ ./fork_simple_example
Child process:
PID     PPID
3213    2965
Child process:
PID     PPID
3214    3213
Parent process:
PID     PPID
3213    2965
Parent process: child PID was 3098
juan@test:~/clone_fork$

on a different shell I also captured the processes with ps:

juan@test:~$ ps axo stat,user,comm,ppid,pid|grep fork
S+   juan     fork_simple_exa  2965  3213
S+   juan     fork_simple_exa  3213  3214
juan@test:~$

So, what do we see from both outputs? The child process printed twice its PID and PPID just for the sake of showing how the first time the values on those variables were actually the ones collected by its father before the fork() call.

 File descriptors are preserved example

As we mentioned before, certain kernel structures are copied to the new child process, one of them are the open file descriptors. Now lets see that in an example using a pipe.

Note: a pipe is a type of Inter Process Communication mechanism, you can think the pipe as that simply a pipe with two ends, one where you can write to and one where you can read from. For more details please have a look at this.

The code is:

#include <stdio.h>
#include <sys/types.h>
#include <unistd.h>
#include <sys/syscall.h>
#include <errno.h>
#include <string.h>
#define SIZE 250

int main()
{
        char buffer[SIZE],ch;
        int keep_it,childpid,aux,count;
        int my_pipe[2];//my_pipe[] will keep two FD, my_pipe[1] to write into the pipe and my_pipe[0] to read from the pipe.

        aux=pipe(my_pipe);//Note how the pipe is created BEFORE the fork() call

        if(aux==-1)//Check if pipe() failed
        {
                printf("Pipe failed due to \"%s\"\n",strerror(keep_it));//print the system error that corresponds to errno
                return -1;
        }

        childpid=fork();//fork() returns the child PID on the parent's code path and 0 on the child's. On failure returns -1
        //From this point on, there are 2 processes running, unless fork failed of course :D.
        keep_it=errno;
        if(childpid==-1)//Check if fork() failed
        {
                printf("Fork failed due to \"%s\"\n",strerror(keep_it));//print the system error that corresponds to errno
                return -1;
        }

        if(childpid==0)//Here is where the paths change, it could be done differently.
        {//Child code path here
                close(my_pipe[0]);//On the child process we can close the read end of the pipe
                printf("Hi, this is the child process, insert message here (:P less than %d letters please): ",SIZE);
                fgets(buffer,sizeof(buffer),stdin);
                count=write(my_pipe[1],buffer,SIZE);
                printf("message sent to parent process.\n");
        }
        else
        {//Parent code path here
                close(my_pipe[1]);//On the parent process we can close the write end of the pipe
                read(my_pipe[0],buffer,SIZE);
                printf("Parent process received message: %s",buffer);
        }
        return 1;
}

This code is a bit more complex, but the comments should help.

You can see how the pipe was open on the parent process (Line 15) and yet it was used on the child process (Line 37) without any problems! Also worth noting how the pipe requires 2 file descriptors, one to read from the pipe (stored in my_pipe[0]) and one to write into the pipe (stored in my_pipe[1]). After the fork() call since child and parent process have copies of these open file descriptors they can safely close the ones they won't use, and then they end up with a unidirectional inter process communication channel (child -> PIPE -> parent).

A funny fact

 
At this point I was tempted to run some straces to show how the fork syscall was being used (using strace) and noticed the following, this is the strace output of running the first simple example:

juan@test:~/clone_fork$ strace ./fork_simple_example
execve("./fork_simple_example", ["./fork_simple_example"], [/* 22 vars */]) = 0
brk(0)                                  = 0x16ba000
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fa76a34a000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=95253, ...}) = 0
mmap(NULL, 95253, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fa76a332000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P \2\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1840928, ...}) = 0
mmap(NULL, 3949248, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fa769d65000
mprotect(0x7fa769f1f000, 2097152, PROT_NONE) = 0
mmap(0x7fa76a11f000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1ba000) = 0x7fa76a11f000
mmap(0x7fa76a125000, 17088, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fa76a125000
close(3)                                = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fa76a331000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fa76a32f000
arch_prctl(ARCH_SET_FS, 0x7fa76a32f740) = 0
mprotect(0x7fa76a11f000, 16384, PROT_READ) = 0
mprotect(0x600000, 4096, PROT_READ)     = 0
mprotect(0x7fa76a34c000, 4096, PROT_READ) = 0
munmap(0x7fa76a332000, 95253)           = 0
getpid()                                = 4981
getppid()                               = 4978
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fa76a32fa10) = 4982
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigaction(SIGCHLD, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
nanosleep({10, 0}, Child process:
PID     PPID
4981    4978
Child process:
PID     PPID
4982    4981
{4, 995479188})      = ? ERESTART_RESTARTBLOCK (Interrupted by signal)
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=4982, si_status=1, si_utime=0, si_stime=0} ---
restart_syscall(<... resuming interrupted call ...>
) = 0
fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 8), ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fa76a349000
write(1, "Parent process: \n", 17Parent process:
)      = 17
write(1, "PID\tPPID\n", 9PID    PPID
)              = 9
write(1, "4981\t4978\n", 104981 4978
)            = 10
write(1, "Parent process: child PID was 49"..., 35Parent process: child PID was 4982
) = 35
exit_group(1)                           = ?
+++ exited with 1 +++
juan@test:~/clone_fork$

do you see any fork() call there? ... exactly there's no fork call!!! But I said fork is a Linux syscall and blah blah blah, right? Well, worry not, I wasn't lying :D all I said is true however...

Since version 2.3.3, rather than invoking the kernel's fork() system call, the glibc fork() wrapper that is provided as part of the NPTL threading implementation invokes clone(2) with flags that provide the same effect as the traditional system call. (A call to fork() is equivalent to a call to clone(2) specifying flags as just SIGCHLD.) 

that's the reason why we do see a clone call instead!

Now my brain needs some rest so I'll finish this post here, any feedback will be more than welcome!

On the next post I'll describe clone() and we'll see some examples to understand even better the differences with fork().