Mostrando entradas con la etiqueta sysadmin. Mostrar todas las entradas
Mostrando entradas con la etiqueta sysadmin. Mostrar todas las entradas

lunes, 15 de mayo de 2017

We all understand free command... don't we???

I know I should be writing the next entry of "Understanding SOMAXCONN parameter Part I" (and I am) but instead I decided to do a little detour first. I faced this question about free a few days ago and noticed odd/different answers for it (including my own xD) so decided to go for a deep dive and found some interesting facts worth sharing.

Free


Free Linux command is one of those tools you will be definitely get exposed to as a sysadmin, and understanding its output is extremely important. A bad interpretation of free output could cause unnecessary panic and you don't want to panic for no reason :D (oh wait... maybe you do, I certainly don't ).

Lets have a look at a simple free output:

juan@test:~$ free
             total       used       free     shared    buffers     cached
Mem:       3007572     584872    2422700      14780      55808     291704
-/+ buffers/cache:     237360    2770212
Swap:            0          0          0
juan@test:~$

In the line starting with "Mem:" we get to see the total available physical memory on the system 3007572 KBytes (3GBytes) and then how it is distributed:
  • used: memory being occupied
  • free: memory that is available in the system
  • shared: amount of memory that is shared between processes
  • buffers: memory used for kernel buffers involving IO (disk, network, etc)
  • cached: memory used for data caching
Then we have a line starting with "-/+ buffers/cache:" which looks kind of cryptic, doesn't it?... it turns out to be a kind of summary of the previous line:
  • There are 237360 KBytes being used, this number is the result of the following math "used - (buffers + cached)". This is the amount of memory the system is actually relying on.
  • There are 2770212 KBytes that could be utilized, this is the result of "free + buffers + cached". What this number actually means is that the kernel should be able to reclaim buffer and cached memory if necessary.
So we have a better understanding of free output now, but lets get more details of where these numbers come from.

The power of the force source


Free binary is shipped as part of procps-ng package in Linux and we can inspect its latest code here. What free actually does for you is summarize memory utilization details coming from /proc/meminfo in a more "human way". So lets dissect this...
  • The main source code can be read here free.c, if you have a look at the lines between 359-377, you can see that there are 6 variables kb_main_total, kb_main_used, kb_main_free, kb_main_shared, kb_main_buffers and kb_main_cached and they are actually the ones containing the values that will show up in free's output. However if you look for these variables on the code you won't find them, they are extern variables included from "proc/sysinfo.h" which means the variables are being initialized somewhere else, particularly in "proc/sysinfo.c". The variables are actually being populated by meminfo() function being called in line 355.
  meminfo();
  /* Translation Hint: You can use 9 character words in
   * the header, and the words need to be right align to
   * beginning of a number. */
  if (flags & FREE_WIDE) {
   printf(_("              total        used        free      shared     buffers       cache   available"));
  } else {
   printf(_("              total        used        free      shared  buff/cache   available"));
  }
  printf("\n");
  printf("%-7s", _("Mem:"));
  printf(" %11s", scale_size(kb_main_total, flags, args));
  printf(" %11s", scale_size(kb_main_used, flags, args));
  printf(" %11s", scale_size(kb_main_free, flags, args));
  printf(" %11s", scale_size(kb_main_shared, flags, args));
  if (flags & FREE_WIDE) {
   printf(" %11s", scale_size(kb_main_buffers, flags, args));
   printf(" %11s", scale_size(kb_main_cached, flags, args));
  } else {
   printf(" %11s", scale_size(kb_main_buffers+kb_main_cached, flags, args));
  }
  printf(" %11s", scale_size(kb_main_available, flags, args));
  printf("\n");

  • Now checking the "proc/sysinfo.c" code where meminfo() function exists, we see:
    • kb_main_buffer is populated with the content of Buffers value from /proc/meminfo (line 691), this is memory used by the kernel to temporarily hold data being sent/received by the system (network IO, disk IO, etc).
    • kb_main_cached is populated with the result of "kb_page_cache + kb_slab_reclaimable" (line 762). These two values come from Cached and SReclaimable respectively (in /proc/meminfo). So the cached value presented by free contains the memory used by the page cache and the reclaimable slab memory (slab memory caches dentry and inodes structures to speed up some fs operations).
  {"Bounce",       &kb_bounce},
  {"Buffers",      &kb_main_buffers}, // important
  {"Cached",       &kb_page_cache},  // important
  {"CommitLimit",  &kb_commit_limit},
  ...
  kb_main_cached = kb_page_cache + kb_slab_reclaimable;
  kb_swap_used = kb_swap_total - kb_swap_free;

It is clear now that free relies completely on the details exposed by the kernel through /proc/meminfo, and the math involved in getting the values isn't really rocket science after all. But here it comes the interesting part...

To have in mind


It looks like the information provided by free can be slightly different between different Linux flavors (because they carry different free versions :D) so I thought it might be worth pointing that out here as well. This could lead to some inconsistencies when some other tools rely on the output of free.

Free version shipped with Ubuntu 14.04 (3.3.9), doesn't even include Slabs under the cached memory, you can see here how kb_main_cached value is just populated with Cached value from /proc/meminfo.

juan@test:~$ free -V
free from procps-ng 3.3.9
juan@test:~$ free
             total       used       free     shared    buffers     cached
Mem:       3007572     530948    2476624      14776      53080     244740
-/+ buffers/cache:     233128    2774444
Swap:            0          0          0
juan@test:~$ grep "^Cached\|^SReclaimable\|^Slab" /proc/meminfo
Cached:           244740 kB
Slab:              32308 kB
SReclaimable:      17572 kB
juan@test:~$

Note: cached = Cached

Free version shipped with CentOS 7 (3.3.10), defines kb_main_cached as "kb_page_cache + kb_slab" (706) this seems a minor thing but not all the slab memory is reclaimable therefore part of this cached content is not really available in case of memory pressure.

[juan@server ~]$ free -V
free from procps-ng 3.3.10
[juan@server ~]$ free -w
              total        used        free      shared     buffers       cache   available
Mem:        1016860       74212      818380        6688         948      123320      810600
Swap:             0           0           0
[juan@server ~]$ grep "^Cached\|^SRecl\|^Slab" /proc/meminfo
Cached:            88876 kB
Slab:              34444 kB
SReclaimable:      13928 kB
[juan@server ~]$

Note: cache = Cached + Slab

Interesting, isn't it? That's the power of open source after all right? Having the chance to really understand what certain piece of software is doing for you and how is doing it!

That's all about free, hope it helps (it helped me at least :P).

domingo, 16 de octubre de 2016

Linux limits 102 - Open files

This post is kind of a follow up of Linux limits 101 - Ulimit from a few weeks ago where we went through ulimit and how they are used to limit users behaviors. This time we'll take a look at another limit the kernel can impose on us and therefore make our lives a bit sower, the number of open files.

I think it's worth mentioning (I may have said this before xD) that when we talk about Open Files in Linux systems we are basically talking about File descriptors (aka file handles). A file descriptor is a data structure used by processes to access: files, Unix sockets, Networking sockets, pipes, etc. Every new process comes by default with 3 file descriptors:
  • FD 0: standard input
  • FD 1: standard output
  • FD 2: standard error
So just by having lets say:
juan@test:~$ ps aux|wc -l
142
juan@test:~$
about 142 processes running on the system, we should expect at least around 426 file descriptors to be in use (142x3=426). What if there was a way to know how many file descriptors a particular process is using?

File descriptors a process is using:


Yeahp, of course that's possible! And as always in Linux, there are at least two different ways. The first approach will be of course the easiest one, each process has a folder under /proc that will provide loads of information, in this case will focus on subfolder fd, where guess what's going to show up? indeed, the file descriptors for the process:
root@test:/home/juan# ls /proc/1/fd | wc -l
24
root@test:/home/juan#
init process (PID=1) has 24 file descriptors in use! We can see more details about them in the next output:
root@test:/home/juan# ls -la /proc/1/fd
total 0
dr-x------ 2 root root  0 Oct 15 10:39 .
dr-xr-xr-x 9 root root  0 Oct 15 10:38 ..
lrwx------ 1 root root 64 Oct 15 10:39 0 -> /dev/null
lrwx------ 1 root root 64 Oct 15 10:39 1 -> /dev/null
lrwx------ 1 root root 64 Oct 15 10:39 10 -> socket:[8662]
lrwx------ 1 root root 64 Oct 15 10:39 11 -> socket:[9485]
l-wx------ 1 root root 64 Oct 15 10:39 12 -> /var/log/upstart/network-manager.log.1 (deleted)
lrwx------ 1 root root 64 Oct 15 10:39 14 -> socket:[10329]
l-wx------ 1 root root 64 Oct 15 10:39 16 -> /var/log/upstart/systemd-logind.log.1 (deleted)
lrwx------ 1 root root 64 Oct 15 10:39 17 -> socket:[8637]
lrwx------ 1 root root 64 Oct 15 10:39 18 -> /dev/ptmx
lrwx------ 1 root root 64 Oct 15 10:39 2 -> /dev/null
lrwx------ 1 root root 64 Oct 15 10:39 20 -> /dev/ptmx
lrwx------ 1 root root 64 Oct 15 10:39 22 -> /dev/ptmx
l-wx------ 1 root root 64 Oct 15 10:39 24 -> /var/log/upstart/modemmanager.log.1 (deleted)
lrwx------ 1 root root 64 Oct 15 10:39 29 -> /dev/ptmx
lr-x------ 1 root root 64 Oct 15 10:39 3 -> pipe:[8403]
lrwx------ 1 root root 64 Oct 15 10:39 30 -> /dev/ptmx
l-wx------ 1 root root 64 Oct 15 10:39 31 -> /var/log/upstart/mysql.log.1 (deleted)
lrwx------ 1 root root 64 Oct 15 10:39 34 -> /dev/ptmx
lrwx------ 1 root root 64 Oct 15 10:39 36 -> /dev/ptmx
l-wx------ 1 root root 64 Oct 15 10:39 4 -> pipe:[8403]
lr-x------ 1 root root 64 Oct 15 10:39 5 -> anon_inode:inotify
lr-x------ 1 root root 64 Oct 15 10:39 6 -> anon_inode:inotify
lrwx------ 1 root root 64 Oct 15 10:39 7 -> socket:[8404]
lrwx------ 1 root root 64 Oct 15 10:39 9 -> socket:[12675]
root@test:/home/juan#
we can see how the file descriptors are represented as links, a brief of the output would be:
  • Default file descriptors (0,1 and 2) have been pointed to /dev/null, which is ok for a process like init that isn't an interactive process.
  • There are a couple of UNIX sockets "socket:[XXXX]" opened (7,9,10, 11, etc), probably to connect to other processes.
  • There's a pipe "pipe:[8403]" as well using two fd (3 and 4) that's normal, pipes provide a fd to write and one to read while data is buffered on the kernel.
  • The rest of the fs point to:
    • /dev/ptmx pseudo terminal device.
    • inotify a way to monitor changes on files, this means init is interested in the events on two particular fd.
    • some deleted log files like /var/log/upstart/mysql.log.1 this is odd. Probably files were rotated or something like that.
 If for some reason these details weren't enough, you can go hardcore and try the second way.  lsof (list open files, makes sense, right?) is the program you need for that.

Lets list all the open files for a particular process using lsof, init in this case:
root@test:/home/juan# lsof -p 1
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/112/gvfs
      Output information may be incomplete.
COMMAND PID USER   FD   TYPE             DEVICE SIZE/OFF   NODE NAME
init      1 root  cwd    DIR                8,1     4096      2 /
init      1 root  rtd    DIR                8,1     4096      2 /
init      1 root  txt    REG                8,1   265848 261189 /sbin/init
init      1 root  mem    REG                8,1    43616 581960 /lib/x86_64-linux-gnu/libnss_files-2.19.so
init      1 root  mem    REG                8,1    47760 555508 /lib/x86_64-linux-gnu/libnss_nis-2.19.so
init      1 root  mem    REG                8,1    97296 555504 /lib/x86_64-linux-gnu/libnsl-2.19.so
init      1 root  mem    REG                8,1    39824 555503 /lib/x86_64-linux-gnu/libnss_compat-2.19.so
init      1 root  mem    REG                8,1    14664 555500 /lib/x86_64-linux-gnu/libdl-2.19.so
init      1 root  mem    REG                8,1   252032 540246 /lib/x86_64-linux-gnu/libpcre.so.3.13.1
init      1 root  mem    REG                8,1   141574 555505 /lib/x86_64-linux-gnu/libpthread-2.19.so
init      1 root  mem    REG                8,1  1840928 581957 /lib/x86_64-linux-gnu/libc-2.19.so
init      1 root  mem    REG                8,1    31792 581956 /lib/x86_64-linux-gnu/librt-2.19.so
init      1 root  mem    REG                8,1    43464 527349 /lib/x86_64-linux-gnu/libjson-c.so.2.0.0
init      1 root  mem    REG                8,1   134296 527439 /lib/x86_64-linux-gnu/libselinux.so.1
init      1 root  mem    REG                8,1   281552 527323 /lib/x86_64-linux-gnu/libdbus-1.so.3.7.6
init      1 root  mem    REG                8,1    38920 527371 /lib/x86_64-linux-gnu/libnih-dbus.so.1.0.0
init      1 root  mem    REG                8,1    96280 527373 /lib/x86_64-linux-gnu/libnih.so.1.0.0
init      1 root  mem    REG                8,1   149120 555506 /lib/x86_64-linux-gnu/ld-2.19.so
init      1 root    0u   CHR                1,3      0t0   1029 /dev/null
init      1 root    1u   CHR                1,3      0t0   1029 /dev/null
init      1 root    2u   CHR                1,3      0t0   1029 /dev/null
init      1 root    3r  FIFO                0,9      0t0   8403 pipe
init      1 root    4w  FIFO                0,9      0t0   8403 pipe
init      1 root    5r  0000               0,10        0   7661 anon_inode
init      1 root    6r  0000               0,10        0   7661 anon_inode
init      1 root    7u  unix 0xffff8800b37c8780      0t0   8404 @/com/ubuntu/upstart
init      1 root    9u  unix 0xffff8800a14e7c00      0t0  12675 @/com/ubuntu/upstart
init      1 root   10u  unix 0xffff8800b37c9a40      0t0   8662 @/com/ubuntu/upstart
init      1 root   11u  unix 0xffff8800b37a1e00      0t0   9485 @/com/ubuntu/upstart
init      1 root   12w   REG                8,1      283 551619 /var/log/upstart/network-manager.log.1 (deleted)
init      1 root   14u  unix 0xffff8800b37a3c00      0t0  10329 @/com/ubuntu/upstart
init      1 root   16w   REG                8,1      451 522345 /var/log/upstart/systemd-logind.log.1 (deleted)
init      1 root   17u  unix 0xffff8800b37cb0c0      0t0   8637 socket
init      1 root   18u   CHR                5,2      0t0   1932 /dev/ptmx
init      1 root   20u   CHR                5,2      0t0   1932 /dev/ptmx
init      1 root   22u   CHR                5,2      0t0   1932 /dev/ptmx
init      1 root   24w   REG                8,1      502 527289 /var/log/upstart/modemmanager.log.1 (deleted)
init      1 root   29u   CHR                5,2      0t0   1932 /dev/ptmx
init      1 root   30u   CHR                5,2      0t0   1932 /dev/ptmx
init      1 root   31w   REG                8,1      881 552236 /var/log/upstart/mysql.log.1 (deleted)
init      1 root   34u   CHR                5,2      0t0   1932 /dev/ptmx
init      1 root   36u   CHR                5,2      0t0   1932 /dev/ptmx
root@test:/home/juan#
now we can see a few more things, like:
  • Details of the FD, like its Type, Device it belongs to, etc.
  • We can see also some things that aren't really opened fd but some extra process information:
    • cwd current working directory
    • rtd root directory
    • txt init's binary code file
    • memory mapped files in this case bunch of system libraries. YES, these had a fd when they were mapped, but the fd was closed right after the mmap call was successful (you can see that checking this entry about strace).
Ok, now we know what a file descriptor or file handle is and how to identify them and map them to our processes. Is there any system wide limit for the file descriptors you can open?
 

Max open files, system wide:


If the answer for the previous question was now, I wouldn't have a reason to write this article in the first place I guess xD, therefore the answer is YES :P. The kernel is cool, and in order to play safe it has to set limits (I sound like a father now...) to avoid bigger problems.

The maximum number of open files the kernel can handle can be obtained from our beloved /proc, particularly in file file-nr under sys/fs directory. Here we can see the numbers for my current system:
root@test:/home/juan# cat /proc/sys/fs/file-nr
2944 0 298505
root@test:/home/juan#
These values mean the following:
  • First value (2944) indicates the number of allocated file descriptors, these are allocated dynamically by the kernel.
  • Second value (0) is the number of allocated but unused file descriptor. Kernels from 2.6.something free any unused fd, so this value should always be 0.
  • Third value (298505) indicates the maximum number of file descriptors that the kernel can allocate (also visible on file-max file).
Summing up, there are 2944 file descriptors allocated and in use at this precise moment. Almost 3k file descriptors allocated for about 142, interesting right?

Just for the sake of it, lets track down the process using the most number of file descriptors:
root@test:/home/juan# for i in `ps -Ao pid|grep -v PID`;do count=`ls /proc/$i/fd/ 2> /dev/null|wc -l`; echo "$count $i";done | sort -nr | head
61 1075
48 393
32 1264
31 1365
27 1132
24 1
21 440
20 1265
19 1325
19 1311
root@test:/home/juan#
there we see the top ten (first column is the number of FD and the second is the PID). Interestingly enough there's a process using 61 file descriptors, turns out I had mysqld installed on this VM (had no idea...):
root@test:/home/juan# ps aux|grep 1075
mysql     1075  0.1  1.9 624040 57792 ?        Ssl  10:38   0:07 /usr/sbin/mysqld
root      9664  0.0  0.0  15948  2232 pts/1    S+   12:35   0:00 grep --color=auto 1075
root@test:/home/juan#
 

Increasing the limit


If by any chance the almost 300k file descriptors the kernel allows to open is not enough (some busy systems may reach that limit) you will notice logs like "VFS: file-max limit reached" on dmesg and probably in messages or syslog files. In that case,  you can increase that limit using one of the following ways:
  • iminproductionpainchangeitrightnowgoddamnit way, by just updating /proc with the new value, like
root@test:/home/juan# echo 400000 > /proc/sys/fs/file-max
root@test:/home/juan# cat /proc/sys/fs/file-nr
3072    0       400000
root@test:/home/juan#
  • Or you can be more elegant and use sysctl command:
root@test:/home/juan# sysctl -w fs.file-max=500000
fs.file-max = 500000
root@test:/home/juan# cat /proc/sys/fs/file-nr
3072    0       500000
root@test:/home/juan#

In any case, don't forget to make the change persistent by doing:
root@test:/home/juan# echo "fs.file-max=500000" >> /etc/sysctl.conf
root@test:/home/juan#

If the error showing on your logs is instead "Too many open files" then the limit you've reached is most likely the ulimit for the user :D, which you know how to deal with because you've read this.

And that's about it!