Mostrando entradas con la etiqueta XEN. Mostrar todas las entradas
Mostrando entradas con la etiqueta XEN. Mostrar todas las entradas

martes, 7 de marzo de 2017

Where did my IO Scheduler go? Where are you my beloved [noop|deadline|cfq]?

That was exactly the question I asked myself when I came across the following output:
root@ip-172-31-22-167:/home/ubuntu# cat /sys/block/xvda/queue/scheduler
none
root@ip-172-31-22-167:/home/ubuntu#
root@ip-172-31-22-167:/home/ubuntu# uname -a
Linux ip-172-31-22-167 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20 11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
root@ip-172-31-22-167:/home/ubuntu#

Of course my first approach was to curse, blame politicians, global warming and even capitalism..., but somehow that didn't provide any answers.

Well turns out that a few things have changed in the Linux Kernel without my consent (xD), how dared them, right? Developers and their creativity :P.

In order to understand a bit what this none "scheduler" means, I'll try to sum up a bit of history, Linux and politics in this post.

Time between Kernel 2.6.10 to somewhere before 3.13 / 3.16


So between these two kernel versions there were basically two different methods for the block drivers to interact with the system. First method was called Request Mode, and the second method you could call it "Make your own Request Interface" Mode.

Request Mode 


In this mode the Linux Block Layer maintains a simple request queue (using a linked list structure) where all the IO requests are placed one after the other (yeah, like a queue xD). The device driver then receives the requests from the head of the queue, and that's pretty much about it in terms of its logic. Having the IOs waiting there in the queue actually made some interesting stuff possible, for example:
  • Why don't we reorder them in order to make a better use of the HW? Makes perfect sense right? Well for the young ones... there used to be something called magnetic drives that due to their HW design (plates, spindles, head, etc) in order to get the better out of them you should try to do sequential operations as opposed to random ones, to minimize the seek times for example.
  • Also, now that we have them reorder, why don't we merge them to get even better throughput? And so they did.
  • Another interesting feature that was implemented later on is the chance to apply certain policies to achieve a kind of fairness in terms of processes getting access to IO.
These are the pillars where the IO schedulers like noop, deadline and CFQ were build on. You can still see them in some of the current Linux like in Ubuntu 14.04.5:

juan@test:~$ uname -a
Linux test 3.19.0-51-generic #58~14.04.1-Ubuntu SMP Fri Feb 26 22:02:58 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
juan@test:~$ cat /sys/block/sda/queue/scheduler
noop [deadline] cfq
juan@test:~$

Note: even though the kernel is 3.19, blk-mq had already been added by then, but it was not the default for some drivers like the scsi driver in this case.

So why was it necessary for the developers to replace this lovely piece of Software after all??? Well there were a few reasons:
  • Today's drives (mainly SSDs) are way faster that the drives we had a couple of years ago, Magnetic drives are kind of on their way to extinction as well...
  • Nowadays due to Virtualization the guest OS doesn't really get to know the real physical layout of the drives so using CPU cycles trying to sort the requests probed to be a waste of time.
  • With really fast SSD drives/arrays the single queue mechanism can easily become a bottle neck mainly in multiprocessor environments (yeah, everywhere now). 
    • The queue can only be accessed by a single processor at the time, so some contention can be generated because of this as well when trying to push high number of IO operations.

"Make your own Request Interface" Mode


This second Mode is just a way to skip the Request Mode and it's IO scheduling algorithms, allowing the driver to be directly accessed by higher parts of the IO stack. This is used for example for MD to build software RAID, and other tools that need to process the requests themselves before actually sending them to the devices.

The Block Multiqueue IO mechanism (that's quite a name, isn't it?)


Kernel developers realized that the previous Single queuing mechanism had become a huge bottleneck on the Linux IO stack so they came up with this new mechanism. On Kernel 3.13 the first Multiqueue compatible driver was introduced in order to start testing the future IO queuing mechanism (blk-mq), finally on 3.16 and beyond complete blk-mq implementations began to show up for different drivers.

This new mechanism includes two levels of queuing. On the first level, what used to be the single queue where the IO operations where placed has been split into multiple submission queues, one per CPU (or per Node), improving considerably the number of IO operations the system can handle (now each CPU has its own submission queue, getting the contention issue totally off the table). The second level of queuing is the hardware queues the device itself may provide.

The mapping between the submission queues and the hardware queues might be 1:1 or N:M depending on the configuration.

Therefore with this new Multiqueue mechanism, IO schedulers became pretty much useless and they are no longer available for many drivers already. Maybe in the future new schedulers/logic might show up to work on the individual submission queues.

I guess this was the longest possible way to explain where the IO schedulers went on my Ubuntu 16.04 instance after all :D, hopefully will help someone to understand what is going on:
root@ip-172-31-22-167:/home/ubuntu# uname -a
Linux ip-172-31-22-167 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20 11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
root@ip-172-31-22-167:/home/ubuntu# cat /sys/block/xvda/queue/scheduler
none
root@ip-172-31-22-167:/home/ubuntu#

Note: in kernel 4.3 blkfront driver (Xen block driver) was converted to blk-mq

If I have the time will shoot some tests in order to see the different behaviors I mentioned, just for fun xD.

Some interesting bibliography:


https://lwn.net/Articles/552904/
https://www.thomas-krenn.com/en/wiki/Linux_Multi-Queue_Block_IO_Queueing_Mechanism_(blk-mq)
https://www.thomas-krenn.com/en/wiki/Linux_I/O_Stack_Diagram#Diagram_for_Linux_Kernel_3.17
http://kernel.dk/systor13-final18.pdf

domingo, 4 de octubre de 2015

Cómo convertir de MBR a GPT nuestro disco y no morir en el intento!

Este post es un poco la continuación de ¿MBR o GPT? deep dive pero ahora voy a mostrar de qué manera se puede convertir un volumen root de MBR a GPT sin morir en el intento. Para hacer esto voy a utilizar dos instancias EC2 corriendo Ubuntu 14.04 LTS.

Siendo sincero, la verdad es que este no es un escenario que se le vaya a presentar a muchas personas, pero me pareció interesante explorarlo.

Entonces tenemos dos instancias idénticas:
  • Corriendo en la misma AZ (para poder intercambiar los discos)
  • Una instancia la llamaré auxiliar, que es donde se hará el trabajo sucio.
  • La otra instancia se llama oveja_negra y es la que recibirá el cambio de sistema de particionado.

Estado inicial de oveja_negra


Como primera medida voy a ver cuál es el estado general del volumen y cómo está compuesto.

ubuntu@oveja-negra:~$ sudo parted /dev/xvda
GNU Parted 2.3
Using /dev/xvda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) unit s                                                          
(parted) print                                                           
Model: Xen Virtual Block Device (xvd)
Disk /dev/xvda: 16777216s
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start   End        Size       Type     File system  Flags
 1      16065s  16771859s  16755795s  primary  ext4         boot

(parted) quit                                                            
ubuntu@oveja-negra:~$


Vemos que se trata de un disco de 8GB, con tabla de partición msdos y con una sola partición primaria booteable que comienza en el sector 16065 y termina en el sector 16771859. Aquí me voy a detener para hace foco en dos cosas:
  • La partición comienza en el sector 16065!!! Es decir que los sectores entre el LBA1 y LBA16064 están disponibles (poco menos de 8 Mbytes).
  • La partición termina en el sector 16771859, mientras que el disco tiene 16777216 sectores... Osea que entre el final de la partición y el del disco hay unos 5357 sectores disponibles (unos 2.5Mbytes).
Si se acuerdan, GPT utliza bastante mas espacio que el sistema convencional de particiones, por lo tanto es importante saber que se cuenta con ese espacio disponible. Habiendo confirmado que todo está en orden para continuar, lancé shutdown a la instancia para comenzar.

NOTA: este es el momento donde deberías tomar un snapshot del disco raiz en caso de que algo salga mal!!!

Que comience la cirugía


Una vez que la instancia se encuentre detenida, es necesario desconectar el disco y conectar a la instancia auxiliar. Con el disco conectado a la instancia auxiliar vemos que fue agregado como /dev/xvdf:

ubuntu@auxiliar:~$ sudo lsblk
NAME    MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda    202:0    0   8G  0 disk
`-xvda1 202:1    0   8G  0 part /
xvdf    202:80   0   8G  0 disk
`-xvdf1 202:81   0   8G  0 part

ubuntu@auxiliar:~$


Damos una última mirada a la tabla de particiones de nuevo

ubuntu@auxiliar:~$ sudo parted /dev/xvdf
GNU Parted 2.3
Using /dev/xvdf
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted)unit s                                             
(parted)print                                                    
Model: Xen Virtual Block Device (xvd)
Disk /dev/xvdf: 16777216s
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start   End        Size       Type     File system  Flags
 1      16065s  16771859s  16755795s  primary  ext4         boot

(parted)  


Ahora en un acto de casi locura procedemos a eliminar la tabla de particiones

(parted)mklabel                                                         
New disk label type? gpt                                
Warning: Partition(s) on /dev/xvdf are being used.                       
Ignore/Cancel? Ignore                                 
Warning: The existing disk label on /dev/xvdf will be destroyed and all data on
this disk will be lost. Do you want to continue?
Yes/No? Yes                                            
(parted) print                                                   

Model: Xen Virtual Block Device (xvd)
Disk /dev/xvdf: 16777216s
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start  End  Size  File system  Name  Flags

(parted)  


Este es el momento donde nuestra información se encuentra en lo que yo llamaría el limbo informático. hemos perdido toda referencia  e información de la misma, ya que hemos reemplazada la tabla de particiones msdos por una gpt. De hecho si cerramos parted podemos corroborar que ya no existe /dev/xvdf1:

ubuntu@auxiliar:~$ sudo lsblk
NAME    MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda    202:0    0   8G  0 disk
`-xvda1 202:1    0   8G  0 part /
xvdf    202:80   0   8G  0 disk
ubuntu@auxiliar:~$  


Lo bueno de todo esto es que nada está perdido! Ahora hay que reconstruir la tabla de particiones con la información que teníamos previamente:

ubuntu@auxiliar:~$ sudo parted /dev/xvdf
GNU Parted 2.3
Using /dev/xvdf
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted)unit s                                         
(parted)mkpart
Partition name?  []?                                                     
File system type?  [ext2]? ext4                   
Start? 16065                                   
End? 16771859
Warning: The resulting partition is not properly aligned for best performance.
Ignore/Cancel? Ignore                                            
(parted) print                                           
Model: Xen Virtual Block Device (xvd)
Disk /dev/xvdf: 16777216s
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start   End        Size       File system  Name  Flags
 1      16065s  16771859s  16755795s  ext4

(parted) 

Sencillamente creamos una entra en la tabla de particiones GPT que indique donde se encontraba nuestra partición previamente. Ahora ya deberíamos ser capaces de ver la partición y montarla perfectamente de nuevo:

ubuntu@auxiliar:~$ sudo lsblk
NAME    MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda    202:0    0   8G  0 disk
`-xvda1 202:1    0   8G  0 part /
xvdf    202:80   0   8G  0 disk
`-xvdf1 202:81   0   8G  0 part

ubuntu@auxiliar:~$ sudo fsck.ext4 -f /dev/xvdf1
e2fsck 1.42.9 (4-Feb-2014)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
cloudimg-rootfs: 58327/524288 files (0.2% non-contiguous), 265348/2094474 blocks

ubuntu@auxiliar:~$ sudo mount /dev/xvdf1 /mnt/
ubuntu@auxiliar:~$ ls /mnt/
bin   dev  home        lib    lost+found  mnt  proc  run   srv  tmp  var
boot  etc  initrd.img  lib64  media       opt  root  sbin  sys  usr  vmlinuz

ubuntu@auxiliar:~$


En este punto ya está casi todo terminado, de hecho si no se tratara del volumen principal del sistema podríamos terminar acá. Pero si volviésemos a poner el volumen en la instancia oveja-negra notaríamos que la instancia no arranca y jamás lo volverá a hacer en esas condiciones. ¿Por qué? Básicamente porque hemos jugado con la tabla de particiones y los sectores de arranque del sistema. Para arreglar esto re-instalamos GRUB para que repare cualquier situación extraña (/dev/xvdf1 está montado en /mnt)

ubuntu@auxiliar:~$ sudo grub-install --boot-directory=/mnt/boot /dev/xvdf
Installing for i386-pc platform.
grub-install: warning: this GPT partition label contains no BIOS Boot Partition; embedding won't be possible.
grub-install: warning: Embedding is not possible.  GRUB can only be installed in this setup by using blocklists.  However, blocklists are UNRELIABLE and their use is discouraged..
grub-install: error: will not proceed with blocklists.
ubuntu@auxiliar:~$


WTF!!! Si, si bien GRUB 2 permite bootear de particiones GPT, en sistemas BIOS requiere de algo llamado BIOS Boot Partition donde aloja la segunda etapa de su código (NO necesario en sistemas UEFI). Esto se debe a que al utilizar GPT los bloques que se encuentran a continuación del sector LBA0 (MBR) están ocupados por el encabezado GPT y el arreglo de particiones. Teniendo esta partición GRUB 2 se asegura de tener espacio suficiente donde poner el código necesario, la partición no requiere mucho espacio unos 2MB son mas que suficiente.

Creando la partición "BIOS boot partition"


Crearemos una partición de aproximadamente 2MB entre los sectores 2048 y 6144, esto nos debería garantizar que no estamos sobre escribiendo nada que no debamos:

(parted)mkpart                                                          
Partition name?  []?                                                    
File system type?  [ext2]?                                               
Start?2048                                                              
End?6144                                                                
(parted)print                                                           
Model: Xen Virtual Block Device (xvd)
Disk /dev/xvdf: 16777216s
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start   End        Size       File system  Name  Flags
 2      2048s   6144s      4097s                  
 1      16065s  16771859s  16755795s  ext4               boot

(parted) 


un detalle mas es que debemos marcar la partición con el flag bios_grub para que GRUB sepa que es allí donde debe buscar el código:

(parted) set 2 bios_grub on
(parted)print                                                           
Model: Xen Virtual Block Device (xvd)
Disk /dev/xvdf: 16777216s
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start   End        Size       File system  Name  Flags
 2      2048s   6144s      4097s                        bios_grub
 1      16065s  16771859s  16755795s  ext4               boot

(parted) 


Ahora deberíamos poder instalar grub perfectamente:

ubuntu@auxiliar:~$ sudo mount /dev/xvdf1 /mnt/
ubuntu@auxiliar:~$ sudo grub-install --boot-directory=/mnt/boot /dev/xvdf
Installing for i386-pc platform.
Installation finished. No error reported.
ubuntu@auxiliar:~$ sudo umount /mnt
ubuntu@auxiliar:~$ lsblk
NAME    MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda    202:0    0   8G  0 disk
`-xvda1 202:1    0   8G  0 part /
xvdf    202:80   0   8G  0 disk
|-xvdf1 202:81   0   8G  0 part
`-xvdf2 202:82   0   2M  0 part
ubuntu@auxiliar:~$


Podemos ver que si intentamos montar la partición resulta complicado:

ubuntu@auxiliar:~$ sudo mount /dev/xvdf2 /mnt/
mount: block device /dev/xvdf2 is write-protected, mounting read-only
mount: you must specify the filesystem type
ubuntu@auxiliar:~$


Si bien no tengo certezas es bastante probable que esta partición no tenga ningún formato conocido sino que sea solamente una reserva de espacio donde hay código binario ejecutable correspondiente a la etapa 2 de GRUB.

Ahora llegó el momento de la verdad, volvemos el disco a oveja-negra y le damos start a la instancia:

ubuntu@oveja-negra:~$ sudo parted /dev/xvda
GNU Parted 2.3
Using /dev/xvda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) print                                                           
Model: Xen Virtual Block Device (xvd)
Disk /dev/xvda: 8590MB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start   End     Size    File system  Name  Flags
 2      1049kB  3146kB  2098kB                     bios_grub
 1      8225kB  8587MB  8579MB  ext4               boot


(parted)   quit                                                            
ubuntu@oveja-negra:~$


Sin un raspón, sana, salva y con GPT!!!