Showing posts with label top. Show all posts
Showing posts with label top. Show all posts

Thursday 17 October 2019

Elementary Linux Performance Monitoring

The basic tool here is top
Monitoring a single process can be done with -p option, in the next example we measure the MySQL process:

[root@(db-master) ~]# top -p 2521
top - 15:42:54 up 40 days, 10:46,  4 users,  load average: 0.14, 0.24, 0.48
Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
%Cpu0  :  1.0 us,  1.0 sy,  0.0 ni, 98.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu1  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:  32551020 total, 32285684 used,   265336 free,   149660 buffers
KiB Swap:  3129340 total,   402572 used,  2726768 free. 16662620 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
 2521 mysql     20   0 18.725g 0.014t   4548 S 6.000 46.50   2735:03 mysqld

Load Average is a linux/unix mystery: Linux load averages are "system load averages" that show the running thread (task) demand on the system as an average number of running plus waiting threads. This measures demand, which can be greater than what the system is currently processing. 
For an extended excellent article around Linux Load Average, refer to Brendan Gregg's Blog

On the other hand good old ps which is available on all UNIX flavors and LINUX distributions can also help. The following command shows the most CPU consuming processes  in ascending order along with their virtual size 

[root@(db-master) ~]# ps -e -o pid,pcpu,vsz,comm= | sort -n  --key=3
...
 1669  0.0 752396 isecespd
 1759  0.0 1561472 isectpd
 2521 52.4 19634584 mysqld

To get the process tree try pstree -aAl:

[root@(db-master) ~]# pstree -aAl
systemd --switched-root --system --deserialize 24
  |-VGAuthService -s
  |-agetty --noclear tty1 linux
  |-automount -p /var/run/automount.pid
  |   `-5*[{automount}]
  |-cron -n
  |-dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
  |-discagnt /etc/init.d/discagnt start
  |   `-discagnt
  |-haveged -w 1024 -v 0 -F
...

For systems that do not have  pstree  try ps -ejH  

To get information about threads created by processes  try  ps -eLf

To get information about disk performance try iostat:

 [root@(mmcp_prod_corp)(db-master) ~]# iostat -dcm
Linux 4.4.121-92.117-default (mo-1400a55c2)     10/17/19        _x86_64_        (8 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           7.22    0.00    0.59    1.19    0.00   91.00

Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
sda               1.56         0.01         0.01      44144      51244
sdb             146.49         5.48         1.79   19159479    6250758

Finally to see all open files by a process such as data/shared objects/dynamic libraries and sockets use lsof. In the following example we can see all open files of mysql process:

[root@(db-master) ~]# lsof -p 2521
COMMAND  PID  USER   FD   TYPE             DEVICE     SIZE/OFF     NODE NAME
mysqld  2521 mysql  cwd    DIR              254,2         4096  6815769 /monsoon/mysql/data
mysqld  2521 mysql  rtd    DIR              254,0         4096        2 /
mysqld  2521 mysql  txt    REG              254,0    250387936   794500 /usr/sbin/mysqld
mysqld  2521 mysql  mem    REG              254,0        97056  1065145 /lib64/libresolv-2.22.so
mysqld  2521 mysql  mem    REG              254,0        26976  1065107 /lib64/libnss_dns-2.22.so


To see the TCP listening server sockets on a linux server, we can do that with netstat -tulpn

[root@(db-master) ~]# netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:3306            0.0.0.0:*               LISTEN      2521/mysqld
tcp        0      0 0.0.0.0:2738            0.0.0.0:*               LISTEN      3282/discagnt
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      3289/sshd
tcp        0      0 127.0.0.2:25            0.0.0.0:*               LISTEN      3671/master
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      3671/master
tcp        0      0 127.0.0.1:6010          0.0.0.0:*               LISTEN      38622/0
tcp        0      0 :::7938                 :::*                    LISTEN      3317/nsrexecd
tcp        0      0 :::5666                 :::*                    LISTEN      1/systemd
udp     4352      0 0.0.0.0:68              0.0.0.0:*                           1521/wickedd-dhcp4
udp        0      0 10.97.6.160:123         0.0.0.0:*                           3343/ntpd


while for all open TCP sockets:

[root@(db-master) ~]# netstat -t
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 mo-1400a55c2.zone:mysql mo-6740a22da.zone:46138 ESTABLISHED
tcp        0     64 mo-1400a55c2.zone1.:ssh mo-657dabf53.zone:58606 ESTABLISHED
tcp        0      0 mo-1400a55c2.zone:mysql mo-23acddcc0.zone:50068 ESTABLISHED