The basic tool here is
top
Monitoring a single process can be done with -p option, in the next example we measure the MySQL process:
[root@(db-master) ~]# top -p 2521
top - 15:42:54 up 40 days, 10:46, 4 users, load average: 0.14, 0.24, 0.48
Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie
%Cpu0 : 1.0 us, 1.0 sy, 0.0 ni, 98.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu1 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem: 32551020 total, 32285684 used, 265336 free, 149660 buffers
KiB Swap: 3129340 total, 402572 used, 2726768 free. 16662620 cached Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2521 mysql 20 0 18.725g 0.014t 4548 S 6.000 46.50 2735:03 mysqld
Load Average is a linux/unix mystery: Linux load averages are "system load averages" that show the running thread (task) demand on the system as an average number of running plus waiting threads. This measures demand, which can be greater than what the system is currently processing.
On the other hand good old ps which is available on all UNIX flavors and LINUX distributions can also help. The following command shows the most CPU consuming processes in ascending order along with their virtual size
[root@(db-master) ~]# ps -e -o pid,pcpu,vsz,comm= | sort -n --key=3
...
1669 0.0 752396 isecespd
1759 0.0 1561472 isectpd
2521 52.4 19634584 mysqld
To get the process tree try pstree -aAl:
[root@(db-master) ~]# pstree -aAl
systemd --switched-root --system --deserialize 24
|-VGAuthService -s
|-agetty --noclear tty1 linux
|-automount -p /var/run/automount.pid
| `-5*[{automount}]
|-cron -n
|-dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
|-discagnt /etc/init.d/discagnt start
| `-discagnt
|-haveged -w 1024 -v 0 -F
...
For systems that do not have
pstree try
ps -ejH
To get information about threads created by processes try
ps -eLf
To get information about disk performance try
iostat:
[root@(mmcp_prod_corp)(db-master) ~]# iostat -dcm
Linux 4.4.121-92.117-default (mo-1400a55c2) 10/17/19 _x86_64_ (8 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
7.22 0.00 0.59 1.19 0.00 91.00
Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
sda 1.56 0.01 0.01 44144 51244
sdb 146.49 5.48 1.79 19159479 6250758
Finally to see
all open files by a process such as data/shared objects/dynamic libraries and sockets use
lsof. In the following example we can see all open files of mysql process:
[root@(db-master) ~]# lsof -p 2521
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
mysqld 2521 mysql cwd DIR 254,2 4096 6815769 /monsoon/mysql/data
mysqld 2521 mysql rtd DIR 254,0 4096 2 /
mysqld 2521 mysql txt REG 254,0 250387936 794500 /usr/sbin/mysqld
mysqld 2521 mysql mem REG 254,0 97056 1065145 /lib64/libresolv-2.22.so
mysqld 2521 mysql mem REG 254,0 26976 1065107 /lib64/libnss_dns-2.22.so
To see the TCP listening server sockets on a linux server, we can do that with
netstat -tulpn
[root@(db-master) ~]# netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN 2521/mysqld
tcp 0 0 0.0.0.0:2738 0.0.0.0:* LISTEN 3282/discagnt
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 3289/sshd
tcp 0 0 127.0.0.2:25 0.0.0.0:* LISTEN 3671/master
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 3671/master
tcp 0 0 127.0.0.1:6010 0.0.0.0:* LISTEN 38622/0
tcp 0 0 :::7938 :::* LISTEN 3317/nsrexecd
tcp 0 0 :::5666 :::* LISTEN 1/systemd
udp 4352 0 0.0.0.0:68 0.0.0.0:* 1521/wickedd-dhcp4
udp 0 0 10.97.6.160:123 0.0.0.0:* 3343/ntpd
while for all open TCP sockets:
[root@(db-master) ~]# netstat -t
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 mo-1400a55c2.zone:mysql mo-6740a22da.zone:46138 ESTABLISHED
tcp 0 64 mo-1400a55c2.zone1.:ssh mo-657dabf53.zone:58606 ESTABLISHED
tcp 0 0 mo-1400a55c2.zone:mysql mo-23acddcc0.zone:50068 ESTABLISHED
