There is no golden
rule for troubleshooting performance issues. To troubleshoot performance
issues, strategy will depend on the nature of the problem.
- Is it always slow,
- or it is this problem
irregular -
- it appears as suddenly
as it goes away?
In UNIX there are 5
major resource types that need to be monitored and tuned -
CPU
Memory
Network
Disk I/O
Applications
programs.
Check the
application, ask the app owner.
Has anything changed on the end user
side
Is there an increase in workload.
Verify if there are any run away
processes.
Steps:
Always start with creating a
baseline. Use tools like nmon and perfpmr and save a baseline of the
system when it working properly.
As and when changes are made create a new baseline before and
after so you have performance points for comparison.
When you suspect a problem, rerun the baseline to see if
anything is obvious to you. Without a baseline, you have nothing to compare
your problem to, which makes resolution challenging. And, of course, good
change control and documentation are critical.
Step 1 –
Understand (and reproduce) the problem
Step 2 – Monitor and collect data: After
defining the problem clearly, monitor the system and try to collect as much
data as possible on various subsystems. Based on this data, come up list of
potential issues.
Step 3 – Eliminate and narrow down issues After
having a list of potential issues, dive into each one of them and eliminate any
non issues. Narrow it down further to see whether it is an application issue,
or an infrastructure issue. Drill down further and narrow it down to a specific
component. For example, if it is an infrastructure issue, narrow it down and
identify the subsystem that is causing the issue. If it is an I/O subsystem
issue, narrow it down to a specific partition, or raid group, or LUN, or disk.
Basically, keep drilling down until you put your finger on the root cause of
the issue.
Step 4 – One change at a time Once
you’ve narrowed down to a small list of potential issues, don’t try to make
multiple changes at one time. If you make multiple changes, you wouldn’t know
which one fixed the original issue. Multiple changes at one time might also
cause new issues, which you’ll be chasing after instead of fixing the original
issue. So, make one change at a time, and see if it fixes the original problem.
TOP:
The
top command used to dipslay all the running and active real-time processes in
ordered list and updates it regularly. It display CPU
usage, Memory
usage, Swap
Memory, Cache
Size, Buffer Size, Process PID, User, Commands and much more. It also shows high memory and cpu utilization of a running processess.
Vmstat:
Linux VmStat command used to display statistics of virtual
memory, kernerl
threads, disks,system processes, I/O blocks, interrupts, CPU activity and much more. By default vmstat command is not
available under Linux systems you need to install a package called sysstat that includes a vmstat program.
LSOF:
Lsof command used in many Linux/Unix like system that is used to display list of all
the open files and the processes. The open files included are disk
files, network
sockets, pipes, devices andprocesses. One of the main reason for using this command is
when a disk cannot be unmounted and displays the error that files are being
used or opened. With this commmand you can easily identify which files are in
use.
TCPDUMP:
Tcpdump one of the most widely used command-line network
packet analyzer or packets
snifferprogram that is used capture or
filter TCP/IP packets
that received or transferred on a specific interface over a network.
Netstat is a command line tool for monitoring incoming and outgoing network packets statistics as well as interface
statistics. It is very useful tool for every system administrator to monitor
network performance and troubleshoot network related problems.
Htop is a much advanced interactive and real time
Linux process monitoring tool. This is much similar to Linux top
command but it has some rich
features like user friendly interface to manage process, shortcut keys, vertical and horizontal view
of the processes and much more. Htop
is a third party tool and doesn’t included in Linux systems, you need to
install it using YUM package
manager tool.
Iotop is also much similar to top
command and Htop
program, but it has accounting function
to monitor and display real time Disk I/O and processes. This tool is much useful for finding the exact
process and high used disk read/writes of the processes.
IoStat is simple tool that will collect and show
system input and output storage device statistics. This tool is often
used to trace storage device performance issues including devices, local disks,remote disks such
as NFS.
