Performance

There is no golden rule for troubleshooting performance issues. To troubleshoot performance issues, strategy will depend on the nature of the problem.

1. Is it always slow,
2. or it is this problem irregular -
3. it appears as suddenly as it goes away?

In UNIX there are 5 major resource types that need to be monitored and tuned -

CPU

Memory

Network

Disk I/O

Applications programs.

Check the application,  ask the app owner.

        Has anything changed on the end user side

        Is there an increase in workload.

        Verify if there are any run away processes.

Steps:

Always start with creating a baseline. Use tools like nmon and perfpmr and save a baseline of the system when it working properly.

As and when changes are made create a new baseline before and after so you have performance points for comparison.

When you suspect a problem, rerun the baseline to see if anything is obvious to you. Without a baseline, you have nothing to compare your problem to, which makes resolution challenging. And, of course, good change control and documentation are critical.

Step 1 – Understand (and reproduce) the problem

Step 2 – Monitor and collect data:  After defining the problem clearly, monitor the system and try to collect as much data as possible on various subsystems. Based on this data, come up list of potential issues.

Step 3 – Eliminate and narrow down issues After having a list of potential issues, dive into each one of them and eliminate any non issues. Narrow it down further to see whether it is an application issue, or an infrastructure issue. Drill down further and narrow it down to a specific component. For example, if it is an infrastructure issue, narrow it down and identify the subsystem that is causing the issue. If it is an I/O subsystem issue, narrow it down to a specific partition, or raid group, or LUN, or disk. Basically, keep drilling down until you put your finger on the root cause of the issue.

Step 4 – One change at a time Once you’ve narrowed down to a small list of potential issues, don’t try to make multiple changes at one time. If you make multiple changes, you wouldn’t know which one fixed the original issue. Multiple changes at one time might also cause new issues, which you’ll be chasing after instead of fixing the original issue. So, make one change at a time, and see if it fixes the original problem.

TOP:

The top command used to dipslay all the running and active real-time processes in ordered list and updates it regularly. It display CPU usage, Memory usage, Swap Memory, Cache Size, Buffer Size, Process PID, User, Commands and much more. It also shows high memory and cpu utilization of a running processess.

Vmstat:

Linux VmStat command used to display statistics of virtual memory, kernerl threads, disks,system processes, I/O blocks, interrupts, CPU activity and much more. By default vmstat command is not available under Linux systems you need to install a package called sysstat that includes a vmstat program. 

LSOF:

Lsof command used in many Linux/Unix like system that is used to display list of all the open files and the processes. The open files included are disk files, network sockets, pipes, devices andprocesses. One of the main reason for using this command is when a disk cannot be unmounted and displays the error that files are being used or opened. With this commmand you can easily identify which files are in use.

TCPDUMP:

Tcpdump one of the most widely used command-line network packet analyzer or packets snifferprogram that is used capture or filter TCP/IP packets that received or transferred on a specific interface over a network. 

Netstat is a command line tool for monitoring incoming and outgoing network packets statistics as well as interface statistics. It is very useful tool for every system administrator to monitor network performance and troubleshoot network related problems.

Htop is a much advanced interactive and real time Linux process monitoring tool. This is much similar to Linux top command but it has some rich features like user friendly interface to manage process, shortcut keys, vertical and horizontal view of the processes and much more. Htop is a third party tool and doesn’t included in Linux systems, you need to install it using YUM package manager tool.

Iotop is also much similar to top command and Htop program, but it has accounting function to monitor and display real time Disk I/O and processes. This tool is much useful for finding the exact process and high used disk read/writes of the processes.

IoStat is simple tool that will collect and show system input and output storage device statistics. This tool is often used to trace storage device performance issues including devices, local disks,remote disks such as NFS.