kernel crash debugging technique using "crash" utility.
Most frequent issue in debugging kernel programs is lack of information; lack of information in terms of stack trace, kernel logs and crash screen-shots. Crash utility provides solution for this. It provides gdb like interface for debugging.
vmlinux object file.
This is kernel image with debugging information. By default, we get compressed kernel image with no debug information. For getting, vmlinux object file, you need to install "kernel-debuginfo" rpm. If you want to debug any module, make sure it is compiled with gcc flag '-g'.
We receive crash utility as rpm.
rpm name is of format: crash-$version.$arch.rpm
Here, $version shows version number and $arch shows system architecture.
For getting crash dump: For getting crash dump, we need to add "crashkernel" option to grub command line. i.e. we can have:
ro root=LABEL=/1 rhgb quiet crashkernel=128M@16M
ro root=LABEL=/1 rhgb quiet
Here, parameter "crashkernel=128M@16M" reserves 128MB of physical memory starting at 16MB. This reserved memory is used to preload and run the capture kernel (i.e. to capture crash dump). This command line option ensures that, whenever crash occurs, it stores crash dump at "/var/crash" and they are stored according to date and time.
There are other options to get crash dump like diskdump, netdump, etc.
Running crash Utility: 1. Debugging last kernel panic:
#crash <patht to vmlinux> /var/crash/crash_dump_name
Watching current running kernel:
This will prompt to crash shell. Just as example, following are commands:
crash> help crash> bt ---> for backtrace crash> log ---> for dumping current system buffer
For more information on this, you can visit at: http://www.redhatmagazine.com/2007/08/15/a-quick-overview-of-linux-kernel-crash-dump-analysis/ http://people.redhat.com/anderson/crash_whitepaper/