Analyzing application crash in CentOS Linux.

By | October 29, 2017

To diagnose program crashing failure let us create our own simple c++ application which will be crashing during execution.
Example of crashing application, c++
Save it as exemplo.cpp.
Build executable without debug information:
g++ -o exemplo exemplo.cpp
When we start it we receive information that application crashed because of segmentation fault:


./exemplo
Crashing application
Segmentation fault (core dumped)

The simplest way is to analyze crash starting the application under gdb debugger. What to do if gdb is not install and you do not want to install it and it is the third party implementation?
In this case is crash reporter tool. For CentOS/RehHat/Fedora open /etc/abrt/abrt-action-save-package-data.conf in text editor, change value of ProcessUnpackaged to “yes”

ProcessUnpackaged = yes

Start exemplo application and when it crashes, examine /var/log/messages log file, find the message related to exemplo application crash, it should look like this:

Oct 24 13:23:07 centos6-64 kernel: exemplo[24139]: segfault at 0 ip 00000037104898e1 sp 00007ffe6b675bf8 error 6 in libc-2.12.so[3710400000+18a000]
Oct 24 13:23:07 centos6-64 abrtd: Directory 'ccpp-2017-10-24-13:23:07-24139' creation detected
Oct 24 13:23:07 centos6-64 abrt[24140]: Saved core dump of pid 24139 (/projects/exemplo) to /var/spool/abrt/ccpp-2017-10-24-13:23:07-24139 (401408 bytes)

The /var/log/messages shows that crash dump information save into /var/spool/abrt/ccpp-2017-10-24-13:23:07-24139 directory. This directory contains a lot of useful for analyzing information:

# ls -l /var/spool/abrt/ccpp-2017-10-24-13:23:07-24139
total 2268
-rw-r-----. 1 root abrt 5 Oct 24 13:23 abrt_version
-rw-r-----. 1 root abrt 4 Oct 24 13:23 analyzer
-rw-r-----. 1 root abrt 6 Oct 24 13:23 architecture
-rw-r-----. 1 root abrt 0 Oct 24 13:23 cgroup
-rw-r-----. 1 root abrt 9 Oct 24 13:23 cmdline
-rw-r-----. 1 root abrt 606 Oct 24 13:23 core_backtrace
-rw-r-----. 1 root abrt 401408 Oct 24 13:23 coredump
-rw-r-----. 1 root abrt 1 Oct 24 13:23 count
-rw-r-----. 1 root abrt 366 Oct 24 13:23 dso_list
-rw-r-----. 1 root abrt 1990 Oct 24 13:23 environ
-rw-r-----. 1 root abrt 0 Oct 24 13:23 event_log
-rw-r-----. 1 root abrt 22 Oct 24 13:23 executable
-rw-r-----. 1 root abrt 22 Oct 24 13:23 hostname
-rw-r-----. 1 root abrt 7 Oct 24 13:23 kernel
-rw-r-----. 1 root abrt 10 Oct 24 13:23 last_occurrence
-rw-r-----. 1 root abrt 1323 Oct 24 13:23 limits
-rw-r-----. 1 root abrt 93 Oct 24 13:23 machineid
-rw-r-----. 1 root abrt 2430 Oct 24 13:23 maps
-rw-r-----. 1 root abrt 105 Oct 24 13:23 open_fds
-rw-r-----. 1 root abrt 26 Oct 24 13:23 os_release
-rw-r-----. 1 root abrt 5 Oct 24 13:23 pid
-rw-r-----. 1 root abrt 14 Oct 24 13:23 pwd
-rw-r-----. 1 root abrt 64 Oct 24 13:23 reason
-rw-r-----. 1 root abrt 2026668 Oct 24 13:23 sosreport.tar.xz
-rw-r-----. 1 root abrt 10 Oct 24 13:23 time
-rw-r-----. 1 root abrt 4 Oct 24 13:23 type
-rw-r-----. 1 root abrt 1 Oct 24 13:23 uid
-rw-r-----. 1 root abrt 5 Oct 24 13:23 username
-rw-r-----. 1 root abrt 40 Oct 24 13:23 uuid
-rw-r-----. 1 root abrt 1860 Oct 24 13:23 var_log_messages

Open core_backtrace file and you find crash stack:

# cat /var/spool/abrt/ccpp-2017-10-24-13:23:07-24139/core_backtrace
{   "signal": 11
,   "executable": "/Alex/projects/example"
,   "stacktrace":
      [ {   "crash_thread": true
        ,   "frames":
              [ {   "address": 236496394465
                ,   "build_id": "f14f3f5b55fc325fdfa7fb6f22f73ca1c6977379"
                ,   "build_id_offset": 563425
                ,   "function_name": "memcpy"
                ,   "file_name": "/lib64/libc.so.6"
                }
              , {   "address": 4196026
                ,   "build_id": "9dcd0db2fdf3c058dea3f41478b3487be91131df"
                ,   "build_id_offset": 1722
                } ]
        } ]
}

Pay attention the g++ compiler substitutes strcpy function with memcpy.

Leave a Reply

Your email address will not be published. Required fields are marked *