Today I Learned

Find out which Python program did a core dump

Thu Dec 22, 2022 (Updated on Fri Aug 25, 2023)

This morning, while casually looking at my syslogs, I saw a big red stacktrace about a Python process. I got curious and wanted to look at which script crashed like that. However, the name of the faulty script was nowhere to be found in the trace.

09:48:24 kernel: python3[2550364]: segfault at 0 ip 00007fd9f65cb8d9 sp 00007fffc69143d0 error 4 in libgtk-3.so.0.2404.30[7fd9f65ad000+382000]
09:48:24 systemd-coredump[2563786]: [/] Process 2550364 (python3) of user 1000 dumped core.
                                    Module /home/romain/.local/lib/python3.10/site-packages/Pillow.libs/libXau-154567c4.so.6.0.0 without build-id.
                                    Module /home/romain/.local/lib/python3.10/site-packages/Pillow.libs/liblzma-96284f0d.so.5.2.5 without build-id.
                                    Module /home/romain/.local/lib/python3.10/site-packages/Pillow.libs/libtiff-f706b4a5.so.5.8.0 without build-id.
                                    Module /home/romain/.local/lib/python3.10/site-packages/Pillow.libs/libopenjp2-8fa4ced9.so.2.5.0 without build-id.
                                    Module /home/romain/.local/lib/python3.10/site-packages/Pillow.libs/libjpeg-a4c3d5e9.so.62.3.0 without build-id.
                                    [...]
                                    Stack trace of thread 2550400:
                                    #0  0x00007fd9f8e250bf __poll (libc.so.6 + 0xfb0bf)
                                    #1  0x00007fd9e8de08c7 n/a (libpulse.so.0 + 0x338c7)
                                    #2  0x00007fd9e8dca46c pa_mainloop_poll (libpulse.so.0 + 0x1d46c)
                                    #3  0x00007fd9e8dd442c pa_mainloop_iterate (libpulse.so.0 + 0x2742c)
                                    #4  0x00007fd9e8dd44e1 pa_mainloop_run (libpulse.so.0 + 0x274e1)
                                    #5  0x00007fd9e8de4c02 n/a (libpulse.so.0 + 0x37c02)
                                    #6  0x00007fd9e8d81c57 n/a (libpulsecommon-16.1.so + 0x5bc57)
                                    #7  0x00007fd9f8db08fd n/a (libc.so.6 + 0x868fd)
                                    #8  0x00007fd9f8e32a60 n/a (libc.so.6 + 0x108a60)
                                    ELF object binary architecture: AMD x86-64

To get an idea of what went wrong, I needed the script path and other args given to Python. Looking at the logs, we can see the PID of the crashed process: 2550364. As I just want the basic info, coredumpctl info <PID> is just fine. But if I needed to do a deep dive into the stacktrace, I could have used coredumpctl debug <PID> to open a debugger and trace the problem back to its origin.

coredumpctl info 2550364 | head

             PID: 2550364 (python3)
             UID: 1000 (romain)
             GID: 1001 (romain)
          Signal: 11 (SEGV)
       Timestamp: Thu 2022-12-22 09:48:24 CET (1h 32min ago)
    Command Line: python3 /usr/bin/lutris
      Executable: /usr/bin/python3.10
   Control Group: /user.slice/user-1000.slice/session-1.scope
            Unit: session-1.scope
           Slice: user-1000.slice

The culprit is in the Command Line field: Lutris! I’m surprised as I didn’t have any problem this morning. I suspect it encountered errors when I closed it and it tried to cleanup its resources. As long as it doesn’t crash when I’m actively using it, I don’t feel the need to investigate more. It’s certainly nice to know which program it was!

Here is a helper command to inspect the 3 latest coredumps :

ls -t /var/lib/systemd/coredump/ |                      # List the coredumps, most recent first
  head -n 3 |                                           # Keep only 3
  cut -d. -f 5 |                                        # Get the PID from the filename
  xargs -I % bash -c "echo; coredumpctl info % | head"  # Print the info