When you're in total trouble, a debugger is the way to help out. It is also the proper tool to report information back to the citadel development team (like Mozilla will do with its talkback facility). There are several instances of debuggers around, mdb / dbx for Solaris for example, but the one we will talk about here is the most common in the open source world: GDB that you can also operate through DDD, KDevelop, emacs, or simple through its command line, which all of the above front ends also offer. We will show some basic operations here, you may want to search the gdb man page if you want to go deeper into detail. Please remember that it's always a good idea to check whether your troubles persist with the newest version of Citadel (after performing a full backup of your data, of course).
By default, the binaries installed to your system are stripped to safe system resources. You can determine whether you have stripped or unstripped binaries like this:
# file /usr/sbin/citserver /usr/sbin/citserver: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for GNU/Linux 2.4.1, dynamically linked (uses shared libs), stripped
These will produce almost meaningless output in gdb, except maybe the addr2line utility can gather some information from that. However, the better method is to use an unstripped binary (the install process does this), that you will find inside your source tree. If you like to do stepping and not want gdb to jump back and forth, compile it with -ggdb -o0 compiler flags.
Citadel brings you unstripped/citserver, which matches your active citserver. Use this to analyze your core files.
On Linux this can be achieved with the ulimit command which is documented in the bash man page.
ulimit -c 999999
The core files are created in the working directory. Be careful, they can grow and eat up your disk space. Making the whole system create core files on crashes in a central folder can be achieved like this:
# local to citadel by editing /etc/init.d/citadel or /etc/init.d/webcit and add in the second line: ulimit -c unlimited # systemwide by doing: echo 1 > /proc/sys/kernel/core_uses_pid echo /tmp/core-%e-%p-%t > /proc/sys/kernel/core_pattern
If a program crashes due to illegal commands its state may be conserved into a core dump. You can use this core dump with gdb to find out at which point it ceased operation.
gdb citserver core dump
From now on you can do all data examination you could do if you just stopped the program.
backtrace bt thread apply all bt
will show you a stack trace of the functions in the citadel binary that were active. If you just want to report the bug, please use the output of
thread apply all bt full
If you want to save the contents of a pointer, you can do it like this:
dump binary memory /tmp/decoded decoded &decoded[10000]
would save the found 10k bytes after the place decoded points to.
Setting a breakpoint can be done with
break main
for example. start it with
run -x7
(which will pass -x7 to the debugged process as commandline parameter) and it will stop at the start. you can make it step forward with
next
and enter a function with
step
print variables with
print varname
or on every step with
display varname handle SIGPIPE nostop
will stop gdb from stopping on other programs disconnecting from a citadel server.
Nobody's perfect. Valgrind is the most decent tool around these days to detect tiny flaws in programs and identify memory leaks (the process growing constantly because of it forgets about memory it demanded from the system). We use valgrind to ensure the quality of citadel. This is how you can run it:
valgrind --log-file=/tmp/valgrindlog \
--show-reachable=yes --leak-check=full citserver
Sometimes code doesn't work as expected in special situations, a program develops hotspots. Profilers can help here, gprof does in our case. It needs special binaries. Here's how to create them.
If you want to create them via the deb-build, just do
export DEB_BUILD_OPTIONS="debug profiling" dpkg-buildpackage
If you want to configure and compile yourself, do
CFLAGS = -O0 -ggdb -rdynamic -MD -MP
and add %%-%%-with-gprof to your configure files.
Both of them will write a gmon.out into their working directory (/var/lib/citadel, /usr/local/citadel, /var/run/citadel, or /usr/local/webcit).