Whence you're in total trouble, a debugger is the way to help out. It is also the proper tool to report information back to the citadel development team (like Mozilla will do with its talkback facility). There are several instances of debuggers around, mdb / dbx for Solaris for example, but the one we will talk about here is the in the open source world most used GDB that you can also operate through DDD, KDevelop, Emacs or plain through its command line, which all of the above front ends also offer. We will show some basic operations here, you may want to search the gdb man page if you want to go deeper into detail.
Please remember that its always a good idea to check whether your troubles persist with the SVN Head revision (after doing a backup of your data).
By default your binaries installed to your system are stripped to safe system resources. You can find it out like that:
# file /usr/sbin/citserver /usr/sbin/citserver: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for GNU/Linux 2.4.1, dynamically linked (uses shared libs), stripped
These will produce allmost meaningless output in gdb, just maybe the addr2line utility can gather some information from that. However, the better method is to use an unstripped binary (the install process does this), that you will find inside your source tree. If you like to do stepping and not want gdb to jump back and forth, compile it with -ggdb -o0 compiler flags.
Citadel brings you untripped/citserver, which matches your active citserver. Use this to analyse your core files.
On Linux this can be achieved with the ulimit command which is documented in the bash man page.
ulimit -c 999999
The core files are created in the working directory. Be careful, they can grow and eat up your disk space. Making the whole system create core files on crashes in a central folder can be achieved like that:
- local to citadel by editing /etc/init.d/citadel or /etc/init.d/webcit and add in the second line:
ulimit -c unlimited
- systemwide by doing:
echo 1 > /proc/sys/kernel/core_uses_pid echo /tmp/core-%e-%p-%t > /proc/sys/kernel/core_pattern
If a program crashes due to illegal commands its state may be conserved into a core-dump. You can use this core dump with gdb to find out at which point it ceased operation.
gdb citserver core dump
From now on you can do all data examination you could do if you just stopped the program.
backtrace bt thread apply all bt
will show you a stack trace of the functions in the citadel binary that were active. If you just want to report the bug, please use the output of
thread apply all bt full
If you want to save the contents of a pointer, you can do it like that:
dump binary memory /tmp/decoded decoded &decoded
would save the found 10k bytes after the place decoded points to.
Setting a breakpoint can be done with
for example. start it with
(which will pass -x7 to the debugged process as commandline parameter) and it will stop at the start. you can make it step forward with
and enter a function with
print variables with
or on every step with
handle SIGPIPE nostop
will stop gdb from stopping on other programs disconnecting from a citadel server.
Nobody's perfect. Valgrind is the most decent tool around these days to detect tiny flaws in programs and bust memory leaks. (the process growing constantly because of it forgets about memory it demanded from the system) We use valgrind to ensure the quality of citadel. This is how you can run it:
valgrind --log-file=/tmp/valgrindlog \ --show-reachable=yes --leak-check=full citserver
Sometimes code doesn't work as expected in special situations, a program develops hotspots. Profilers can help here, gprof does in our case. It needs special binaries, heres how to create them: If you want to create them via the deb-build, just do
export DEB_BUILD_OPTIONS="debug profiling" dpkg-buildpackage
If you want to configure and compile yourself, do
CFLAGS = -O0 -ggdb -rdynamic -MD -MP
and add --with-gprof to your configure files
Now you're equipped with those special binaries, run them, produce the high load situation you've experienced, and stop them regularly afterwards, so they can write their 'gmon.out' file containing the profiling raw data. WebCit can be shut down with 'killall -HUP webserver', citserver will finish its job with 'sendcommand down'.
Both of them will write a gmon.out into their working directory: