Home Download FAQ / Knowledge Base Screenshots Documentation Support


When you're in total trouble, a debugger is the way to help out. It is also the proper tool to report information back to the citadel development team (like Mozilla will do with its talkback facility). There are several instances of debuggers around, mdb / dbx for Solaris for example, but the one we will talk about here is the most common in the open source world: GDB that you can also operate through DDD, KDevelop, emacs, or simple through its command line, which all of the above front ends also offer. We will show some basic operations here, you may want to search the gdb man page if you want to go deeper into detail. Please remember that it's always a good idea to check whether your troubles persist with the newest version of Citadel (after performing a full backup of your data, of course).

Which kind of binaries produce useful output?

By default, the binaries installed to your system are stripped to safe system resources. You can determine whether you have stripped or unstripped binaries like this:

  # file /usr/sbin/citserver 
  /usr/sbin/citserver: ELF 32-bit LSB executable, Intel
  80386, version 1 (SYSV), for GNU/Linux 2.4.1, dynamically
  linked (uses shared libs), stripped

These will produce almost meaningless output in gdb, except maybe the addr2line utility can gather some information from that. However, the better method is to use an unstripped binary (the install process does this), that you will find inside your source tree. If you like to do stepping and not want gdb to jump back and forth, compile it with -ggdb -o0 compiler flags.

Citadel brings you unstripped/citserver, which matches your active citserver. Use this to analyze your core files.

How do I make my system produce core dump files?

On Linux this can be achieved with the ulimit command which is documented in the bash man page.

  ulimit -c 999999

The core files are created in the working directory. Be careful, they can grow and eat up your disk space. Making the whole system create core files on crashes in a central folder can be achieved like this:

# local to citadel by editing /etc/init.d/citadel or /etc/init.d/webcit and add in the second line:
ulimit -c unlimited

# systemwide by doing:
echo 1 > /proc/sys/kernel/core_uses_pid
echo /tmp/core-%e-%p-%t > /proc/sys/kernel/core_pattern

Analyzing Core files

If a program crashes due to illegal commands its state may be conserved into a core dump. You can use this core dump with gdb to find out at which point it ceased operation.

gdb citserver core dump

From now on you can do all data examination you could do if you just stopped the program.

  thread apply all bt

will show you a stack trace of the functions in the citadel binary that were active. If you just want to report the bug, please use the output of

  thread apply all bt full

If you want to save the contents of a pointer, you can do it like this:

dump binary memory /tmp/decoded  decoded &decoded[10000]

would save the found 10k bytes after the place decoded points to.

Running Programs in GDB

Setting a breakpoint can be done with

break main

for example. start it with

run -x7

(which will pass -x7 to the debugged process as commandline parameter) and it will stop at the start. you can make it step forward with


and enter a function with


print variables with

print varname

or on every step with

  display varname
  handle SIGPIPE nostop

will stop gdb from stopping on other programs disconnecting from a citadel server.

Using Valgrind

Nobody's perfect. Valgrind is the most decent tool around these days to detect tiny flaws in programs and identify memory leaks (the process growing constantly because of it forgets about memory it demanded from the system). We use valgrind to ensure the quality of citadel. This is how you can run it:

  valgrind --log-file=/tmp/valgrindlog \
     --show-reachable=yes --leak-check=full citserver 



Sometimes code doesn't work as expected in special situations, a program develops hotspots. Profilers can help here, gprof does in our case. It needs special binaries. Here's how to create them.

If you want to create them via the deb-build, just do

  export DEB_BUILD_OPTIONS="debug profiling"

If you want to configure and compile yourself, do

  CFLAGS = -O0 -ggdb -rdynamic -MD -MP

and add %%-%%-with-gprof to your configure files.


Now you're equipped with those special binaries, run them, produce the high load situation you've experienced, and stop them regularly afterwards, so they can write their 'gmon.out' file containing the profiling raw data. WebCit can be shut down with 'killall -HUP webserver', citserver will finish its job with 'sendcommand down'.

Both of them will write a gmon.out into their working directory (/var/lib/citadel, /usr/local/citadel, /var/run/citadel, or /usr/local/webcit).

There are no social media links here. Enjoy a friendly Citadel community instead. Or go outside.