So it has happened. One of your users account was hacked, and it has been used to send spam. Since these guys don't care about your system's health, your system is probably under heavy load right now. This is what best to do:
Since you want to be nice to others, and don't want to end up being listed on some RBL DNS stopping the outbound mail stack is the first thing to do. This is only possible via restarting citserver with a special environment varable set:
export CITSERVER_n_session_max=0
Remember this, since you have to remove it after the situation is cleared and you want to resume normal operations.
Now you closed the outbound floodgates, it's time to close the inbound one. Most probably the spammer is sending mails in via SMTP. Closing the SMTP ports (25 / 465 / 587) via your firewall is a good idea, but you also can close them using WebCit: Administration -- Edit site-wide configuration -- SMTP , and now set MTA/MSA/SMTP + SSL to -1; Save it and restart citserver again. If you run another mailer in front of Citadel, it's probably a good idea to stop this one too. If you've got a secondary MX running on another host, stop it too. Now you're offline; which is not as bad as you may think; remote systems will generally keep trying to send messages for several days, so you won't lose legitimate mail which is sent to you by others.
This is probably a good time to tell your users that the mail system is unavailable, and you're working on the situation. Now you need to find out whose account was hacked. Therefore you need to inspect your outbound mail queue.
As an administrator, go to Administration -- View the outbound SMTP queue.
The "Jobs waiting for further processing" table shows you probably a huge list of mail jobs being processed. The "Sender" Column should show you who is the one that was hacked. Contact this person, and tell him he has to clear the situation on his system. Right now its probably a good idea to change his password.
And now you're in the need for a way to delete a huge number of jobs from the queue, right? Since WebCit dynamicaly decides which representation to provide for a room, and the mail queue is just another (hidden) room, you simply need to chose another representation by changing the "view" parameter in the URL in your browser from "11" to "1" and press enter. Now you've got the comfy mailbox view, you will get a better view of the situation. There are two sorts of messages in this room: those with regular subjects, and those with QMSG as subject. It's always pairs of them; the QMSG with the queueing information, and the regular mail. We will assume that none of the messages currently in the queue will be sent. Select all of them and press "Del" to move them into your personal trash. You can later check, whether there are mails from your users that they should send again. Please note that you should respect the privacy of your users.
If your situation is so bad that managing it through WebCit is untenable, we can go a little bit more
low-tech.
Instead of webcit talking to citserver, we write a little shellscript utilizing netcat //(Note: some distros call the Unix-Domain-Socket enabled netcat we need 'openbsd-netcat', test with nc -h whether 'U' is in the possible arguments)//.
**SIDENOTE:** this script will delete messages permanentely, you won't be able to look at them later.
You've probably already read about citadel scripting; more details can be found at the application protocol documentation; eventualy you may want to implement the MOVE command...
Please adjust the location of your unix domain socket according to your installation.
You need to run this script with write permissions to the admin-socket.
#!/bin/bash
# get all message IDs from the spool folder:
(printf 'GOTO __CitadelSMTPspoolout__\nMSGS\n'; sleep 1; ) |nc -U /var/run/citadel/citadel-admin.socket |grep -v '^200 .*' |grep -v '^000$' |grep -v '^100 *$'> /tmp/msgids
# (grep out status-codes etc.)
# check the file /tmp/msgids for validity, then remove this:
exit
# now we know all of them, chunksize items per command.
chunksize=30
count=`wc -l < /tmp/msgids`
chunks=$(($((count/${chunksize})) + 1))
start=${chunksize}
(
printf 'GOTO __CitadelSMTPspoolout__\n'
for i in `seq 1 $chunks`; do
line=''
line=''
messages=`head /tmp/msgids -n${start}|tail -n ${chunksize}`
messages=`head /tmp/msgids -n${start}|tail -n ${chunksize}`
#printf "${messages}"
#printf "${messages}"
for msgid in ${messages}; do
for msgid in ${messages}; do
if test -n "${line}"; then
line=${line}','
fi
line=${line}$msgid
done
done
printf "DELE $line\n"
printf "DELE $line\n"
start=$((${start} + ${chunksize}))
start=$((${start} + ${chunksize}))
done
# after we sent all commands, give citserver a while to catch up,
# before we close the connection.
sleep 1000000
)|nc -U /var/run/citadel/citadel-admin.socket
* removed the spam from your mailqueue * identified & cleared possible infections on the client host * identified & cleared possible infections on the client hostyou need to * configure the new password on the client and tell the user his new password for the web login (or better let him choose a new one which mustn't be the old one)* configure the new password on the client and tell the user his new password for the web login (or better let him choose a new one which mustn't be the old one) * educate you user about possible attack vectors, and that his credentials are possibly burned, and he should change passwords on other accounts * educate you user about possible attack vectors, and that his credentials are possibly burned, and he should change passwords on other accounts * re-enable the inbound SMTP-Ports * re-enable the inbound SMTP-Ports * remove the outbound mailblock setting from your initscript * remove the outbound mailblock setting from your initscript * restart citserver * restart citserver * send a test mail! * send a test mail! * check whether you made it into some RBL, and clear the situation with them; else some remote hosts may reject mails from you as spam... * check whether you made it into some RBL, and clear the situation with them; else some remote hosts may reject mails from you as spam...