Posts for ubuntu

Setting-up PAM and LDAP

by Sebastien Mirolo on Tue, 15 May 2012

The Plan

I wanted to setup my web app to authenticate through PAM as a general authentication mechanism. Since users are allowed to register and update password through the web app directly, I indented to use LDAP to hold user profile information. So I delve into setting-up PAM and LDAP.

As it turns out, libpam-ldapd is used in conjunction with posixAccount objectClasses DNs. I wanted my user DNs to solely require inetOrgPerson and thus in the end, I authenticated to LDAP directly. None-the-less here are my notes on setting-up PAM and LDAP.

Enabling PAM debugging

Debugging authentication setups is often a pain. Many times you get an "authentication failed" error message with little help on what went wrong. First thing is thus to figure out how to enable debug messages in the system logs.

# Enable debugging for all modules:
$ touch /etc/pam_debug

# Adding "debug" at the end of lines in pam.d/*app* files, ex:
$ diff -u prev /etc/pam.d/common-auth
-auth    required
+auth    required debug

# Logging of debug messages in syslog
$ touch /var/log/debug.log
$ chmod 666 /var/log/debug.log 
$ diff -u prev /etc/rsyslog.conf
+*.debug /var/log/debug.log
$ service rsyslog restart

using LDAP for PAM authentication

First we need to populate our LDAP server as done in a previous post.

# To find out what is associated to cn=config.
$ sudo ldapsearch -Y EXTERNAL -H ldapi:/// -b cn=config | grep '^#'
# {0}core, schema, config
# {1}cosine, schema, config
# {2}nis, schema, config
# {3}inetorgperson, schema, config
$ ldapadd -Y EXTERNAL -H ldapi:/// \
  	   -f /etc/ldap/slapd.d/
$ ldapadd -x -W -D cn=admin,dc=example,dc=com \
  		  -f /etc/ldap/slapd.d/

# Add a user account
$ cat john.doe.ldif
dn: uid=john,ou=people,dc=example,dc=com
objectClass: inetOrgPerson
objectClass: posixAccount
objectClass: shadowAccount
uid: john
sn: Doe
givenName: John
cn: John Doe
displayName: John Doe
uidNumber: 1001
gidNumber: 1001
userPassword: password
gecos: John Doe
loginShell: /bin/bash
homeDirectory: /home/john
shadowExpire: -1
shadowFlag: 0
shadowWarning: 7
shadowMin: 8
shadowMax: 999999
shadowLastChange: 10877
$ ldapadd -x -W -D cn=admin,dc=example,dc=com -f john.doe.ldif

# list all people in LDAP database
$ ldapsearch -x -W -D cn=admin,dc=example,dc=com \
  			 -b ou=people,dc=example,dc=com

$ getent passwd

There are two packages that seem to fit the bill of setting PAM with LDAP. Following advices in here, I went with libpam-ldapd.

$ aptitude search ldap | grep pam
p   libpam-ldap                     - Pluggable Authentication Module for LDAP  
p   libpam-ldapd                    - PAM module for using LDAP as an authentica

$ aptitude install libpam-ldapd
$ cat /etc/nslcd.conf
uri ldap://
base dc=example,dc=com

$ grep -rl ldap /etc/pam.d/

$ diff -u prev /etc/nsswitch.conf
-passwd:         compat
-group:          compat
-shadow:         compat
+passwd:         compat ldap
+group:          compat ldap
+shadow:         compat ldap

$ sudo /etc/init.d/nscd restart
$ sudo /etc/init.d/nslcd restart

# Enabling ldap debugging (in case)
$ diff -u prev /etc/default/slapd 
+LAPD_OPTIONS="-d 1 -s 1"

Packaging Python Apps

by Sebastien Mirolo on Sun, 22 Apr 2012

I have never quite understood why python (or ruby) packages are delivered through their own manager (pip, gem) instead of the local system package manager (apt, yum, etc). I mean I understand the python developer rationale and disagrees with it. My main problem with side-stepping the local package manager is that now the programming stack of prerequisites matters, a lot. It is not just about dependencies and APIs anymore but also about which package manager to use. That creates tons of problems especially when you rely on something with some very problematic design issues like easy_install.

If you are interested to learn more, here is a great post about the subject.

Starting with Condor

by Sebastien Mirolo on Fri, 2 Mar 2012

Condor is a job scheduler used to do batch processing on a cluster of machines. dagman is built on top of condor to manage jobs dependencies. The manual is pretty good and after you read through it a few times you will surely want to bookmark the condor_submit reference page to quickly find the Submit Description File Commands. I will just go through the issues I stumbled upon as a newbie.

Condor is available as both an Ubuntu and Fedora package. I made the mistake to start with condor on Ubuntu (this post). It wasn't long before I realize Fedora rides the latest condor version. Fedora also provides a lot more packages not available on Ubuntu that tie up with condor.

After you install condor, you are ready to write your first job description file and run the job. Running DAGs of jobs is not much more difficult. The only thing to remember is start with a vanilla universe. That will run the jobs "rsh-like".

$ condor_submit jobfile
$ condor_submit_dag dagfile

The first problems I encountered were all related to authentication. For some reasons, the condor tools would resolve my hostname two different ways and complain with messages like (see files in /var/lib/condor/log):

ERROR: Failed to connect to local queue manager
AUTHENTICATE:1002:Failure performing handshake

OfflineCollectorPlugin::configure: no persistent store was defined

PERMISSION DENIED to unauthenticated user from host ... for command 48 (QUERY_ANY_ADS), access level READ: reason: READ authorization policy contains no matching ALLOW entry for this request; identifiers used for this host: ...

condor_read(): recv() returned -1, errno = 104, assuming failure reading 5 bytes from unknown source.

The logs are a good way to find out what is going on. Other useful commands are

$ condor_version
$ condor_config_val FULL_HOSTNAME
$ condor_config_val SHADOW_LOG
$ condor_status -long -debug
$ condor_q -analyze
$ condor_q -better-analyze

The configuration file (/etc/condor/condor_config) is often where you will have to make changes to fix things up. For example, I disabled authentication for now in order to make progress on what I cared about: running jobs remotely to completion. Note that setting predefined macros (like FULL_HOSTNAME) will have no effect.

$ diff -u /etc/condor/condor_config

In some situation jobs will match but they do not run. Take a look in the condor manual 7.7.6, Section 2.6.5 - Why is a job not running? to see if you have a swap space issue. Otherwise you can also try the following commands in order to get a clue.

$ condor_config_val LOG
$ grep -r job_id /var/lib/condor/log/
$ condor_q -ana -l job_id 

There are also a bunch of useful commands that come handy in trial/error mode. These include starting the condor daemons, reconfiguring them, releasing jobs on hold and removing jobs from the queue.

$ condor_restart -all
$ condor_reconfig -all
$ condor_on
$ condor_release jobid
$ condor_rm jobid

Later I also found this post very useful to debug condor submit issues.

With massive cloud infrastructure popping up everywhere, we can even say job scheduling becomes a rather crowded town. Condor is free, available in the Fedora repo and has worked quite reliably for some time. It is definitely worth looking at. Alternatives include Simple Linux Utility for Resource Management (SLURM) and many more less known projects.


After switching to Fedora and installing condor 7.7.3, here the commands I ran to be back up developing code.

$ diff -u prev /etc/condor/condor_config
@@ -243,8 +243,8 @@
 ##    ALLOW_WRITE = *
 ##  but note that this will allow anyone to submit jobs or add
 ##  machines to your pool and is a serious security risk.
 #ALLOW_WRITE = *.your.domain, your-friend's-machine.other.domain
 #DENY_WRITE = bad-machine.your.domain

# systemctl enable condor.service
$ systemctl start condor.service

Backups with rsnapshot

by Sebastien Mirolo on Sat, 22 Oct 2011

First I created a cron job to dump the databases running on production servers in SQL format.

# On the target machine
$ mkdir -p /var/cache/sql
$ cat /etc/cron.d/backups
# postgresql
30 3          * * *           root /usr/bin/pg_dump -U username -w -f /var/cache/sql/database.sql database
# mysql
30 3          * * *           root /usr/bin/mysqldump --databases database -u root --password=rootpasswd > /var/cache/sql/database.sql
$ chmod 600 /etc/cron.d/backups

On another (backup) machine, I ran rsnapshot.

# On the backup machine
$ apt-get install rsnapshot
$ ssh-keygen -q -t rsa -P "" -f /etc/ssl/private/backup_rsa
$ diff -u prev /etc/rsnapshot.conf
-#cmd_cp		/bin/cp
+cmd_cp		/bin/cp
-#cmd_ssh	/usr/bin/ssh
+cmd_ssh	/usr/bin/ssh
-#ssh_args	-p 22
+ssh_args	-i /etc/ssl/private/backup_rsa

+backup	backup@hostname:/etc/	hostname/
+backup	backup@hostname:/var/log/	hostname/
+backup	backup@hostname:/var/www/	hostname/
+backup	backup@hostname:/var/cache/sql/	hostname/

At first, I was thinking to use a duplicate account for root (uid=0,gid=0) with a shell restricted to scponly as suggested in rsnapshot HOWTO

# On the target machine
$ apt-get install scponly
$ useradd -o --uid 0 --gid 0 --shell /usr/bin/scponly backup
$ mkdir -p /home/backup/.ssh
$ echo "PUBKEYDATA" >>  /home/backup/.ssh/authorized_keys
$ chmod 644 /home/backup/.ssh/authorized_keys
$ chown -R backup /home/backup/.ssh
$ chmod 700 /home/backup/.ssh

Unfortunately the first attempt to connect as backup is rejected because I have disable root logins through sshd for security reasons.

# On the backup machine
$ rsync -Raz --rsh="ssh -i /etc/ssl/private/backup_rsa" \
  		backup@hostname:/etc .

Since I am not to permit root logins through sshd, I need to find another solution. I decided to enable sudo with no password for backup when executing /usr/bin/rsync. It seems less of a security risk as backup does not have to be a well-known username.

Note: Whenever editing the /etc/sudoers file, always make sure to have two shell connection; one with the /etc/sudoers file open and one where you can test sudo commands. If there are a syntax error in the /etc/sudoers and root logins are disabled, you might just get stuck and resort to safe boot voodoo to get out of the mess.

# On the target machine
$ useradd backup
$ mkdir -p /home/backup/.ssh
$ echo "PUBKEYDATA" >>  /home/backup/.ssh/authorized_keys
$ chmod 644 /home/backup/.ssh/authorized_keys
$ chown -R backup:backup /home/backup/.ssh
$ chmod 700 /home/backup/.ssh

$ diff prev /etc/sudoers
-Defaults requiretty
+#Defaults requiretty

+backup   ALL=(ALL:ALL) NOPASSWD: /usr/bin/rsync
# On the backup machine
$ diff -u prev /etc/rsnapshot.conf
-#rsync_long_args	--delete --numeric-ids --relative --delete-excluded
+rsync_long_args	--delete --numeric-ids --relative \
					--delete-excluded --rsync-path "sudo /usr/bin/rsync"

$ rsnapshot -t hourly

Authentication using OpenLDAP

by Sebastien Mirolo on Sat, 8 Oct 2011

In the most part I followed the ubuntu 11.04 openldap tutorial. The wikipedia article is also useful to understand some of the basics. I later stumbled upon LDAP for Rocket Scientists which definitely helped clarify some.

$ apt-get install slapd ldap-utils

Getting the appropriate LDAP setup and running ldapadd with the appropriate command line arguments has been challenging. Depending on the combination of flags, I ran in different error messages such as:

ldap_sasl_interactive_bind_s: Can't contact LDAP server (-1)
# Check /etc/default/slapd

ldap_sasl_interactive_bind_s: No such attribute (16)

ldap_add: Strong(er) authentication required (8)
	additional info: modifications require authentication
# Use -D and -W command line options

Ubuntu 11.04 comes with OpenLDAP 2.4. It took a while to google through to fact that /etc/ldap/sldap.conf is deprecated and the way to configure LDAP is now through the /etc/ldap/sldap.d. The major headache was that ldapadd insisted on me providing a password I never set nor was aware of. A grep for password into /etc/ldap/slapd.d/ did not help reveal what the default might be either. Finally the light came through the following article: switch to dynamic config backend.

$ slappasswd
New password: 
Re-enter new password: 
$ diff -u prev "/etc/ldap/slapd.d/cn=config/olcDatabase={1}hdb.ldif"
  olcRootDN: cn=admin,dc=localdomain
+ olcRootPW: {SSHA}aWHEjvHchgtWH97Pz4PwAQu/yH+1RLnd
$ diff -u prev "/etc/ldap/slapd.d/cn=config/olcDatabase={0}hdb.ldif"
-olcAccess: {0}to * by dn.exact=gidNumber=0+uidNumber=0,cn=peercred,cn=external
- ,cn=auth manage by * break
+olcAccess: {0}to *  by * none
+olcAddContentAcl: TRUE
+olcLastMod: TRUE
+olcMaxDerefDepth: 15
+olcReadOnly: FALSE
+olcRootDN: cn=admin,cn=config
+olcRootPW: {SSHA}aWHEjvHchgtWH97Pz4PwAQu/yH+1RLnd
+olcSyncUseSubentry: FALSE
+olcMonitoring: FALSE
$ diff -u prev "/etc/ldap/slapd.d/cn=config/cn=module{0}.ldif"
-creatorsName: cn=config
-modifiersName: cn=config
+creatorsName: cn=admin,cn=config
+modifiersName: cn=admin,cn=config

$ ldapadd -x -H ldap:/// -f /etc/ldap/schema/cosine.ldif \
  		  -D "cn=admin,cn=config" -W
$ ldapadd -x -H ldap:/// -f /etc/ldap/schema/nis.ldif \
  		  -D "cn=admin,cn=config" -W
$ ldapadd -x -H ldap:/// -f /etc/ldap/schema/inetorgperson.ldif \
  		  -D "cn=admin,cn=config" -W
$ ldapadd -x -H ldap:/// -f backend.domain.ldif -D "cn=admin,cn=config" -W
$ ldapadd -x -H ldap:/// -f frontend.domain.ldif -D "cn=admin,dc=domain,dc=com" -W
$ ldapsearch -xLLL -b "dc=domain,dc=com" uid=login sn givenName cn

Adding entries as ldif files can be quite cumbersome so I hoped to make my life simpler by using but the link to download the script is broken. I also hoped to install web2ldap but that ended with a python exception.

$ apt-get install python-weblib python-ldap python-pyasn1 
# $ apt-get install python-dns python-imaging
$ wget
$ tar zxvf web2ldap-1.1.0a41.tar.gz
$ pushd web2ldap
$ sbin/
from ldap.controls import ValueLessRequestControl,AssertionControl,AuthorizationIdentityControl
ImportError: cannot import name ValueLessRequestControl

Michael Ströder pointed me to Installing web2ldap on Debian. The problem is that I had installed an outdated version of python-ldap through ubuntu package manager. Recent web2ldap versions need newer modules. So I followed the instructions and installed the latest versions of python modules through easy_install instead of apt-get as described in the previous link.

$ /usr/bin/python -V
Python 2.7.1+
$ apt-get install build-essential python-dev python-setuptools
$ apt-get install libsasl2-dev libldap-dev 
$ easy_install python-ldap pyweblib pyasn1 pyasn1_modules
# Optional
$ apt-get install python-imaging
$ easy_install pydns pyexcelerator
$ wget
$ tar zxvf web2ldap-1.1.0a51.tar.gz
$ pushd web2ldap-1.1.0a51
$ python sbin/
diff -u prev etc/web2ldap/web2ldapcnf/
-#access_allowed = ['']
-access_allowed = ['']
+access_allowed = ['']
+#access_allowed = ['']
$ python sbin/ -l -d off

I was then able to access the web2ladp web interface and start browsing through it.

A search through ubuntu packages reports ldap-account-manager and ldaptor-webui so I might try them at some point.

Denying comment spam bots

by Sebastien Mirolo on Sat, 23 Apr 2011

It is kind of fun to look through your application logs and find traces of a hacker trying to break in. It might even be intellectually stimulating to play this game of hide and seek with another human being. Unfortunately most malicious attempts hitting your server will come from bots. Those don't get discouraged. Those don't change tactics. They keep trying to brute force passwords, even when you only allow private key login in your ssh daemon. They keep trying to access PHP scripts, even when you do not have any PHP stack running on your web server. Worse, if you allow people to leave comments on your web site, you are almost guarantee to attract spam bots that will waste precious bandwidth and mess up statistics you use to learn about your audience.

You could hire an army of private investigators traveling around the world to unplug those bot machines. It might actually be a very cool job (I would definitely apply for it). You might even think to pull off a good scenario ala "Blade Runner" to sell to Hollywood to offset the investigation cost.

For practical matters (or just because you are not huge on adventure around the world), you might want to try setting-up iptables, fail2ban and spamassassin. The idea is to use spamassassin to categorize comments as spam or not, use fail2ban to dynamically insert rules into the firewall when an IP definitely generates too much spam and of course use iptables to prevent those machines to reach your application stack.


Iptables comes pre-installed on all official Ubuntu distributions but unfortunately it does not come with logging enabled by default. Since we will want to verify our setup works and drops packets from banned addresses, first thing is to enable iptables logging. I also enjoyed reading Linux Firewalls Using iptables for generic information.

$ ls /etc/iptables.*
$ grep -r 'iptables.conf' /etc
/etc/network/if-up.d/load-iptables:iptables-restore < /etc/iptables.conf
$ diff -U 1 /etc/iptables.conf.prev /etc/iptables.conf
--- /etc/iptables.conf.prev	2011-03-15 23:48:03.000000000 +0000
+++ /etc/iptables.conf	2011-03-16 00:27:30.000000000 +0000
@@ -3,2 +3,3 @@
+:LOGNDROP - [0:0]
@@ -14,2 +15,7 @@
+-A LOGNDROP -p tcp -m limit --limit 5/min -j LOG\
  --log-prefix "Denied TCP: " --log-level 7
+-A LOGNDROP -p udp -m limit --limit 5/min -j LOG\
  --log-prefix "Denied UDP: " --log-level 7
+-A LOGNDROP -p icmp -m limit --limit 5/min -j LOG\
  --log-prefix "Denied ICMP: " --log-level 7
$ iptables -N LOGNDROP
$ iptables -A INPUT -j LOGNDROP
$ iptables -A LOGNDROP -p tcp -m limit --limit 5/min -j LOG\
  --log-prefix "Denied TCP: " --log-level 7
$ iptables -A LOGNDROP -p udp -m limit --limit 5/min -j LOG\
  --log-prefix "Denied UDP: " --log-level 7
$ iptables -A LOGNDROP -p icmp -m limit --limit 5/min -j LOG\
  --log-prefix "Denied ICMP: " --log-level 7
$ iptables -A LOGNDROP -j DROP
$ iptables -L
Chain INPUT (policy DROP)
LOGNDROP   all  --  anywhere             anywhere            
Chain LOGNDROP (1 references)
target     prot opt source               destination         
LOG        tcp  --  anywhere             anywhere\
            limit: avg 5/min burst 5 LOG level debug prefix `Denied TCP: ' 
LOG        udp  --  anywhere             anywhere\
            limit: avg 5/min burst 5 LOG level debug prefix `Denied UDP: ' 
LOG        icmp --  anywhere             anywhere\
            limit: avg 5/min burst 5 LOG level debug prefix `Denied ICMP: ' 
DROP       all  --  anywhere             anywhere            

From now on, iptables will use syslog to log drop packets. Since iptables is actually updating the firewall rules inside the kernel, we first figure out how syslog is configured by looking for kern in /etc/syslog.conf.

$ grep kern /etc/syslog.conf
kern.*				-/var/log/kern.log

OK so all kernel messages are going into the /var/log/kernel.log file. We will later look there to correlate fail2ban banned IPs to iptables drop packets.


Spamassassin is a very well regarded spam filter for e-mails. We plan to route all comments to the web site through spamassassin as well. There does not seem any reason to think comment spam is any different from e-mail spam and it will reduce complexity and maintenance cost to rely on a single spam filter daemon.

$ aptitude install spamassassin
$ useradd -m -s /bin/false spamassassin
$ diff -U /etc/postfix/ postfix/
--- postfix/	2011-03-16 01:00:46.000000000 +0000
+++ /etc/postfix/	2011-03-12 22:36:43.000000000 +0000
@@ -10,3 +10,3 @@
 # ==========================================================================
-smtp      inet  n       -       -       -       -       smtpd 
-submission inet n       -       -       -       -       smtpd
+smtp      inet  n       -       -       -       -       smtpd 
+       -o content_filter=spamassassin
+submission inet n       -       -       -       -       smtpd
+       -o content_filter=spamassassin
#  -o smtpd_tls_security_level=encrypt
@@ -81,2 +81,4 @@
   ${nexthop} ${user}
+spamassassin unix  -       n       n       -       -       pipe
+   user=spamassassin argv=/usr/bin/spamc -e /usr/sbin/sendmail -oi\
  -f ${sender} ${recipient}

Postfix is a very versatile Mail Transfer Agent (MTA) that can be configured in many different ways to achieve similar results. Documentation related to spamassassin and filtering that is worth reading include Integrating SpamAssassinwith Postfix, Postfix Virtual Domain Hosting Howto and Postfix After-Queue Content Filter.

We will create a special user account for spamassassin and used the content_filter= method on both smtp (for out of network incoming e-mail) and submission (for local e-mails). As described earlier, the semilla web application submits comments as e-mails through a local account on the mail server. I would have preferred to put the spamassassin filter later, i.e. just before delivery to the local agent but I haven't managed to do that successfully yet. Right now, spamassassin will scan all outgoing e-mails as well (content_filter on submission agent).

At this point, we can see in /var/log/mail.log that messages are filtered through spamassassin. A little bit of testing can be done by sending something like the following e-mail:

  | sendmail info
$ tail -f /var/log/mail.log
Mar 16 01:19:51 hostname spamd[26879]: spamd: identified spam\
  (1000.0/5.0) for spamassassin:1003 in 0.3 seconds, 1491 bytes. 
Mar 16 01:19:51 hostname spamd[26879]: spamd: result: Y 1000 \
  - GTUBE,HTML_MESSAGE scantime=0.3,size=1491,user=spamassassin,uid=1003,\


We will now get fail2ban to dynamically insert rules for bots trying to break into ssh or obviously referencing pages that do not exist on the web site (such as PHP scripts).

$ aptitude install fail2ban
$ diff -U 3 /etc/fail2ban/jail.conf.prev /etc/fail2ban/jail.conf 
--- /etc/fail2ban/jail.conf.prev 2011-03-10 15:49:53.000000000 +0000
+++ /etc/fail2ban/jail.conf	2011-03-12 23:23:34.000000000 +0000
@@ -133,7 +133,7 @@
-enabled = false
+enabled = true
 port	= http,https
 filter	= apache-auth
 logpath = /var/log/apache*/*error.log
@@ -151,7 +151,7 @@
-enabled = false
+enabled = true
 port    = http,https
 filter  = apache-noscript
 logpath = /var/log/apache*/*error.log
@@ -159,7 +159,7 @@
-enabled = false
+enabled = true
 port    = http,https
 filter  = apache-overflows
 logpath = /var/log/apache*/*error.log

$ /etc/init.d/fail2ban restart

At this point, if you do see errors like "fail2ban.server : ERROR Unexpected communication error" in /var/log/fail2ban.log, you will need to apply the following patch to /usr/bin/fail2ban-server.

$ diff -U 1 /usr/bin/fail2ban-server.prev /usr/bin/fail2ban-server 
--- fail2ban-server	2011-03-16 00:48:29.000000000 +0000
+++ /usr/bin/fail2ban-server	2011-03-15 19:55:13.000000000 +0000
@@ -1,2 +1,2 @@
 # This file is part of Fail2Ban.

$ /etc/init.d/fail2ban restart

We now want to insert a new jail in fail2ban for host that are identified as sending spam but if we look into the /var/log/mail.log for spamd messages, we can see there are no IP associated to the originator of a mail identified as spam. A little patch in /usr/sbin/spamd that will print the first IP found in "Received" header fields of a mail will do. At the same time, I modified the semilla web application to send mail with a specially crafted "Received" header containing the REMOTE_ADDR environment variable.

$ diff -u /usr/sbin/spamd 
---	2011-04-21 23:35:10.000000000 +0000
+++ /usr/sbin/spamd	2011-04-22 00:11:17.000000000 +0000
@@ -1593,7 +1593,10 @@
   my $scantime = sprintf( "%.1f", time - $start_time );
-  info("spamd: $was_it_spam ($msg_score/$msg_threshold) for\
  $current_user:$> in"
+  my @from_addrs = $mail->get_pristine_header("Received");
+  join("\n",@from_addrs) =~ m/(\[\d+\.\d+\.\d+\.\d+\])/;
+  my $from_addr = $1;
+  info("spamd: $was_it_spam ($msg_score/$msg_threshold) from\
  $from_addr for $current_user:$> in"
        . " $scantime seconds, $actual_length bytes." );
   # add a summary "result:" line, based on mass-check format

The spamd related lines in /var/log/mail.log thus now look like:

Apr 22 21:20:23 hostname spamd[17844]: spamd: identified spam\
   (999.0/5.0) from [remoteaddr] for spamassassin:1003\
   in 0.2 seconds, 2152 bytes.

It is now trivial to add the following filter in /etc/fail2ban/filter.d/spamassassin.conf

failregex = spamd: identified spam .* from [[][]]

ignoreregex = 

and the following jail in /etc/fail2ban/jail.conf

enabled  = true
port     = http,https,smtp,ssmtp
filter   = spamassassin
logpath  = /var/log/mail.log

The script fail2ban-regex is very convenient to check your filter expression is doing what you are expecting. Later, while the system is up and running, you can use fail2ban-client to check the status of the jail.

$ fail2ban-regex "Apr 22 21:20:23 hostname spamd[17844]: \
  spamd: identified spam (999.0/5.0) from [remoteaddr] for\
  spamassassin:1003 in 0.2 seconds, 2152 bytes." \
  "spamd: identified spam .* from [[][]]"
$ sudo fail2ban-client status spamassassin


At this point, iptables, spamassassin and fail2ban are configured to ban spam bots from hitting our application stack. It is all great but without generating statistics and reports, there is no easy way to find out how effective the solution is. So I started to investigate log reporting tools. Lire seemed the most promising so I started there. Since lire is present in the Ubuntu repository, that is a breeze to install it.

sudo aptitude install lire

lr_log2report seems to be the major command to generate reports.

lr_log2report --help dlf-converters
iptables         Iptables firewall log
postfix          postfix log file
spamassassin     spamassassin log file

If you are running into the following error while running your first report, you will have to apply a little patch into /usr/share/perl5/Lire/

$ lr_log2report postfix /var/log/mail.log 
Parsing log file using postfix DLF Converter...
lr_log2report: ERROR store doesn't contain a 'lire_import_log'\
   stream at /usr/share/perl5/Lire/ line 170
$ diff -u /usr/share/perl5/Lire/
 sub dlf_streams {
     my $self = $_[0];
     my @streams = ();
-    my $sth = $self->{'_dbh'}->table_info( "", "", "dlf_%", "TABLE" );
-    $sth->execute();
-    while ( my $table_info = $sth->fetchrow_hashref() ) {
-        next unless $table_info->{'TABLE_NAME'} =~ /^dlf_(.*)/;
-        next if $table_info->{'TABLE_NAME'} =~ /_links$/;
-        push @streams, $1;
-    }
-    $sth->finish();
+ # JB : table_info seems to fail
+    my @table_list = $self->{'_dbh'}->tables;
+    foreach my $table ( @table_list) {
+        next unless $table =~ /dlf_(.*)"/;
+ 	 next if $table =~ /_links$/;
+ 	 push @streams, $1;
+     }
      return @streams;
$ lr_log2report iptables /var/log/kern.log
$ lr_log2report postfix /var/log/mail.log
$ lr_log2report spamassassin /var/log/mail.log
$ lr_log2report combined /var/log/apache2/domainname-access.log

We also want to add a converter for fail2ban logs so that we can correlate fail2ban actions to iptables dropped packets. Since fail2ban adds rules into the firewall through iptables, we will base its lire schema of the firewall schema (/usr/share/lire/schemas/firewall.xml). We then also add a perl script based of one of the previously existing converter (for example /usr/share/perl5/Lire/Firewall/ and a fail2ban_init to load our converter into the lire executable. Relevant interesting lines are

$ cat /usr/share/perl5/Lire/Firewall/
sub process_log_line {
    my ( $self, $process, $line ) = @_;
    my($date, $time, $name, $warning, $jail, $action, $source) 
        = split / /, $line, 7;
    if ( $@ ) {
        $process->error( $@, $line );
    } elsif ( $action ne 'Ban' ) {
        $process->ignore_log_line( $line, "not a Ban record" );
    } else {	
	use Time::Local;
        my $dlf_rec = {};
	if( "$date $time" 
          =~ /(\d\d\d\d)-(\d\d)-(\d\d) (\d\d):(\d\d):(\d\d),(\d\d\d)$/) {
	    $year = $1;
	    $month = $2;
	    $day = $3;
	    $hours = $4;
	    $min = $5;
	    $sec = $6;	    
	my $timestamp = timelocal($sec,$min,$hours,$day,$month,$year); 
        # replace 'timelocal' with 'timegm' if your input date is GMT/UTC
	$dlf_rec->{time} = $timestamp;
	$dlf_rec->{action} = "denied";
	$dlf_rec->{protocol} = "TCP";
	$dlf_rec->{rule} = $jail;		
	$dlf_rec->{from_ip} = $source;
	$dlf_rec->{count} = 1;
	$process->write_dlf( "firewall", $dlf_rec );
$ cat /etc/lire/plugins/fail2ban_init
use Lire::PluginManager;
use Fail2BanConverter;

            Fail2BanConverter->new() );

$ lr_log2report fail2ban /var/log/fail2ban.log

On Ubuntu, "aptitude install lire" will setup the appropriate cron jobs to send e-mail reports by running /usr/sbin/lr_vendor_cron.

$ find /etc -name '*lire*'
$ cat /etc/cron.weekly/lire
LIREUSER='lire' /usr/sbin/lr_vendor_cron weekly
$ less /usr/sbin/lr_vendor_cron
for d in /etc/sysconfig/lire.d /etc/default/lire.d
    test -d $d && CONFDIR=$d && break
for f in $CONFDIR/*.cfg

So we will add a few more .cfg files for spamassassin, iptables and fail2ban.

Be careful that testing /usr/sbin/lr_vendor_cron from the command line is a little tricky. You will most likely run into cryptic su errors because of the following line in /usr/sbin/lr_vendor_cron.

eval "$filter" < $logfile | \
    su - $LIREUSER -c \
    "lr_log2mail -s '$rotateperiod $service report from $logfile'\
  $extraopts $service root" 2>&1 | logger -p $PRIORITY -t lire


Voila, we are now running spamassassin on all comments posted through the web interface. Traffic from remote machines dynamically identified as spam originators is actively dropped before reaching our application stack. One last word, we add to setup a second aliases database such that the comment archiver writes files as the correct owner.

Virtual Machines Server

by Sebastien Mirolo on Sat, 8 Jan 2011

Ubuntu 10.10 comes with support for Virtual Machines. As I started to port fortylines testing infrastructure to a new more quiet machine, it is a good time to start playing around with virtual machine provisioning and decommissioning.

Ubuntu's default virtual machine infrastructure is built around KVM. The Ubuntu wiki has some information on booting a virtual image from an UEC Image. A blog post titled "Setting up virtualization on Ubuntu with KVM" contains also a lot of useful information. After browsing around, the tools to get familiar with include vmbuilder, kvm, qemu, cloud-init and eucalyptus.

$ which kvm
$ which qemu-system-x86_64

First, the easiest seems to try booting from a pre-built image. So I downloaded the current uec image and looked forward to boot the virtual machine from it. I tried the amd64 unsuccessfully and the i386 image goes through the same set of errors: "Could not initialize SDL" (add -curses option), then "General error mounting filesystems" (replave if=virtio by if=scsi,bus=0,unit=6). Finally I got a login prompt running the following commands.

# download UEC image
mkdir maverick-server-uec-i386
cd maverick-server-uec-i386
tar zxvf maverick-server-uec-i386.tar.gz
chmod 444 maverick-server-uec-i386*

# booting the virtual machine from the UEC image
kvm -drive file=maverick-server-uec-i386.img,if=scsi,bus=0,unit=6,boot=on \ -kernel "maverick-server-uec-i386-vmlinuz-virtual" \ -append "root=/dev/sda ec2init=0 ro init=/usr/lib/cloud-init/uncloud-init \ ds=nocloud ubuntu-pass=ubuntu" -net nic,model=virtio \ -net "user,hostfwd=tcp::5555-:22" -snapshot -curses

The idea behind the Ubuntu virtual machine support investigation is to run nightly mechanical builds on a virtual machine. The virtual machine is provisioned with a standard EUC image, the build is performed, installing prerequisites as necessary, the generated log is communicated back to the forum server and the virtual machine decommissioned.

The two main issues to be solved are starting the automatic build in the virtual machine, communicating the log back to forum server. A third issue not directly related to the cloud infrastructure is to run a sudo command on the virtual instance through a batch script.

The documentation and the kernel command line hint at a "xupdate=" option to the /usr/lib/cloud-init/uncloud-init init process. I thus mounted the disk image and starting digging through the uncloud-init script to find clues on how it could be useful for my purpose.

mkdir image
losetup /dev/loop2 maverick-server-uec-i386.img
mount /dev/loop2 image
less image/usr/lib/cloud-init/uncloud-init
 if [ -d "${mp}/updates" ]; then
      rsync -av "${mp}/updates/" "/" ||
               { log FAIL "failed rsync updates/ /"; return 1; }
if [ -d "${mp}/updates.tar" ]; then
        tar -C / -xvf "${mp}/updates.tar" ||
                { log FAIL "failed tar -C / -xvf ${mp}/updates.tar"; return 1; }
if [ -f "${script}" -a -x "${script}" ]; then
        MP_DIR=${mp} "${mp}/updates.script" ||
                { log FAIL "failed to run updates.script"; return 1; }

The uncloud-init script is designed to customize a virtual instance before the system fully boots and becomes operational, thus it is no surprise that xupdate mechanism cannot be used for starting the build process. It seems we will have to login into the instance and run the build process

For our purpose of a mechanical build system, it is possible to run virtual instances without bringing up an ssh server. Once the build is finished, we could mount the disk image through a loopback device on the host and retrieve the files from the mounted drive. That requires to add an entry like the following in /etc/fstab. Some blogs suggest to use autofs instead but I haven't been able to get it to work properly nor do I understand how it gets rid of the "mount as root" requirement.

/var/images/build-uec-i386.img /mnt/images/build auto ro,user,noauto,loop 0 0

Once the virtual machines are not provisioned locally but rather spawn into the cloud, that approach does not work anymore. So we might look into using the virtual instance ssh server to transfer logs around. All that is required is to copy the build master controller ssh public key into the virtual instance ubuntu account authorized_keys file, something that can be done by uncloud-init through the xupdate mechanism. So we create a custom update disk as follow.

mkdir -p overlay/updates
# ... set subdirectory structure to match the updated root ...
genisoimage -rock --output updates.iso overlay
qemu-img create -f qcow2 -b maverick-server-uec-i386.img disk.img
# This command works but still prompts for login.
kvm -drive file=disk.img,if=scsi,bus=0,unit=5,boot=on \
  -drive file=updates.iso,if=scsi,bus=1,unit=6 \
  -kernel "maverick-server-uec-i386-vmlinuz-virtual" \
  -append "root=/dev/sda ro init=/usr/lib/cloud-init/uncloud-init \
  ds=nocloud ubuntu-pass=ubuntu xupdate=sdb:mnt" \
  -net nic,model=virtio -net "user,hostfwd=tcp::5555-:22" -nographic

The dws script needs to communicate to the source control repository through the Internet. I found out that the edition of /etc/network/interfaces is unnecessary once you install libvirt. Despite some posts around the web, it seems the virtual bridge is only necessary to access the virtual machine from outside the host if either.

sudo aptitude install libvirt0

Ubuntu 10.10 had already done it as part of the installation for me as shown through the ifconfig command.

virbr0    Link encap:Ethernet  HWaddr 9e:66:65:fc:97:5b  
          inet addr:  Bcast:  Mask:

Two commands require sudo access, apt-get and shutdown. We use apt-get to install system prerequisites and shutdown to cleanly stop the virtual machine. We thus add the following two lines to the /etc/sudoers file. The batch script can then execute both commands without prompting for a password.

%admin ALL = NOPASSWD: /sbin/shutdown
%admin ALL = NOPASSWD: /usr/bin/apt-get

Once the virtual machine and the ssh server is started, it is then possible to execute the build script on the guest, copy the log file and shutdown the virtual machine in three successive ssh commands.

ssh -p 5555 ubuntu@localhost /home/ubuntu/bin/dkicks
scp -P 5555 -r ubuntu@localhost:/home/ubuntu/log .
ssh -p 5555 ubuntu@localhost /sbin/shutdown -P 0

The following python code can be used to wait until the ssh server responds.

def waitUntilSSHUp(hostname,login=None,port=22,timeout=120):
    '''wait until an ssh connection can be established to *hostname*
    or the attempt timed out after *timeout* seconds.'''
    import time

    up = False
    waited = 0
    sshConnect = hostname
    if login:
        sshConnect = login + '@' + hostname
    while not up and (waited <= timeout):
        waited = waited + 30
        cmd = subprocess.Popen(['ssh',
                                '-o', 'BatchMode yes',
                                '-p', str(port),
        if cmd.returncode == 0:
            up = True
            sys.stdout.write("waiting 30 more seconds (" \
                                 + str(waited) + " so far)...\n")
    if waited > timeout:
        raise Error("ssh connection attempt to " + hostname + " timed out.")

As it turns out, the build script is running out of space while installing all the prerequisites and compiling the repository. The original disk image (1.4Gb) seems to small for that purpose.

There seem to be three solutions to this problem.

  • Find a base image with a bigger disk
  • Create a new image with a bigger disk
  • Increase the size of the disk on the original disk image

As the next steps in our vm mechanical build project consist of running centOS disk images, it is a good time to start investigating running EC2 images locally. Apparently, there is a large library of those and we should find a public one that is sized correctly for our purpose. Looking around on the web, there is a lot of documentation creating and uploading EC2 images but I couldn't find relevant information on downloading a public image and running it locally in kvm. I was looking for something as simple as a url to a disk image but no luck so far.

To increase the size of disk image, the most common solution consists of concating two raw files together and update the partition table. The partition update part looks like a lot of complexity to code in a batch system. Currently we are using an update disk to customize the default disk image and now we also need to resize it which seem tricky enough. So I looked into building an image with vm-builder. Apparently that is how the UEC image I used earlier was put together.

$ aptitude search vm-builder
p   python-vm-builder                        - VM builder
p   python-vm-builder-ec2                    - EC2 Ubuntu VM builder
p   ubuntu-vm-builder                        - Ubuntu VM builder

I am not yet certain vmbuilder will also provide a mean to create CentOS images or if I will need a different tool for that purpose. None-the-less, let's start there for now.

sudo vmbuilder kvm ubuntu --rootsize=8192

The ubuntu-kvm directory was created with two files in it:, a shell script to with the kvm invoke command and a tmp78iihO.qcow2 file of 389Mb, the system disk image. Let's launch the image and see what's in it.

cd ubuntu-kvm && ./

Using the "ubuntu" login and "ubuntu" password, I am able to to get a shell prompt.

$ df -h
Filesystem  Size Used Avail Use% Mounted on
/dev/sda1   7.6G 482M  6.7G   7% /
$ ps aux | grep sshd
$ find /etc -name 'sshd*'

So we have a bootable image with 6.7G of space available. The sshd daemon is not running nor installed and most likely the scripts necessary to make a copy of that image unique in the cloud are not there either. Let's add our modifications to run the build script first, see how far it goes.

$ sudo aptitude install openssh-server
$ mkdir -p /home/ubuntu/bin
$ mkdir -p /home/ubuntu/.ssh
$ sudo vi /etc/sudoers
  # Defaults
+ # Preserve environment variables such that we do not get the error message: 
+ # "sorry, you are not allowed to set the following 
+ #  environment variables: DEBIAN_FRONT"
+ Defaults        !env_reset

  # Members of the admin group may gain root privileges
  %admin ALL=(ALL) ALL
+ %admin ALL = NOPASSWD: /sbin/shutdown
+ %admin ALL = NOPASSWD: /usr/bin/apt-get
$ sudo shutdown -P 0
> kvm -drive file=tmp78iihO.qcow2 \
  -net nic,model=virtio -net "user,hostfwd=tcp::5555-:22"

The previous command hangs the virtual machine in start-up while the following command does not permit to ssh into the virtual machine.

> kvm -drive file=tmp78iihO.qcow2 -net "user,hostfwd=tcp::5555-:22" &
> ssh -v -p 5555 ubuntu@localhost
ssh_exchange_identification: Connection closed by remote host

There are apparently more to vmbuilder that the documentation suggests to build an equivalent image to the one I originaly used...

Looking through the vmbuilder source repository I found a README.files in automated-ec2-builds that mentioned a uec-resize-image script.

sudo aptitude install bzr
bzr branch lp:~ubuntu-on-ec2/vmbuilder/automated-ec2-builds

I might actually be able to resize my original image with a single command after all.

aptitude search *uec*
bzr branch lp:~ubuntu-on-ec2/ubuntu-on-ec2/uec-tools
ls uec-tools/resize-uec-image
sudo install -m 755 uec-tools/resize-uec-image /usr/local/bin

Let's use the resize script and check the free space on our new image.

> resize-uec-image maverick-server-uec-i386.img 5G
> ls -la maverick-server-uec-i386.img
-rw-r--r--  5368709120 2011-01-03 07:28 maverick-server-uec-i386.img
>kvm -drive file=maverick-server-uec-i386.img,if=scsi,bus=0,unit=6,boot=on \
    -kernel "maverick-server-uec-i386-vmlinuz-virtual" \
    -append "root=/dev/sda ec2init=0 ro \
    init=/usr/lib/cloud-init/uncloud-init ds=nocloud \
    ubuntu-pass=ubuntu" -net nic,model=virtio \
    -net "user,hostfwd=tcp::5555-:22" -snapshot -curses
$ df -h
Filesystem  Size Used Avail Use% Mounted on
/dev/sda1   5.0G 516M  4.2G  11% /

Finally, I managed to get a script that starts a virtual machine, runs the mechanical build end-to-end and copies the build logs back out. It is a good start but there remains a few issues with the current approach. The cloud-init script re-enables ssh password authentication after it updates sshd_config with our version.

# /usr/lib/cloud-init/uncloud-init
sed -i "s,${pa} no,${pa} yes," /etc/ssh/sshd_config 2>/dev/null &&
        log "enabled passwd auth in ssh" ||
        log "failed to enable passwd ssh"

The IP address of our virtual machine is always the same but uncloud-init will generate different ssh server keys every time we run our build on fresh virtual-machine script. That creates identification issues for the ssh client that the "StrictHostKeyChecking no" parameter does not always solve.

I looked quickly through virsh and eucalyptus. It seems each running instance needs to be registered with a global store, i.e. requires sudo access on the host. Those tools do not seem suited for the kind of thirty minute life span virtual machine (start, build, throw away) I need.

It took way longer than I anticipated to figure the pieces out and it surely a long and winding road ahead before we have a simple to use cloud-based build infrastructure.

Booting Ubuntu from USB stick

by Sebastien Mirolo on Tue, 16 Nov 2010

Ubuntu 10.10 comes with support for Virtual Machines. As I started to port fortylines testing infrastructure to a new more quiet machine, it is a good time to start playing around with virtual machine provisionning and decomisionning.

After downloading ubuntu-10.10-server-amd64.iso from the Ubuntu official website, I stumbled upon the issue of booting it from a usb stick. I followed the instructions to create a bootable usb drive from the iso on an OSX host. That turned out to be a disaster. A long time in debugging and searching through the web revealed that if the dd command fails to copy a read-only ISO on top a read/write drive, you won't be able to change the content of that drive afterwards. After I bought a new usb stick and create the usb bootable drive from within an Ubuntu virtual machine, I had no problem to boot and install the new machine that will serve as the virtual machine provider server. There is a caution message about following the OSX instructions but in my opinion it should be a big "don't do it".

Running fop on an headless server

by Sebastien Mirolo on Sun, 31 Oct 2010

fop is a useful tool in the process of formatting documentation as pdf when you start with docbook marked-up documents. Unfortunately on some machine you end-up with the following result when running the tool.

fop -fo -pdf drop.pdf
(.:13431): Gtk-WARNING **: cannot open display: 

As it turns out, fop relies on different java libraries. One of them, the SVG renderer batik uses the Java AWT framework and ultimately requires an X11 server being present.

There are thus a few alternatives to get rid of the problem. They either boil down to installing an X11 server or getting rid of the X11 dependency somehow. A popular (as far as google searches tell) solution out of the second category is to run the java virtual machine in headless mode using the following command line argument

java -Djava.awt.headless=true 

On Ubuntu 8.10 server, this fix did not work at first. The reason there lies in the Java Runtime Environment (JRE) installed by default. Ubuntu installs gcj, the GNU Java Implementation, and even though in theory it should support the java.awt.headless property, there are enough reports of major flaws in the GNU implementation that it does not seem surprising to fail here.

Fortunately there is a simple way to force the system to use the Sun (Now Oracle) Java Runtime.

sudo apt-get install sun-java6-jre
sudo update-alternatives --config java
fop --execdebug -fo -pdf drop.pdf

Headless mode works and the pdfs can now be produced on the build machine.

Lost SSH Connection and GNU screen

by Sebastien Mirolo on Thu, 9 Sep 2010

I had to do work remotely on a machine that required long session recently and the ssh connection kept dropping at random. It is very annoying and it seems almost impossible to track it down. The best explanation so far looks to lie with the way routers, firewalls and other bridges between me and the remote machine maintain some sanity into their NAT tables. In any case, I started to use GNU screen such that least running jobs do not just get killed off when the connection dropped.

There is a lot of good stuff written about GNU screen but it hardly mentioned how much it can get in the way of your command line habits. First, the backspace does not work as expected unless you configure your session accordingly. I thus added the following line to my ~/.bash_profile:

alias screen='TERM=screen screen'

I also found out this little piece of code very useful to know when I am in a screen session and when I am in raw ssh session. So I've added into ~/.screenrc:

shell -${SHELL}
caption always "%n(%t) : %C"

Finally, annoying as hell, Ctrl-A is kind of the meta-used-for-everything key combination in screen. This means to go to the beginning of the line, you have to type "Ctrl-A A" to pass Ctrl-A to bash.

Ubuntu Server and VMware Fusion

by Sebastien Mirolo on Sat, 17 Oct 2009

Anticipated the upcoming next release of Ubuntu, I decided to setup an Ubuntu Server Edition sandbox under VMware to test the configuration scripts. I do not know about you, but I feel a little anxious about upgrading the customer-facing Internet server, fixing bugs in real-time.

VMware Fusion's update tool suggested me to download and install version 2.06. I did that then created an Ubuntu sandbox virtual machine and installed the 64-bit Server Edition ISO image (ubuntu-9.04-server-amd64.iso).

That is when I tried to mount an OSX user account folder on the sandbox virtual machine that things starting to break down. The "Install VMware Tools" menu did not seem to do anything. I could not find the VMware Tools .tar.gz package anywhere on the filesystem either. Hopefully, I already had a Ubuntu Desktop Edition virtual machine and the VMware Tools DVD-ROM mounted correctly there as /media/cdrom1/VMwareTools-7.9.7-196839.tar.gz. Since I already had setup that Ubuntu Desktop Edition virtual machine with an ssh server, I remote copied the package onto the Ubuntu Server Edition sandbox. After installing the VMware tools and rebooting the virtual machine, I could then access the OSX files through /mnt/hgfs/username as usual.

# On the Ubuntu Desktop Edition virtual machine:
ifconfig | grep 'inet addr'
# On the Ubuntu Server Edition virtual machine:
scp ipaddress:/media/cdrom1/VMwareTools-7.9.7-196839.tar.gz .
tar zxvf VMwareTools-7.9.7-196839.tar.gz
cd vmware-tools-distrib
sudo ./
sudo shutdown -r 0

I still cannot copy or paste from and to the Ubuntu Server Edition virtual machine and still require to use ctrl + command key combination; maybe because there is no X11 installed on that virtual machine. In any case, I have a way to move files from the OSX filesystem to the Ubuntu Server Edition sandbox so debugging of the upgrade path for the live server is under way.

Share with your network