Building-up the infrastructure

Network overview

Network Diagram

The network infrastructure is divided into three categories: third-parties, production and development. Third-parties include all machines that are not directly owned and operated by our service. Production machines are delivering the core of our service. Development machines host the communication forum, work in progress and other activities necessary to manage a software project.

Third-parties

customers
  • At first these are machines owned and operated by people paying for our service. As the ecosystem grows, these also includes third-party services that use APIs (Application Programming Interfaces) made publicly available.

  • provider APIs
  • These are machines owned and operated by third-parties and provides services our service rely on. The most basic service in that category is naturally the payment processing system.

  • penetration testing
  • These machines constantly scans our service infrastructure for vulnarabilities. Technically if no adequate third-party service can be found, these machines might be owned by our service but they should always be operated outside the service infrastructure. These machines simulate rogue customers and other remote network threats.

  • Production

    auth
  • This machine hosts the "homepage", creates sessions and does load-balancing between production machines.

    They provide the authentication server

  • prod
  • These machines host the actual service, web application and databases.

  • mail
  • This machine hosts the mail server

  • Development

    Access to the development machines is strictly restricted to contributors.

    forum
    • This machine hosts the development forum.
    • repository
    • continuous build
    • hids web interface
    • wiki
  • backup
    • This machine hosts the backup service.
    • physical
  • build
  • These machine host build, running tests.

  • \todo Services

    Fortylines created dservices as part of the tero product. It is a script to turn a bare Ubuntu iso install into a communication infrastructure suitable for software development. The following sections give explanations on the rationale of different internet services and pointers to their configuration.

    At least, the following technical components should be understood in order to establish a professional presence on the internet.

    Services
    name application of choice standard ports details
    Domain Name Sever (DNS) bind 53 RFC1034, RFC1035
    Internet Message Access Protocol (IMAP) dovecot 143, 993 RFC3501
    Message Transfer Agent (MTA) postfix 25, 587 RFC821, RFC2821
    Secure Shell (SSH) openssh 22 RFC4252-RFC4255
    Hypertext Transfer Protocol (HTTP) apache 80, 443 RFC2616, RFC2818

    Good practice requires to review pre-installed configuration files and read the associated documentation carefully. That said it is not straightforward to debug an issue with your services configuration. There are lots of reference manuals on the web about these different services but you need to know them back and forth to find the information you care about. There are even more working examples of configurations, most of them blindly applied to your particular setup and you just just end-up with a cryptic error message you very soon realize you are not the only one wondering about the meaning.

    In the following sections, we will present a short description of the purpose of specific services, their rough principle of operation and where to look when an issue arises. Examples refer to fortylines' setup of its Ubuntu server.

    First tip, most logs are written directly in /var/log or a subdirectory of /var/log. Whenever you try to connect to a service and the request reaches the server, most likely one log file somewhere will have been updated; ls -lrt /var/log can help determine the last log modified.

    Base

    Secure Shell

    Secure Shell Login

    • Installation: apt-get install openssh-server

    • Configuration: /etc/ssh/sshd_config (see also /etc/pam.d/sshd)

    • Restarting: /etc/init.d/ssh restart

    • Logs: /var/log/auth.log

    OpenSSH, the secure shell service is used to create encrypted tunnels between a client machine and the server. All traffic going through the tunnel is encrypted and thus resistent to eavesdropping.

    There are two important aspects of ssh tunnels to understand. First, you must insure you are talking to the actual server when authenticating. If you don't, a proxy can pretend to be the server and happily acts as a man in the middle reading all traffic in clear. In details, it works as such. The client thinks it is talking to the server and send its authentication information which the proxy happily uses right away to connect to the server. The client then starts to send messages encrypted with the client/proxy key to the proxy. The proxy decrypts the message and encrypts it with the proxy/server key before forwarding it to the server. When the proxy gets the response, it decrypts it and encrypts it with the client/proxy key before forwarding it back to the client.

    The second important aspect of ssh tunnels is that all applications on the client machine are configured to use a local client port while for the services on the server the connection is coming from a local server port. This is of particular importance for domain name resolution because the DNS queried by the client and server machines will most likely be different.

    Once sshd is installed on the server, there are two things to do. First, from the local console, use the ssh-keygen command to retrieve the server ssh key fingerprint and publish it to the people that will be allowed to remotely login into the server. Contributors will use the fingerprint to check they are connecting directly to the actual server. For example:

    
    <span class="emphasis">ssh-keygen -lf /etc/ssh/ssh_host_rsa_key.pub</span>
    2048 c2:0d:b3:19:bf:cf:c8:09:f7:bd:76:dc:67:cd:59:7b \
        /etc/ssh/ssh_host_rsa_key.pub (RSA)
        

    Second, review the configuration file (/etc/ssh/sshd_config) and edit the settings to match your requirements. The sshd daemon offers to authenticate a user using one of many schemes. It is important to note that, so far, it does not offer to authenticate through a combinaison of schemes. Each scheme has advantages and drawbacks that you need to take into concideration while designing the access policy. For example

    Public/Private Key Password
    Major threat Compromised client Brute force cracking
    Server administrator stance Unaware of breach attempts before reported or actively exploited. Pro-actively evaluate weaknesses and detect breach attempts.

    The risk of a compromised client is very real when all contributors have just enough skills to administrate their own machines, a laptop they carry around everywhere. Ideally, both are valuable and should be used together by the sshd daemon.

    The steps to configure an ssh server and client to use RSA public/private key for authentication.

    Client Server (i.e. hostname)
    
    man ssh-keygen
    man ssh_config
    	
    
    man sshd_config
              
    
    ssh-keygen -q \
      -f ~/.ssh/<span class="emphasis">hostname</span>_rsa -t rsa
    	  
    
    sudo apt-get upgrade sshd
    diff -u sshd_config.prev \
        /etc/sshd_config
    + Protocol 2
    + PasswordAuthentication no
    + PubkeyAuthentication yes
    + AuthorizedKeysFile    \
        %h/.ssh/authorized_keys
    sudo /etc/init.d/ssh restart
    Append the content of the client's file hostname_rsa.pub to the server's file /home/username/.ssh/authorized_keys
    
    ls -la ~/.ssh
    drwx------   .
    -rw-r--r--   config
    -rw-------   <span class="emphasis">hostname</span>_rsa
    -rw-r--r--   <span class="emphasis">hostname</span>_rsa.pub
    -rw-r--r--   known_hosts
    	  
    
    ls -la ~/.ssh
    drwx------ .
    -rw------- authorized_keys
    
    ssh -v <span class="emphasis">hostname</span>
    ...
    debug1: Next authentication \
        method: publickey
    debug1: Offering public key: \
        /Users/<span class="emphasis">username</span>/.ssh/<span class="emphasis">hostname</span>_rsa
    debug1: Server accepts key: \
        pkalg ssh-rsa blen 277
    debug1: Authentication succeeded \
        (publickey).
    ...
    	
    
    vi /etc/ssh/sshd_config
    + LogLevel VERBOSE
    tail -f /var/log/auth.log
    ...
    ... Found matching RSA key ...
    ... Accepted publickey for \
        <span class="emphasis">username</span> from <span class="emphasis">client</span> \
        port 50818 sstitle
    ... pam_unix(sshd:session): \
        session opened for user <span class="emphasis">username</span> \ 
        by (uid=0)
    ... User child is on pid ...
          

    Tip: At some point, you might be temped to update sshd or its configuration through a secure shell connection instead of at the local console. In that case, it is recommended to add a cron job that will restore previously known working configuration after a few minutes before logging out. Just in case.

    Secure Shell Tunnels

    Port 22 is open in the firewall for secure shell login. We will also use it to tunnel all traffic to services on the Virtual Private Network. This also has the advantage of encrypting all traffic regardless of services provided. The server needs to permit tunnels or you will see "permission denied" messages.

    Client Server (i.e. hostname)
    
    diff -u sshd_config.org \
        /etc/ssh/sshd_config
    +PermitTunnel yes
    
    
    ssh -fN \
     -L <span class="emphasis">localport</span>:<span class="emphasis">hostname</span>:<span class="emphasis">hostport</span> \
     <span class="emphasis">hostname</span>
    	
    
    tail -f /var/log/auth.log
    	  

    At that point, you can open any secure shell tunnel you want. Two tunnels are open to access e-mails, one for IMAP and one for SMTP. One tunnel is open to access the private web site. The following code is thus added in the client ~/.bash_profile.

    
    # http://wiki.metawerx.net/wiki/SSHTunnelTroubleshooting
    function <span class="emphasis">hostname</span>_tunnels {
        ssh -fN -i ~/.ssh/<span class="emphasis">hostname</span>_rsa \
          -L 9930:localhost:993 -L 5870:localhost:587 \
          -L 8000:localhost:80 <span class="emphasis">hostname</span>
    }
        

    The first number of the "-L" ssh command line option is the port on the client machine, the hostname between ":" will be resolved on the server machine and the second number is the port to access on the server machine.

    Invoking hostname_tunnels from a terminal command line will create the tunnels. Those tunnels will be open until the ssh process is killed. It is possible to also create auto closing ssh tunnels with some more shell scripting if required.

    The Mail client is configured with the following parameters.

    IMAP Incoming Mail Server 127.0.0.1
    User Name username on hostname
    password password for username on hostname
    SMTP Server Name 127.0.0.1
    Custom port 5870
    Use Secure Sockets yes
    Authentication password
    User Name username on hostname
    password password for username on hostname

    The private website can be accessed by typing "http://localhost:8000" into your favorite web browser's url bar.

    Tip: test the tunnel with the webserver first. It will be easier to debug tunnel related issues rather than mail setup.

    \todo Firewall

    Communication services are either used to

    • 1. Reach out to the community
    • 2. Coordinate trusted developpers

    Services in the first category are accessible by anyone and everyone. Technically they will require to open and forward a dedicated port in the firewall. Services in the second category require authentication.

    The public website and the mail transfer agent need to be accessible by anyone so we will open port 25 and 80 in the firewall facing the outside world. Only authorized contributors are allowed to access other running services and they do such through secure shell tunnel. Hence we also open port 22 in the firewall facing the outside world.

    \todo Backup

    Once ssh is up and running, almost immediately you need to setup the backup infrastructure.

    \todo Audit

    Host Intrusion Detection System (HIDS) and Network Intrusion Detection System (NIDS).

    Samhain is an Host Intrusion Detection System.

    
    	  sudo apt-get install samhain	  
    	

    Yule not supported on ubuntu? https://bugs.launchpad.net/ubuntu/+source/samhain/+bug/151643

    \todo Git Repository

    We want to support a development model with M contributors (read-only) and N committers (read/write). Furthermore we want to restrict shell access to repository machine to a few committers if any at all. The git daemon does not support authentication by itself so we will setup gitosis to add the functionality.

    
    # Installing gitosis on the server machine
    $ apt-get install gitosis
    $ diff -u  /etc/passwd.prev /etc/passwd
    -gitosis:x:107:116:git repository hosting,,,:/srv/gitosis:/bin/sh
    +git:x:107:116:git repository hosting,,,:/var/reps:/bin/sh
    $ diff -u  /etc/group.prev /etc/group
    -git:x:116:
    +gitosis:x:116:
    $ sudo mkdir -p /var/reps
    $ sudo chown git:git /var/reps
    
    # Create committers and contributors group
    $ diff -u prev /var/reps/repositories/gitosis-admin.git/gitosis.conf
    
    # Make new key for contributor
    $ ssh-keygen -q -f ~/.ssh/<span class="emphasis">hostname</span>_rsa -t rsa
    
    # Create a new repository
    $ diff -u prev /var/reps/repositories/gitosis-admin.git/gitosis.conf
    [repo dummy]
    description = Dummy repository
    owner = No one
    gitweb = no
    daemon = no
    	  

    \todo Web service

    • Installation: apt-get install apache2

    • Configuration: /etc/apache2/apache2.conf, /etc/apache2/httpd.conf

    • Restarting: /etc/init.d/apache2 restart

    • Logs: /var/log/apache2/access.log, /var/log/apache2/error.log

    Looking at the configuration file, Ubuntu apache2 setup is configured to use a directory /etc/apache2/sites-enabled with links to files in /etc/apache2/sites-available. You most likely want to /etc/apache2/sites-enabled/000-default to point to your own configuration snipset or edit /etc/apache2/sites-available/default directly. You will also want to add a ServerName directive to /etc/apache2/apache2.conf in order to get rid of the warning "Could not reliably determine the server's fully qualified domain name..." message.

    Another directive you might want to take a close look at is DirectoryIndex. All urls pointing to the directory will try to load a specified default file.

    We configured fortylines apache server with two virtual hosts to service the public and private websites. Because the private website can only be accessed through the ssh tunnel, as priorly seen, all connections will originate from localhost (aka 127.0.0.1). We thus restrict the access to the private website by IP address using the directive:

    
          <VirtualHost 127.0.0.1:80>
        

    \todo E-mail Service

    • Installation: apt-get install postfix

    • Configuration: /etc/postfix/master.cf /etc/postfix/main.cf

    • Restarting: /etc/init.d/postfix restart

    • Logs: /var/log/syslog, /var/log/mail.log

    Postfix acts as a central dispatcher that hand over mail to different agents for processing based on addresses patterns. The list of available agents and how to invoke them is specified in /etc/postfix/master.cf while the list of (pattern,agent) tuples is specified in /etc/postfix/transport.

    A good point to start reading are the Ubuntu Postix Tutorials.

    It is not straightforward to figure what is going on when e-mails do not work as expected. For debugging postfix, you might want to append -v to the smtp line in /etc/postfix/master.cf as a first step and then enable logging in main.cf (i.e. smtp_tls_loglevel and smtpd_tls_loglevel).

    Identification of the mail server:

    
    +myorigin = /etc/mailname
    +mydestination = <span class="emphasis">domainname</span>, <span class="emphasis">hostname</span>, localhost
    +home_mailbox= Maildir/
        

    At some point, you will want to enable Transport Layer Security (TLS) to avoid usernames, passwords and messages to travel in clear text. There are two set of settings in main.cf, starting by smtp_ and smtpd_ respectively and their usage is rather confusing at first. The postfix daemon uses the first set, smtp_* to establish outbound connection to other MTAs (client-side) and uses the second set, smtpd_*, to authorize inbound connections (server-side).

    Outgoing Mail

    Mail servers send each other messages on port 25. Most Internet Service Providers (ISP) block outbound port 25 as a way to prevent spam relays. The prefered solution for a profitable business is to negotiate with your ISP to let outbound traffic on port 25 flow out of your mail server. This usually involves to buy a business package and a higher monthly fee. If you are only experimenting and do not mind receipients to get e-mails showing "From ... on the behalf of ... " in the sender field, ISPs currently leave outbound traffic on port 587 open for submission of messages by users that can authenticate with an outgoing mail server. In that setting, you will have to configure postfix to use a relay mail transfer agent to send outgoing e-mails. Any mail server allowed to talk on port 25 on which you have an account can be used as a relay. If you have a gmail account for example, that will do. Since this kind of setup is very common in conjunction with dynamic DNS, it is explained in the later section on dynamic DNS setup.

    When e-mails do not seem to be arriving to their destinations, the first things to do are to read the mail log (/var/log/mail.log) and what stands in the queue (/usr/sbin/postqueue -p).

    Incoming Mail

    • Installation: apt-get install dovecot-imapd

    • Configuration: /etc/dovecot/dovecot.conf

    • Restarting: /etc/init.d/dovecot restart

    • Logs: /var/log/syslog, /var/log/mail.log

    Mbox, the default format for e-mail storage by postfix is simple and straightforward but incompatible with the dovecot POP3/IMAP service. The service can only retrieved e-mails stored locally as Maildir mailboxes. One of the first thing is to set Maildir as the type of mailboxes used by postfix.

    Locate the line starting with myorigin and update the referenced file. On Ubuntu, it is currently /etc/mailname. Edit this file and set the first line to domainname.

    Restart the server and try sending mail to an account on the server. The logs and /home/username/Maildir/new/ should show some acitivity.

    To access your mail through your normal mail client, you will want to install a POP or IMAP server such as dovecot.

    Dovecot on Ubuntu is configured to use PAM for user authentication by default. We configure postfix smtpd to authenticate users through dovecot, hence effectively using the PAM authentication mechanism by transition. This requires to export dovecot's auth-client to a socket file (see /etc/dovecot/dovecot.conf) and set both postfix smtpd_sasl_type and smtpd_sasl_path configuration variables (see /etc/etc/postfix/main.cf). Since postfix runs in a chroot jail, the auth-client socket of course has to be accessible within the jail.

    \todo: SPF FAQ (Mail configuration)

    \todo Spamassassin

    Mailing Aliases

    It is often useful to set some e-mail addresses as alias of another address. For example, the info@hostname should remain constant and standard regardless of the actual contributor servicing it. It is thus a perfect candidate for aliasing.

    Locate lines begining with alias in /etc/postfix/main.cf, edit the referenced file (/etc/aliases on Ubuntu) to add aliases and run the newaliases command to rebuild the fast-lookup database.

    Mailman Mailing lists

    • Installation: apt-get install mailman

    • Configuration: /etc/mailman/mm_cfg.py

    • Restarting: /etc/init.d/mailman restart && /etc/init.d/apache2 restart && /etc/init.d/postfix restart

    • Logs: /var/log/apache2/access.log, /var/log/apache2/error.log

    • Archives: /var/lib/mailman/archives

    Mailing lists are used to communicate information to a known and managed set of receipients in a push model. They usually gather questions, answers and the train of thoughts through list threads.

    We will create one mailing list called "dev" for contributors to exchange information, questions and answers in a public forum.

    Mailing lists are a typical case of mailing aliases and could be setup as such. Most times though, it becomes cumbersome to administer mailing list as plain aliases. Mailmain simplifies the management of mailing lists and adds many more features such as archiving for example.

    The steps to create a new mailing list consists of running newlist nameOfMailingList and add appropriate aliases for postfix to find. After this is done, it is usually easier to use Mailman's web interface (i.e. CGI scripts) to configure and manage mailing lists. It is enabled by adding a couple aliases into the apache configuration as such:

    
    
    ...
    ScriptAlias /cgi-bin/mailman /usr/lib/cgi-bin/mailman
    Alias /pipermail /var/lib/mailman/archives/public
    ...
    
    	

    Fortylines Mailman's web interface is not available publicly and runs on the private web site. It means it is only available through http://localhost:8000/cgi-bin/mailman/ after the ssh tunnels have been started. As a result, if the web_page_url is not properly set in the mailing list configuration, some links will appear broken because pointing to the actual public domain instead of localhost:8000. This can be fixed with the following commands

    
    echo "web_page_url = 'http://localhost:8000/cgi-bin/mailman/'" > <span class="emphasis">listName</span>.conf
    sudo /usr/sbin/config_list -i <span class="emphasis">listName</span>.conf <span class="emphasis">listName</span>
    

    Another solution is to create a new list with a different url for the web interface and the email domain.

    
    /usr/sbin/newlist -q -u localhost:8000 -e domainName listName
    

    A publicly open mailing list will usually define the following aliases in /etc/aliases

    
    <span class="emphasis">listName</span>:              "|/var/lib/mailman/mail/mailman post <span class="emphasis">listName</span>"
    <span class="emphasis">listName</span>-admin:        "|/var/lib/mailman/mail/mailman admin <span class="emphasis">listName</span>"
    <span class="emphasis">listName</span>-bounces:      "|/var/lib/mailman/mail/mailman bounces <span class="emphasis">listName</span>"
    <span class="emphasis">listName</span>-confirm:      "|/var/lib/mailman/mail/mailman confirm <span class="emphasis">listName</span>"
    <span class="emphasis">listName</span>-join:         "|/var/lib/mailman/mail/mailman join <span class="emphasis">listName</span>"
    <span class="emphasis">listName</span>-leave:        "|/var/lib/mailman/mail/mailman leave <span class="emphasis">listName</span>"
    <span class="emphasis">listName</span>-owner:        "|/var/lib/mailman/mail/mailman owner <span class="emphasis">listName</span>"
    <span class="emphasis">listName</span>-request:      "|/var/lib/mailman/mail/mailman request <span class="emphasis">listName</span>"
    <span class="emphasis">listName</span>-subscribe:    "|/var/lib/mailman/mail/mailman subscribe <span class="emphasis">listName</span>"
    <span class="emphasis">listName</span>-unsubscribe:  "|/var/lib/mailman/mail/mailman unsubscribe <span class="emphasis">listName</span>"
    

    Of course, as usual, the aliases look-up database needs to be updated.

    
    /usr/bin/newaliases
    

    Later versions of mailman require a "site-wide" mailing list called mailman. If that "mailman" mailing list is not present, the mailman service will refuse to start. Since by default, all lists are public, you will be required to make that list private after creation (see /usr/bin/config_list).

    To check mailman was installed and configured correctly, you can send a test e-mail the list. This can be done from the command line with the mailx utility (you will have to install either Ubuntu's package mailx or mailutils).

    
    echo "Hello" | mailx -s "test message" listName
    

    It is sometimes useful to add members to a list through a shell command line and this can be done with the following command line:

    
    sudo sh -c 'echo <span class="emphasis">"newMemberEmailAddress"</span> | /usr/sbin/add_members -r - <span class="emphasis">"mailingList"</span>'
    	

    And more often than none, it is useful to reset the mailman main password with the command line:

    
    /usr/sbin/mmsitepass
    

    \todo Daemon tools

    Dynamic DNS (ddclient)

    • Installation: apt-get install ddclient

    • Configuration: /etc/ddclient.conf

    • Restarting: /etc/init.d/ddclient restart

    The modem that connects the server to the internet through your ISP surely does not use a static IP address but rather a dynamic address assigned through DHCP, the same way, most likely, your local network is configured.

    Dynamic DNS is very simple. Everytime your modem renew its DHCP lease and gets a new IP address, this address needs to be forwarded to the DNS server responsible to locate your modem. Either the modem can do that for you or a software daemon running on your server wakes up at regular intervals and generate a specifically crafted update query to the DNS server. On Ubuntu, you can setup dynamic DNS with ddclient.

    There are a lot of companies offering dynamic DNS for free. As an example, you can easily register with dynDNS and get the url hostname.is-a-geek.com for free in a couple clicks. A direct professional domainname.com url will not cost much through "Custom DNS service" and is definitely worth it once you got everything up-and-running. There is a quota on the monthly DNS query volume and if you either reach it, most likely your business is successful enough to upgrade to a more professional infrastructure. Here is an example to setup ddclient together with dynDNS.

    
    sudo apt-get install ddclient
    ls -la /etc/ddclient.conf
    -rw------- 1 root root <span class="emphasis">...</span> /etc/ddclient.conf
    sudo cat /etc/ddclient.conf
    # Example for /etc/ddclient.conf
    
    pid=/var/run/ddclient.pid
    protocol=dyndns2
    use=web, if=eth1
    server=members.dyndns.org
    login=<span class="emphasis">dynDNS login</span>
    password=<span class="emphasis">dynDNS password</span>
    <span class="emphasis">hostname</span>.is-a-geek.com, <span class="emphasis">domainname</span>.com
        

    Mail servers send each other messages on port 25. Most Internet Service Providers (ISP) block outbound port 25 as a way to prevent spam relays. Port 587 remains open for submission of messages by users that can authenticate with an outgoing mail server.

    Thus in a dynamic DNS setup it is almost guarenteed you will be required to use a relay mail transfer agent to send outgoing e-mails. To do that, you can use any mail server allowed to talk on port 25, and on which you have an account. For example, you can use a gmail account. The following tutorial is very educative in that regard: Gmail on Home Linux Box using Postfix and Fetchmail.

    Gmail will rewrite your outgoing message to read "from username@gmail.com" as a way to also prevent spam and false identities. Fortunately, you can use any "from" address while relaying mail through gmail as long as you can receive e-mail at that address. All you need is to login in your gmail account and go through "Settings > Accounts and Imports > Send Mail from another address". From a login on your server, test outgoing messages by using the following command:

    
    echo "Email relayed by gmail" | mail -s 'test of relay' <span class="emphasis">username</span>@gmail.com
          

    The side effect of using a mail relay is that on Outlook for example, messages will show "From ... on the behalf of ... " in the sender field.

    For mail delivery, you might also want to setup MX records with your dynamic DNS provider. If your mail server never goes down, it will work without them.