The Django and Ubuntu 8.04 Chronicle

2009-02-24T02:40:00.001-08:00

This is my gift to you. I've spent the better part of a Jōmon Sugi tree's lifespan configuring servers and glancing with a raised eyebrow at the various nuggets of wisdom that people have decided to part with through the intertubes. Some of these do actually contain useful information, but most will actually end up doing you more harm than good. This is where I come in, the savior as it were, saving you from yourself and others.

This guide revolves around deploying Django on Ubuntu 8.04 LTS (Hardy Heron), using PostgreSQL as the database, Nginx as a proxy server infront of Apache2 coupled with mod_wsgi, and Memcached as the cache back-end.

General ↑
We'll be using Ubuntu, and specifically the 8.04 LTS (Hardy Heron) version, due to its ease-of-use and stability coupled with the more than adequate security measures for our needs. I'll update this document when the new 9.04 LTS (Jaunty Jackalope) version is released in April 2009. You can read more about Ubuntu on their site.

Users ↑
After a clean install of Ubuntu 8.04, login via SSH:

$ ssh root@255.255.255.255

Once logged in, the first thing you should do is to make an admin user that will have sudo rights, since the actual root user will ideally only be used during this initial stage. We will also make a dedicated django user.

Make the users and create home folders.

# useradd -m paul
# useradd -m django

Create their passwords.

# passwd paul
# passwd django

Change the default shell from Bourne to Bash.

# chsh -s /bin/bash paul
# chsh -s /bin/bash django

Add your admin user, but not the Django user, to the sudo list. Include this at the end of the file.

# visudo

line:21 paul    ALL=(ALL) ALL

SSH Key ↑
One of the most convenient and secure ways of accessing your server is through a public/private key scheme. This entails having a public key stored on the server and a private key on your local workstation, a benefit of which is that you don't have to supply your password when accessing your server through SSH.

You might find that some of these steps are already completed on your local workstation, if that is the case, jump ahead to the "SSH copy" section.

Open up a new terminal window in addition to the one already connected to the server; it is always wise to keep an open connection to your server throughout when configuring SSH. Create the .ssh directory on your local workstation and limit who has access to it.

localhost$ mkdir ~/.ssh
localhost$ chmod go-rwx ~/.ssh

Create the ssh keys on your local workstation.

localhost$ ssh-keygen -t rsa

That should give you two files in your ~/.ssh folder, id_rsa and id_rsa.pub. The .pub file is your public key and the one we will copy to your server, the other file is your private key and should never be shown, copied or otherwise distributed.

SSH Copy ↑
Copy your public key to the server. Execute this command from your local workstation.

localhost$ scp ~/.ssh/id_rsa.pub paul@255.255.255.255:

Now log in to your server with the admin user, create the .ssh directory, set up authorized_keys and limit its permissions.

$ ssh paul@255.255.255.255
$ mkdir ~/.ssh
$ mv ~/id_rsa.pub ~/.ssh/authorized_keys
$ chmod -R go-rwx ~/.ssh

Reconnect to the server and verify that the keys are interacting, bypassing the password prompt.

SSH Config ↑
People are generally incapable of agreeing on anything so there are a plethora of different suggestions as to how one should configure SSH on a server, but there are a few common threads we can adopt. One popular setting is to disable PasswordAuthentication, i.e. the only way a user can log in to the server is if he has an SSH key pair. This is more secure, but I tend to use a handful of different computers to access my servers and have a few guest accounts on them, so I opt to leave PasswordAuthentication on. Another security measure is to change the default SSH port of 22 to one of your chosing; there are a few reasons why this would be beneficial, but I will be using the default in this guide to strike a median between security and ease-of-use.

First create a group which limits who can access your server through SSH.

$ sudo groupadd sshers
$ sudo usermod -a -G sshers paul

Now edit the sshd_config file. You will have to add line 78 and 79 yourself.

$ sudo vim /etc/ssh/sshd_config

line:26 PermitRootLogin no
line:62 X11Forwarding no
line:77 UsePAM no
line:78 UseDNS no
line:79 AllowGroups sshers

Reload the SSH daemon and reconnect to the server to verify that you still can. Remember to always have at least one active connection to the server throughout when configuring SSH, that way you won't accidentally lock yourself out.

$ sudo /etc/init.d/ssh reload

.vimrc ↑
Include a few rudimentary settings for Vim. This is a very small sample of what's possible.

$ vim ~/.vimrc

" Use the latest
set nocompatible

" Better indent
set autoindent
set smartindent

" Set tabs to represent 4 spaces
set tabstop=4
set shiftwidth=4
set shiftround
set expandtab

.bashrc and .bash_aliases ↑
Create some very basic aliases. Feel free to improvise.

$ vim ~/.bash_aliases

alias ll='ls -lh'
alias la='ls -alh'

Open .bashrc and activate the .bash_aliases file. Uncomment line 67, 68 and 69.

$ vim ~/.bashrc

line:67 if [ -f ~/.bash_aliases ]; then
line:68     . ~/.bash_aliases
line:69 fi

Reload .bashrc to load the new aliases.

$ source ~/.bashrc

Locales ↑
Generate the necessary locales. This will of course vary depending on where you are located on the globe.

$ sudo locale-gen en_US.UTF-8
$ sudo /usr/sbin/update-locale LANG=en_US.UTF-8

Update and Upgrade ↑
Got to make sure that we have the latest and greatest upgrades. First update your local list of sources.

$ sudo aptitude update

Now install any eventual upgrades. Start with the safe-upgrade, then proceed to the full-upgrade.

$ sudo aptitude safe-upgrade
$ sudo aptitude full-upgrade

These are some general programs and utilities that you'll more than likely need at least once in your journeys.

$ sudo apt-get install build-essential git-core python-dev
$ sudo apt-get install python-setuptools python-psycopg2
$ sudo apt-get install subversion rsync postfix

Firewall ↑
A new installation of Ubuntu 8.04 LTS is by default open on all ports from any source, which isn't exactly optimal in this Brave New World. We're going to use Ubuntu's own Uncomplicated Firewall to configure our iptables to only accept connections from the default ports of HTTP, HTTPS and SSH. You might have to customize this to your liking, to include port 5432 for instance if you want to make your PostgreSQL database remotely accessible.

$ sudo apt-get install ufw

Enable ufw and turn on the logging feature.

$ sudo ufw enable
$ sudo ufw logging on

Allow access to the desired ports.

$ sudo ufw allow 22
$ sudo ufw allow 80/tcp
$ sudo ufw allow 443/tcp

Deny everything else.

$ sudo ufw default deny

Take a look at your newly configured iptables.

$ sudo iptables -L

Reload the SSH daemon yet again and reconnect to the server one last time.

$ sudo /etc/init.d/ssh reload

Finally, if all went well, disable password access to the root account.

$ sudo passwd -l root

Nginx ↑
Nginx is a light, but very fast and effective, web server which will act as a proxy to our Apache installation and serve static files.

Installation ↑

$ sudo apt-get install nginx

Config ↑
Optimize Nginx and create a proxy.conf file. Modify the worker_processes and uncomment tcp_nopush.

$ sudo vim /etc/nginx/nginx.conf

line:2 worker_processes  4;
line:18 tcp_nopush     on;

Create the proxy.conf file.

$ sudo vim /etc/nginx/proxy.conf

proxy_redirect          off;
proxy_set_header        Host            $host;
proxy_set_header        X-Real-IP       $remote_addr;
proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
client_max_body_size    10m;
client_body_buffer_size 128k;
proxy_connect_timeout   90;
proxy_send_timeout      90;
proxy_read_timeout      90;
proxy_buffers           32 4k;

Default Site ↑
Remove the default site displayed by Nginx. Reaching a point where the default page is displayed shouldn't be a terribly common event, but it might happen if using the IP of the server explicitly when browsing, opposed to the DNS entry.

$ sudo rm /etc/nginx/sites-enabled/default

Apache ↑
Apache is the most commonly used web server and the recommended way of serving Django sites in a production environment. There are a few brave souls experimenting with Nginx, lighttpd and various other solutions as the main web server, but Apache2 coupled with mod_wsgi is the preferred way of doing things. You can mozy on down to the Apache site here.

Installation ↑
A note to be made here is that we'll be using the libapache2-mod-wsgi package residing on the official repositories which is, as of this writing, version 1.3. The latest version of mod_wsgi is 2.3 and you can download it from here. The reason I opt to use the older version is that the main point of setting up an Ubuntu LTS server is to keep it stable and secure, not to run the latest, cutting edge software. You'll have to manually stop and start Apache after the installation to correctly set up mod_wsgi.

$ sudo apt-get install apache2 libapache2-mod-wsgi
$ sudo /etc/init.d/apache2 stop
$ sudo /etc/init.d/apache2 start

Config ↑
Towards the end of installing Apache you'll see this warning.

apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1 for ServerName

Apache loves FQDNs so let's do what any gallant administrator would do and satisfy its desire. Read more about FQDN here. We'll also modify another setting while we're at it, making Apache play nice with the other children. You'll have to add line 300.

$ sudo vim /etc/apache2/apache2.conf

line:77 KeepAlive Off
line:300 ServerName YourHostname

Ports ↑
Make Apache listen locally, Nginx will act as the front-end web server. NameVirtualHost must be added by you.

$ sudo vim /etc/apache2/ports.conf

line:1 Listen 127.0.0.1:8080
line:2 NameVirtualHost 127.0.0.1:8080

Default Site ↑
Lastly, remove the default site for the same reasons we removed the Nginx default site.

$ sudo rm /etc/apache2/sites-enabled/000-default

Now restart Apache and marvel at your newly configured web server. You'll get a warning but that will be taken care of once we actually map a Django project to a VirtualHost.

$ sudo /etc/init.d/apache2 restart
* Restarting web server apache2
[warn] NameVirtualHost 127.0.0.1:8080 has no VirtualHosts

PostgreSQL ↑
PostgreSQL is the database of choice for Django; I won't initiate a holy war by discussing the pros and cons of the various database choices, I'll leave that to others. You can find PostgreSQL's claim to some internet real estate here.

Installation ↑
You will actually have to run this command twice for Aptitude to correctly configure PostgreSQL.

$ sudo apt-get install postgresql-8.3 postgresql-server-dev-8.3
$ sudo apt-get install postgresql-8.3 postgresql-server-dev-8.3

Config ↑
Give the postgres user an actual password.

$ sudo -u postgres psql template1
template1=# ALTER USER postgres WITH PASSWORD 'password';
template1=# \q

Next, limit who can access the database by changing the METHOD for local.

$ sudo vim /etc/postgresql/8.3/main/pg_hba.conf

line:79 local   all         all          password

Restart PostgreSQL to activate the new settings.

$ sudo /etc/init.d/postgresql-8.3 restart

Users ↑
Create a personal superuser. You can read more about this here.

$ sudo -u postgres createuser -P -s -e paul

phpPgAdmin ↑
This handy piece of software let's you administer your databases through a web browser. We'll tie this to our Nginx and Apache scheme in the Django portion of the guide.

$ sudo apt-get install phppgadmin

Memcached ↑
Memcached is the preferred type of cache for Django, as explained here. The fastest Python binding for Memcached is currently cmemcache.

Installation ↑
Install the necessary packages.

$ sudo apt-get install memcached libmemcache-dev

Start the Memcached daemon. This delegates 64 megabytes of RAM, modify this to your needs.

$ sudo memcached -u www-data -p 11211 -m 64 -d

cmemcache ↑
We can't use Aptitude to install cmemcache so this will be a somewhat more laborious process than what we've been used to, but only marginally so. Create a general directory for the various libraries that we'll download, and grab the latest cmemcache version.

$ mkdir ~/libs
$ cd ~/libs
$ wget http://gijsbert.org/downloads/cmemcache/cmemcache-0.95.tar.bz2

Unpack the archive and go to its directory.

$ tar -xjvf cmemcache-0.95.tar.bz2
$ cd cmemcache-0.95

Use the supplied setup.py to install.

$ sudo python setup.py install

And that's it. A note if you're using a 64-bit OS, cmemcache will give you this warning during the installation procedure.

warning: format '%llu' expects type 'long long unsigned int', but argument 4 has type 'u_int64_t'

This is merely a warning and shouldn't have any tangible effect.

Django ↑
In the words of the Django marketing department: "Django is a high-level Python Web framework that encourages rapid development and clean, pragmatic design". It is, in my humble opinion, easily the best high-level web framework out there at the moment. You can read more about Django here.

One additional note before we install Django, there are a myriad of different ways to organize a Django project, none necessarily better than the other. Wait, that's not entirely true, I've seen some truly horrendous projects; but I digress. What I'm presenting here is a compromise between flexibility and simplicity. If you're planning to host multiple Django projects on the same server you should take a serious look at virtualenv.

Installation ↑
Download and install the latest official version.

$ cd ~/libs/
$ wget http://www.djangoproject.com/download/1.0.2/tarball/
$ tar -xzvf Django-1.0.2-final.tar.gz
$ cd Django-1.0.2-final
$ sudo python setup.py install

Default Project ↑
Create a basic directory skeleton and then a default project. I'll be using gen.ki as a domain.

$ sudo su - django
$ mkdir -p ~/domains/gen.ki/{public,private,log}
$ mkdir ~/domains/gen.ki/public/media
$ mkdir ~/domains/gen.ki/private/apache
$ cd ~/domains/gen.ki/
$ django-admin.py startproject genki
$ exit

Database ↑
Create a database and user for the Django project.

$ sudo su - postgres
$ createuser -P genki
Shall the new role be a superuser? (y/n) n
Shall the new role be allowed to create databases? (y/n) n
Shall the new role be allowed to create more new roles? (y/n) n
$ createdb --encoding=UNICODE genki -O genki
$ exit

Config ↑
Now we need to connect all the bits and pieces. Switch to the django user and stay logged in as him throughout this process.

$ sudo su - django

Link the admin media to the public media folder.

$ ln -s /usr/lib/python2.5/site-packages/django/contrib/admin/media/ ~/domains/gen.ki/public/media/admin

Modify the settings.py file for the Django project. Update the ADMINS, TIME_ZONE and such as you see fit, what I mention here is merely what is required. Feel free to read up on Django's cache and view a complete list of possible settings here.

$ vim ~/domains/gen.ki/genki/settings.py

line:12 DATABASE_ENGINE = 'postgresql_psycopg2'
line:13 DATABASE_NAME = 'genki'
line:14 DATABASE_USER = 'genki'
line:15 DATABASE_PASSWORD = 'password'
line:48 ADMIN_MEDIA_PREFIX = '/media/admin/'
line:60 MIDDLEWARE_CLASSES = (
line:61     'django.middleware.cache.UpdateCacheMiddleware',
line:62     'django.middleware.common.CommonMiddleware',
line:63     'django.contrib.sessions.middleware.SessionMiddleware',
line:64     'django.contrib.auth.middleware.AuthenticationMiddleware',
line:65     'django.middleware.cache.FetchFromCacheMiddleware',
line:66 )
line:70 TEMPLATE_DIRS = (
line:71     '/home/django/domains/gen.ki/genki/templates/',
line:72 )
line:81 CACHE_BACKEND = 'memcached://127.0.0.1:11211/'
line:82 CACHE_MIDDLEWARE_SECONDS = 60 * 5
line:83 CACHE_MIDDLEWARE_KEY_PREFIX = 'genki'

Now that we have a proper config we can sync the database.

$ python ~/domains/gen.ki/genki/manage.py syncdb

Exit from the django user's session.

$ exit

Nginx ↑
Create the Nginx proxy for the Django domain. The setting "expires 30d" indicates that the media files will only be refreshed every 30 days, you will obviously have to change this if the media files are modified frequently in your project.

$ sudo vim /etc/nginx/sites-available/gen.ki

server {
    listen   80;
    server_name www.gen.ki gen.ki;

    access_log /home/django/domains/gen.ki/log/nginx_access.log;
    error_log /home/django/domains/gen.ki/log/nginx_error.log;

    location / {
        proxy_pass    http://127.0.0.1:8080/;
        include       /etc/nginx/proxy.conf;
    }

    location /media/ {
        root   /home/django/domains/gen.ki/public/;
        expires 30d;
    }
}

Enable the site and restart Nginx.

$ sudo ln -s /etc/nginx/sites-available/gen.ki /etc/nginx/sites-enabled/gen.ki
$ sudo /etc/init.d/nginx stop
$ sudo /etc/init.d/nginx start

Apache ↑
Configure mod_wsgi to help Apache serve the project. Notice that we add both the domain and the Django project name to the path, this is due to any 3rd party applications we might include in our project at a later date.

$ sudo -u django vim /home/django/domains/gen.ki/private/apache/genki.wsgi

import os, sys

sys.path.append('/home/django/domains/gen.ki')
sys.path.append('/home/django/domains/gen.ki/genki')

os.environ['DJANGO_SETTINGS_MODULE'] = 'genki.settings'
import django.core.handlers.wsgi
application = django.core.handlers.wsgi.WSGIHandler()

Create a virtual host for the domain. You might have to modify the number of processes and threads the WSGI daemon spawns depending on your server environment.

$ sudo vim /etc/apache2/sites-available/gen.ki

<VirtualHost 127.0.0.1:8080>
    ServerAdmin admin@domain.com
    ServerName www.gen.ki
    ServerAlias gen.ki

    Alias /phppgadmin /usr/share/phppgadmin/

    <Directory /home/django/domains/gen.ki/genki/>
        Order deny,allow
        Allow from all
    </Directory>

    LogLevel warn
    ErrorLog  /home/django/domains/gen.ki/log/apache_error.log
    CustomLog /home/django/domains/gen.ki/log/apache_access.log combined

    WSGIDaemonProcess gen.ki user=www-data group=www-data threads=10
    WSGIProcessGroup gen.ki
    WSGIScriptAlias / /home/django/domains/gen.ki/private/apache/genki.wsgi
</VirtualHost>

Enable the site and restart Apache.

$ sudo ln -s /etc/apache2/sites-available/gen.ki /etc/apache2/sites-enabled/gen.ki
$ sudo /etc/init.d/apache2 restart

Epilogue ↑
And that's it. If you go to the URL you supplied as a ServerName you should now see the default Django page, provided that you've done the necessary DNS configuration. Also try going to /phppgadmin, the interface should load and you can log in with the personal superuser you made during this step.

So what now? One option is to actually learn how to make a Django site. Or you can set up a plethora of these Django servers and make a fortune redistributing them. ~~Or you can quit this computer nonsense and live happily ever after.~~

External Links ↑

Ethics Gradient

The Django and Ubuntu 8.04 Chronicle