Set up Django to use Aiven for Redis®*

Django is a fully-featured high-level Python web framework. Read on to see how to get it to cache its data using Aiven for Redis®*.

Subscribe to RSS

In a previous article we looked at using an Aiven for PostgreSQL® database as the backend for Django. But Aiven also provides other managed services, including Redis®*. So in this article we are going to look at adding a Redis cache to the previous example.

Summary of the previous article

The basics

If you just worked through the previous article, then great, you're all ready to continue with this tutorial, and can skip the rest of this section.

Otherwise, if you just want to work with this article, then you need to get Oscar setup. Here's a summary of the basics:

You will need node and npm (on my Mac I use brew install npm to install them). Then:

 git clone --branch sandbox-requirements-update \
     https://github.com/TibsAtWork/django-oscar.git
 cd django-oscar
 python3 -m venv venv
 source venv/bin/activate
 make sandbox

You can test that by running the following command and going to http://127.0.0.1:8000/ in a web browser to see the Oscar Sandbox store front.

sandbox/manage.py runserver

Talking to Aiven for PostgreSQL®

This leaves you with Oscar storing its data in a local SQLite database. To change to using an Aiven for PostgreSQL® instance, follow the instructions at Create a PostgreSQL® database and Use PostgreSQL as the backend database, from the previous article.

Of course, you can leave Oscar talking to the local SQLite database, although running a cache in the cloud for a local database is not something one would expect to do in real life.

Continue using the Oscar e-commerce example

Note

If you are carrying on from the previous example (so skipped to here) then let's just check everything is still set up correctly.

Remember to make sure you're in the django-oscar directory - if necessary, cd into it:

cd django-oscar

Also make sure that the Python virtual environment is enabled. If not, do:

source venv/bin/activate

Check everything still works by running the sandbox application again with:

sandbox/manage.py runserver

Make sure that the bookshop appears at http://127.0.0.1:8000/.

Then check if the DATABASE_ environment variables are still set, to tell Oscar to use the remote PostgreSQL database:

printenv | grep DATABASE_

which should show something like:

DATABASE_PORT=10143
DATABASE_NAME=defaultdb
DATABASE_PASSWORD=YOUR_DATABASE_PASSWORD_HERE
DATABASE_USER=avnadmin
DATABASE_HOST=tibs-django-pg-project-tibs.aivencloud.com
DATABASE_ENGINE=django.db.backends.postgresql_psycopg2

(that's not a real password there, and your host name should be different as
well).

Note

If you're using Aiven for PostgreSQL, then all of those environment variables should match the values in the service's Overview page in the Aiven Console.

Disable the Django Debug Toolbar

The Oscar Sandbox
documentation
warns that: "The sandbox has Django Debug Toolbar enabled by default, which will affect its performance. You can disable it by setting INTERNAL_IPS to an empty list in your local settings."

Since the point of using Redis as a cache is to improve performance, we should do that.

Edit sandbox/settings.py and search for INTERNAL_IPS. It's probably around line 364. Change the line:

INTERNAL_IPS = ['127.0.0.1', '::1']

to

INTERNAL_IPS = []

Create a Redis®* service

First we need to start a new Aiven for Redis service using the Aiven Console.

  • I want the same cloud (Google Cloud) and location (google-europe-north1) as for the PostgreSQL service
  • Again, Service Plan "Hobbyist" should do for this demo
  • Following my previous naming convention, I named it "tibs-django-redis"

Export the Redis Service URI from the Aiven Console to the following environment variable:

export REDIS_SERVICE_URI='<Service URI>'

## Install the Redis CLI

We want to be able to "talk" to the Redis server, so let's install the Redis
command line tool, `redis-cli`, as described at [connect with redis-cli](https://aiven.io/docs/products/redis/howto/connect-redis-cli.html)

For instance, on on my Mac I can install Redis locally:

```shell
brew install redis

Then run the command using the Redis service's URL from the service overview page:

redis-cli -u $REDIS_SERVICE_URI

This will leave us at a prompt naming the HOST and PORT:

HOST:PORT>

In my case, it looked something like:

tibs-django-redis-project-tibs.aivencloud.com:10144>

We can then ask what keys are in my Redis datastore:

INFO KEYSPACE

At this stage, the keystore is empty:

# Keyspace

We can quit redis-cli using:

QUIT

Tip

It's useful to be able to run redis-cli from a different terminal window, so you can watch how things change while interacting with the running Oscar web app. Don't forget to set REDIS_SERVICE_URI in that second terminal as well!

About Oscar and Django versions

As of the start of June 2023, the current version of Oscar is still 3.2, and it uses Django 3.2.

In the near future, Oscar 4.2 will be released, using Django 4.2.

How can you tell which you've got? You can check the version of Django by typing:

pip show django | grep Version

You can do the same for the version of Oscar:

pip show django-oscar | grep Version

Luckily, this only makes a difference in how to tell Django to use Redis as its cache, where we'll explain for both versions.

Info

It's useful to explain how Django 3.2 works, even if Oscar has moved on, because 3.2 is a Long Term Support (LTS) release which will be supported until at least April 2024. This means it's quite likely that you'll find projects using it until at least that date, and likely beyond as well.

The relevant Python libraries

Setting up the environment for Oscar (in the make sandbox step) will already have installed the necessary Python libraries to allow the application to talk to Redis. You can see the specification for this in requirements.txt - look for the string "redis" in the Sandbox section.

If we were using plain Django, we'd need to install the libraries ourselves, using:

pip install redis

If you're using Oscar 3.2 we also need to install django-redis:

pip install django-redis

Tell Django we're going to be doing caching

Edit the file sandbox/settings.py.

Find the MIDDLEWARE = [ definition, and add the following line to the start of the list:

    'django.middleware.cache.UpdateCacheMiddleware',

Then add the following line to the end of the list:

    'django.middleware.cache.FetchFromCacheMiddleware',

After that, it should look like:

MIDDLEWARE = [
    'django.middleware.cache.UpdateCacheMiddleware',
    'debug_toolbar.middleware.DebugToolbarMiddleware',

    'django.middleware.security.SecurityMiddleware',
    'whitenoise.middleware.WhiteNoiseMiddleware',

    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.contrib.flatpages.middleware.FlatpageFallbackMiddleware',

    # Allow languages to be selected
    'django.middleware.locale.LocaleMiddleware',
    'django.middleware.http.ConditionalGetMiddleware',
    'django.middleware.common.CommonMiddleware',

    # Ensure a valid basket is added to the request instance for every request
    'oscar.apps.basket.middleware.BasketMiddleware',
    'django.middleware.cache.FetchFromCacheMiddleware',
]

Note

The details of the listed middleware may be different in your setup, but the location of the new entries is the important part.

For more on the ordering of the middleware, see the documentation for per-site caching.

Tell Django to use (this) Redis as its cache

We need to tell Django to actually use Redis for caching, and where to find the Redis service.

How to do this is slightly different depending on the version of Django.

For Django 3.2

Since Django 3.2 doesn't come with built in support for using Redis as a cache, it's necessary to use an external package, django-redis. Luckily, setting up the environment for Oscar will already have done pip install django-redis for us, but if you're using some other Django application, you'd need to do that yourself.

Still in sandbox/settings.py, find the CACHES = { definition, and change it from:

CACHES = {
    'default': env.cache(default='locmemcache://'),
}

to:

CACHES = {
    'default': {
        'BACKEND': 'django_redis.cache.RedisCache',
        'LOCATION': os.environ.get('REDIS_SERVICE_URI'),
        'OPTIONS': {
            'CLIENT_CLASS': 'django_redis.client.DefaultClient',
        },
    },
}

This tells it to use the RedisCache backend from django-redis, and to find the instance of
Redis at the URL specified by that environment variable (which we set earlier).

See django-redis itself for more documentation, including on the OPTIONS it uses.

For Django 4.2

Django 4 comes with Redis support built in.

Still in sandbox/settings.py, find the CACHES = { definition, and change it from:

CACHES = {
    'default': env.cache(default='locmemcache://'),
}

to:

CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.redis.RedisCache',
        'LOCATION': os.environ.get('REDIS_SERVICE_URI'),
    },
}

This tells it to use Redis as a backend cache, and to find the instance of
Redis at the URL specified by that environment variable (which we set earlier).

Use variables rather than constants

Although the Django cache framework documentation section on Redis shows setting the LOCATION explicitly to a specific string, it's better practice not to embed such strings into (what might be) production code. In this instance, we know that the REDIS_SERVICE_URI includes a password, as well as other connection details, so we don't want to commit those details to version control, or even expose them unnecessarily in our CI (continuous integrations). Using an environment variable allows us to add this information at run time instead, and use a secrets manager in the actual production environment.

Set the cache behavior

Still in sandbox/settings.py, add the following settings after the CACHES
definition (add them on lines after the final closing } of the CACHES definition -
these lines should not be indented):

CACHE_MIDDLEWARE_ALIAS = 'default'
CACHE_MIDDLEWARE_KEY_PREFIX = ''
CACHE_MIDDLEWARE_SECONDS = 60 * 10

This is actually setting the default behavior, but it seems sensible to be explicit about our choices, and the Django per-site
cache documentation recommends adding these settings.

Show it in action

Let's restart the sandbox application:

sandbox/manage.py runserver

Then log out and back in, add another book or two to the basket, and open the basket.

Run redis-cli agaom:

redis-cli -u $REDIS_SERVICE_URI

Ask it again about the keys it is storing:

INFO KEYSPACE

This time we should see something like this in response:

db0:keys=32,expires=32,avg_ttl=187151332590

In this case, it's telling us that:

  • there are now 32 keys, so 32 cached items (the actual number will vary according to what you did)
  • all 32 of them have expiration times set

The last value, avg_ttl, isn't terribly useful in this case, as it indicates the average TTL for all of those keys, and some of them have very large values set, presumably to stop them expiring.

Getting more detailed, if we then type:

KEYS *

We should then see a list similar to the following:

 1) ":1:CATEGORY_URL_en-gb_3"
 2) ":1:oscar-sandbox||image||ff719c9a5892bf5b4680eb86da85f5aa"
 3) ":1:CATEGORY_URL_en-gb_4"
 4) ":1:oscar-sandbox||image||20242e100a811986ece16c1fac3b2a57"
 5) ":1:views.decorators.cache.cache_header..8ac2d55b8eaa00e52c563c9a18db6736.en-gb.Europe/London"
 6) ":1:oscar-sandbox||image||f544ccac158defae9a7da221d5a79d61"
 7) ":1:views.decorators.cache.cache_header..357698008328fc178c9adfab49a0d197.en-gb.Europe/London"
 8) ":1:CATEGORY_URL_en-gb_7"
 9) ":1:CATEGORY_URL_en-gb_6"
10) ":1:oscar-sandbox||image||05221781d86552e0ab294b1ba2b4d977"
11) ":1:oscar-sandbox||image||0daf8fe2289a1bccdad5cad5a97abefa"
12) ":1:oscar-sandbox||image||eee159e62c22e44531117720e9d9e3e7"
13) ":1:oscar-sandbox||image||f53672bae89061349c4beb1d55721858"
14) ":1:views.decorators.cache.cache_page..GET.357698008328fc178c9adfab49a0d197.95da522d651f032384984b71dab9c668.en-gb.Europe/London"
15) ":1:oscar-sandbox||image||c08074a3afe5aba9e44b7b2a725266c8"
16) ":1:views.decorators.cache.cache_page..GET.8ac2d55b8eaa00e52c563c9a18db6736.7ff9cc28df2e7c550de2e09525c9bf06.en-gb.Europe/London"
17) ":1:oscar-sandbox||image||146cc1138e35c43b8f5a8dc41370dda9"
18) ":1:views.decorators.cache.cache_page..GET.8ac2d55b8eaa00e52c563c9a18db6736.ed66662513025556335a676d702540bc.en-gb.Europe/London"

Let's choose one that sounds likely to represent a page view (a key with
view.decorators.cache.cache_page..GET in its name).

Getting the value for the key I chose:

GET ":1:views.decorators.cache.cache_page..GET.357698008328fc178c9adfab49a0d197.95da522d651f032384984b71dab9c668.en-gb.Europe/London"

Gives me back a "Django template response" with an HTML page embedded in it. The response starts with:

"\x80\x05\x95\x98b\x00\x00\x00\x00\x00\x00\x8c\x18django.template.response\x94\x8c\x10TemplateResponse\x94

And ends with the following:

<title>\n    Basket | Oscar - Sandbox\n</title>

We can also ask for the TTL:

TTL ":1:views.decorators.cache.cache_page..GET.357698008328fc178c9adfab49a0d197.95da522d651f032384984b71dab9c668.en-gb.Europe/London"

Which returns something similar to the following:

(integer) 474

And if you repeat the TTL command for the same key, you will see the time decreasing.

In fact, if you wait long enough, all of the page view keys will disappear, as they time out, leaving only the image keys - here we can see the first four of those.

 1) ":1:oscar-sandbox||image||ff719c9a5892bf5b4680eb86da85f5aa"
 2) ":1:oscar-sandbox||image||20242e100a811986ece16c1fac3b2a57"
 3) ":1:oscar-sandbox||image||f544ccac158defae9a7da221d5a79d61"
 4) ":1:oscar-sandbox||image||05221781d86552e0ab294b1ba2b4d977"

Change the TTL

Stop the application and edit sandbox/settings.py to change the cache TTL to be 20 (twenty seconds):

CACHE_MIDDLEWARE_SETTINGS = 20

Then restart the application and refresh the current page (which in my case was still showing the basket). Use redis-clis to ask for the TTL of a content page again. For example:

TTL ":1:views.decorators.cache.cache_page..GET.357698008328fc178c9adfab49a0d197.95da522d651f032384984b71dab9c668.en-gb.Europe/London"

This time, the values returned should be lower - in my case, I saw 11 because
I wasn't quick enough to see it at 20!

(integer) 11

In other words, all of the page keys will expire within that shorter TTL.

What we've achieved

In the last post, I explored how to use an Aiven service (Aiven for
PostgreSQL®) as a backend for a web platform I already knew,
Django. To avoid the lengthy process of
setting up my own Django application, I decided to use an existing one, the
Oscar sandbox.

In this post, I showed how to add a Redis cache to that web application. As
before, this was not too hard to setup, and it's pleasing to be able to use
Aiven services for both the database and the cache.

More things to look at

The Oscar documentation has a lot more
information about the project.

Both Django 3.2 and 4.2 are Long Term Support (LTS) releases.

For more information on Django 3.2 LTS (supported until April 2024), check out:

For more information on Django 4.2 LTS (supported until at least 2026), check out:

If you're just wanting to see the current state of Django, then check out the latest version of the Django documentation.

Also check out Aiven for Redis®*,
and if you're not using Aiven services yet, go ahead and sign up now for your free trial at https://console.aiven.io/signup.