Automating Django application configuration: an in-depth view
Table of Contents
This article is part of the "django-app-enabler" series
As we have seen in Automating Django application configuration, you can manipulate your project settings by using python to modify python code.
But how does it works in practice?
Let’s go in rather verbose step-by-step details about how
AST - Writing python files in python
Without going into much details, we can summarize
ast as way to map the content of a file (or a string) in a python structure that we can manipulate and write back to file (see Wikipedia and python docs for more formal description of AST).
It means we can read a python file and modify it without parsing its text content, and just work with a python data structure, which of course is much more robust, easier to manipulate and guarantees a valid python file when we write it back.
We are going to use astor in all our examples as it provides a lot of high-level functions to work with
This is a very simplistic example of parsing and editing a python file (by adding a
var = "value" line at the bottom):
In a more realistic worflow, we are going to walk the syntax tree and either react to each node encountered, or writing methods called for node types of interests, called when each node type is encountered during the tree walk by
See this example in which we parse the django
manage.py file and remove a specific function call from it:
We can thus count on a powerful tool to work on our files.
A first approach
Configuring a django project involves two django data structures:
urlconf, both written in python:
settingsis made of python variables in a module (or a package), mostly with literals values or rather simple dictionaries and lists;
urlconfhas one (or more) assignments to the
urlpatternsvariable which in the end is a list of function calls;
Things start looks interesting 🤯.
Just to get the balls rolling, imagine we have a very simple application which only needs to be added to the project applications and its urls added to
In this first case, we only need to add the application to
INSTALLED_APPS; as our
settings file is a flat structure of variables, the code to do this is something like:
urlconf is not much different:
Wrapping up the first attempt
If you have a sample project and a
myapp application, and you execute the snippets above, you will have our application installed and running in the project.
The snippets above are far from a comprehensive solution, but by interating over them we can get closer to our goal.
Just to save you these iterations, I wrapped my attempts in a package (django-app-enabler) whose goal is to allow django application configuration with a single command.
The general idea is to tap into the project manage.py command to automatically hook into the project code and use json files shipped in installed applications, or provided via options to update the setting and the urlconf.
This involve a lot of (at the moment) assumptions, that hopefully will be removed/eased as the project mature.
django-app-enabler works in three steps:
- detecting the Django project
- loading the application configuration
- patching the Django project
Detecting the Django project
Once we have the django project folder at hand, for us humans it’s usually not complicated to understand the project structure and which file is which.
For computers it’s not that easy.
Luckily we can bend the exising Django code to help us in this process.
manage.py command, in fact, must know how to setup the django project and where its file are, because it’s the main entrypoint to interact with the Django code from the command line.
But it’s not meant to be imported and invoked from external code, so what can we do?
We can manipulate it with
Take the standard Django 2.2
The basic information we need is the value of the
DJANGO_SETTINGS_MODULE, to know what’s the project settings file which will provide us the rest of the project configuration.
One strategy would be to scan the file for
DJANGO_SETTINGS_MODULE and to get its value, but a different approach should be more robust in terms of file customizations by the project developer or future changes by the django project.
Django needs the
DJANGO_SETTINGS_MODULE value at startup and everything that can alter it is executed in this file (either directly ) or by importing external code: if we run a modified
manage.py by replacing the call to
execute_from_command_line with the plain
django.setup which just loads the project configuration, we will have the project valid configuration without knowing how it has been loaded.
django-app-enabler achieve this in two steps.
First we load
manage.py in memory, we adjust as needed, and we compile it in executable format without changing the file on the filesystem:
The result of this function is equivalent to the code below:
execute_from_command_line has been removed, call to
main() function has been added at the end because when we execute this code
__name__ will not be
__main__ as it’s not going to be executed from the command line.
Setup the django project
If we execute the
monkeypatch_manage function defined above, we will have everything configured (included
os.environ) as needed for django setup to work:
eval‘ing the patched manage command, we can call
django.setup and we have setup the target django project within the
django-app-enabler code 😎.
Loading the application configuration
As we want to configure a third party application we must know what configuring it means.
This is something the application itself must provide (but this limitation will be lifted soon), and again, we must detect it at runtime.
We can tap in already existing functionalities to achieve this.
setuptools comes with a
pkg_resources package that can list packages installed in the current virtualenv and read their files and metadata.
django-app-enabler define that configurable applications must define a configuration file in the root of the package directory, so that it can be discovered at runtime by only knowing the pypi package name:
First we get the
pkg_resource object wrapping the installed package via
pkg_resources.get_distribution; by parsing the package name as
Requirement object we can use any valid requirements string (i.e.: something like
djangocms-blog~=1.2.0 will work) to load it.
Then we tap into the package metadata and read the list of its available modules: we pick the first, which always exists. This may look strict at first, but as the file we want to load is defined by
django-app-enabler itself and it requires the developer of the target application to create this file, we can easily enforce this apparent restriction.
As we now have the python module name, we can read any arbitrary file content via
pkg_resources.resource_stream and load it as json
Patching the Django project
We can now (😅) go into our real business and update the project.
For now the support is limited to single file
settings.py and single file
urls.py, hopefully this will be relaxed in the future (see limitations), but this already allows to support quite a few use cases.
A Django application can potentially needs changes to any existing setting, and add their own.
The complexity of automatic altering the Django settings is the merge strategy when application configuration defines a matching setting.
The current strategy (implemented in
app_enabler.patcher.update_settings, not reported for brevity) is very basic and it needs work before coming closer to a general purpose stratregy:
- If the application setting variable does not exists in Django setting, add it;
- If the application setting variable exists in Django setting and they are both lists, merge them by appending application configuration values to the Django one
- Else ignore the setting (this to avoid that an application configuration can break an existing project)
The most notable missing parts from this strategy are:
- support for inserting items in specific lists positions (think of ordering of middleware and application in
- support for merging dictionaries or more complex structures (like
The former is currently the most relevant missing piece, as the latter is a far less common need for normal applications.
Urlconf has a theorical simpler structure so we are in a slightly better position here, but in practice this complicated by the fact that, contrary to the django setting, its structure can vary a lot between different projects. You can have multiple statements inside
urlpatterns and multiple
urlpatterns assignments it, you may have
i18npatterns statements etc.
But it’s possible to impose some restriction on the configuration options available to the application without limiting the applicability:
- pattern must be in the form supported by
- only include urlconf are supported
So all the urlconf configuration will be added in the form of:
As of now, no replacement strategy is implemented: application urlconf are simply appended in the bottommost
urlpatterns assignment, if no other include of the same urlconf is already added to the list.