Table of Contents

This article is part of the "django-app-enabler" series

As we have seen in Automating Django application configuration, you can manipulate your project settings by using python to modify python code.

But how does it works in practice?

Let’s go in rather verbose step-by-step details about how django-app-enabler works

AST - Writing python files in python

Without going into much details, we can summarize ast as way to map the content of a file (or a string) in a python structure that we can manipulate and write back to file (see Wikipedia and python docs for more formal description of AST).

It means we can read a python file and modify it without parsing its text content, and just work with a python data structure, which of course is much more robust, easier to manipulate and guarantees a valid python file when we write it back.

We are going to use astor in all our examples as it provides a lot of high-level functions to work with AST.

This is a very simplistic example of parsing and editing a python file (by adding a var = "value" line at the bottom):

1
2
3
4
5
6
7
8

parsed = astor.parse_file("my_file.py")

parsed.body.append(ast.Assign(targets=[ast.Name(id="var")], value=ast.Constant("value")))

src = astor.to_source(parsed)

with open(projec"my_file.py", "w") as file_fp:
    file_fp.write(src)

In a more realistic worflow, we are going to walk the syntax tree and either react to each node encountered, or writing methods called for node types of interests, called when each node type is encountered during the tree walk by astor.

See this example in which we parse the django manage.py file and remove a specific function call from it:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

class DisableExecute(ast.NodeTransformer):
    """
    Patch the ``manage.py`` module to remove the execute_from_command_line execution.
    """

    def visit_Expr(self, node: ast.AST) -> Any:
        """
        Visit the ``Expr`` node and remove it
        if it matches ``'execute_from_command_line'``
        """
        if (
            isinstance(node.value, ast.Call)
            and isinstance(node.value.func, ast.Name)
            and node.value.func.id == "execute_from_command_line"
        ):
            return None
        else:
            return node

parsed = astor.parse_file("manage.py")

modified = DisableExecute().visit(parsed)

src = astor.to_source(modified)

with open(projec"manage.py", "w") as file_fp:
    file_fp.write(src)

We can thus count on a powerful tool to work on our files.

A first approach

Configuring a django project involves two django data structures: settings and urlconf, both written in python:

  • settings is made of python variables in a module (or a package), mostly with literals values or rather simple dictionaries and lists;

  • urlconf has one (or more) assignments to the urlpatterns variable which in the end is a list of function calls;

Things start looks interesting 🤯.

Just to get the balls rolling, imagine we have a very simple application which only needs to be added to the project applications and its urls added to urlconf:

settings.py:

1
2
3
4
5
6

INSTALLED_APPS = (
    ...
    "my_app",
    ...

)

urls.py:

1
2
3

urlpatterns += [
    path("myapp", include("myapps.urls"))
]

Settings

In this first case, we only need to add the application to INSTALLED_APPS; as our settings file is a flat structure of variables, the code to do this is something like:

1
2
3
4
5
6
7
8
9

parsed = astor.parse_file("settings.py")
for node in parsed.body:
    if isinstance(node, ast.Assign) and node.targets[0].id == "INSTALLED_APPS":
        node.value.elts.extend("myapp")

src = astor.to_source(parsed)

with open("settings.py", "w") as settings:
    settings.write(src)

URLConf

Patching urlconf is not much different:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

parsed = astor.parse_file("urls.py")
for node in parsed.body:
    if isinstance(node, ast.Assign) and node.targets[0].id == "urlpatterns":
        url = "myapp.urls"
        app_prefix = "myapp"
        part = ast.parse(f"path('{app_prefix}', include('{url}'))")
        node.value.elts.append(part.body[0].value)

src = astor.to_source(parsed)

with open("urls.py", "w") as urlconf:
    urlconf.write(src)

Wrapping up the first attempt

If you have a sample project and a myapp application, and you execute the snippets above, you will have our application installed and running in the project.

Entering django-app-enabler

The snippets above are far from a comprehensive solution, but by interating over them we can get closer to our goal.

Just to save you these iterations, I wrapped my attempts in a package (django-app-enabler) whose goal is to allow django application configuration with a single command.

The general idea is to tap into the project manage.py command to automatically hook into the project code and use json files shipped in installed applications, or provided via options to update the setting and the urlconf.

This involve a lot of (at the moment) assumptions, that hopefully will be removed/eased as the project mature.

django-app-enabler works in three steps:

  • detecting the Django project
  • loading the application configuration
  • patching the Django project

Detecting the Django project

Once we have the django project folder at hand, for us humans it’s usually not complicated to understand the project structure and which file is which.

For computers it’s not that easy.

Luckily we can bend the exising Django code to help us in this process.

The manage.py command, in fact, must know how to setup the django project and where its file are, because it’s the main entrypoint to interact with the Django code from the command line.

But it’s not meant to be imported and invoked from external code, so what can we do?

We can manipulate it with ast.

Take the standard Django 2.2 manage.py file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

#!/usr/bin/env python
import os
import sys


def main():
    os.environ.setdefault("DJANGO_SETTINGS_MODULE", "test_project.settings")
    try:
        from django.core.management import execute_from_command_line
    except ImportError as exc:
        raise ImportError(...) from exc
    execute_from_command_line(sys.argv)


if __name__ == "__main__":
    main()

The basic information we need is the value of the DJANGO_SETTINGS_MODULE, to know what’s the project settings file which will provide us the rest of the project configuration.

One strategy would be to scan the file for DJANGO_SETTINGS_MODULE and to get its value, but a different approach should be more robust in terms of file customizations by the project developer or future changes by the django project.

Django needs the DJANGO_SETTINGS_MODULE value at startup and everything that can alter it is executed in this file (either directly ) or by importing external code: if we run a modified manage.py by replacing the call to execute_from_command_line with the plain django.setup which just loads the project configuration, we will have the project valid configuration without knowing how it has been loaded.

django-app-enabler achieve this in two steps.

Loading manage.py

First we load manage.py in memory, we adjust as needed, and we compile it in executable format without changing the file on the filesystem:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

def monkeypatch_manage(manage_file: str) -> CodeType:
    """
    Patch ``manage.py`` to be executable without actually running any command.
    """
    parsed = astor.parse_file(manage_file)
    # remove `execute_from_command_line`
    modified = DisableExecute().visit(parsed)
    # add a call to the `main` function
    main_call = ast.Call(func=ast.Name(id="main", ctx=ast.Load()), args=[], keywords=[])
    modified.body.append(ast.Expr(value=main_call))
    fixed = ast.fix_missing_locations(modified)
    return compile(fixed, "<string>", mode="exec")

The result of this function is equivalent to the code below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

#!/usr/bin/env python
import os
import sys


def main():
    os.environ.setdefault("DJANGO_SETTINGS_MODULE", "test_project.settings")
    try:
        from django.core.management import execute_from_command_line
    except ImportError as exc:
        raise ImportError(...) from exc
    # call to `execute_from_command_line` has been removed


if __name__ == "__main__":
    main()
# main function is unconditionally called
main()

execute_from_command_line has been removed, call to main() function has been added at the end because when we execute this code __name__ will not be __main__ as it’s not going to be executed from the command line.

Setup the django project

If we execute the monkeypatch_manage function defined above, we will have everything configured (included os.environ) as needed for django setup to work:

1
2
3
4
5
6

def setup_django():
    import django

    managed_command = monkeypatch_manage("manage.py")
    eval(managed_command)
    django.setup()

After eval‘ing the patched manage command, we can call django.setup and we have setup the target django project within the django-app-enabler code 😎.

Loading the application configuration

As we want to configure a third party application we must know what configuring it means.

This is something the application itself must provide (but this limitation will be lifted soon), and again, we must detect it at runtime.

We can tap in already existing functionalities to achieve this.

setuptools comes with a pkg_resources package that can list packages installed in the current virtualenv and read their files and metadata.

Thus django-app-enabler define that configurable applications must define a configuration file in the root of the package directory, so that it can be discovered at runtime by only knowing the pypi package name:

1
2
3
4
5
6

from pkg_resources import get_distribution, Requirement, resource_stream

distribution = get_distribution(Requirement.parse(package))
application_name = distribution.get_metadata("top_level.txt").split()[0]
with resource_stream(application_name, "addon.json") as fp:
    addon_config = json.load(fp)

First we get the pkg_resource object wrapping the installed package via pkg_resources.get_distribution; by parsing the package name as Requirement object we can use any valid requirements string (i.e.: something like djangocms-blog~=1.2.0 will work) to load it.

Then we tap into the package metadata and read the list of its available modules: we pick the first, which always exists. This may look strict at first, but as the file we want to load is defined by django-app-enabler itself and it requires the developer of the target application to create this file, we can easily enforce this apparent restriction.

As we now have the python module name, we can read any arbitrary file content via pkg_resources.resource_stream and load it as json

Patching the Django project

We can now (😅) go into our real business and update the project.

For now the support is limited to single file settings.py and single file urls.py, hopefully this will be relaxed in the future (see limitations), but this already allows to support quite a few use cases.

Patch settings

A Django application can potentially needs changes to any existing setting, and add their own.

The complexity of automatic altering the Django settings is the merge strategy when application configuration defines a matching setting.

The current strategy (implemented in app_enabler.patcher.update_settings, not reported for brevity) is very basic and it needs work before coming closer to a general purpose stratregy:

  • If the application setting variable does not exists in Django setting, add it;
  • If the application setting variable exists in Django setting and they are both lists, merge them by appending application configuration values to the Django one
  • Else ignore the setting (this to avoid that an application configuration can break an existing project)

The most notable missing parts from this strategy are:

  • support for inserting items in specific lists positions (think of ordering of middleware and application in INSTALLED_APPS)
  • support for merging dictionaries or more complex structures (like TEMPLATES)

The former is currently the most relevant missing piece, as the latter is a far less common need for normal applications.

Patch urlconf

Urlconf has a theorical simpler structure so we are in a slightly better position here, but in practice this complicated by the fact that, contrary to the django setting, its structure can vary a lot between different projects. You can have multiple statements inside urlpatterns and multiple urlpatterns assignments it, you may have i18npatterns statements etc.

But it’s possible to impose some restriction on the configuration options available to the application without limiting the applicability:

  • pattern must be in the form supported by path() function
  • only include urlconf are supported

So all the urlconf configuration will be added in the form of:

1
2
3
4
5

    urlpatterns += [
        ...
        path("<application-pattern>", include("<application-urlconf>))
        ...
    ]

As of now, no replacement strategy is implemented: application urlconf are simply appended in the bottommost urlpatterns assignment, if no other include of the same urlconf is already added to the list.