Blog

Dynamic Django Models with Model Models

Warning: You probably don't want to do this because it's unsupported and very hacky. Consider JSONField.

Django models are usually written in a models.py file by hand, then migrations are written to disk with

./manage.py makemigrations

./manage.py migrate

If you want to make or change models at runtime, there's little help apart from a talk from 2011.

Some applications of Dynamic Models are:

• Surveys
• Prototyping
• Storing data from many different sources
• Alternative to EAV / JSON

A simple Model looks like:

class Book(models.Model):
    title = models.TextField()
    pages = models.IntegerField()

In this case we have three things to store:

• The model name
• The title field
• The pages field

To store that, we need a Model Model, and a Field Model. To make the naming slightly easier, we'll call them MakeModel and MakeField.

class MakeModel(models.Model):
    name = models.CharField(verbose_name='Model Name', max_length=50, unique=True)

    def __str__(self):
        return self.name

FIELD_TYPE_CHOICES = (
    (name, name) for name in ['TextField', 'IntegerField']
)

class MakeField(models.Model):
    name = models.CharField(max_length=50)
    make_model = models.ForeignKey(MakeModel, on_delete=models.CASCADE)
    field_type = models.CharField(max_length=255, choices=FIELD_TYPE_CHOICES)

Django Admin

The quickest way to interact with MakeModel is to use django admin:

from django.contrib import admin
from .models import MakeModel, MakeField


class MakeFieldInline(admin.TabularInline):
    model = MakeField


class MakeModelAdmin(admin.ModelAdmin):
    inlines = [MakeFieldInline]


admin.site.register(MakeModel, MakeModelAdmin)

To construct a model dynamically, we can use "type". For example, to create the Book Model:

model_cls = type(
    'Book',
    (models.Model, ),
    {
        'title': models.TextField(),
        'pages': models.IntegerField(),
    },
)

To do this from the MakeModel (There's a lot of magic here):

def reload_models():
    for makemodel in MakeModel.objects.all():
        attrs = {'__module__': 'meta'}
        for field in makemodel.makefield_set.all():
            model_field_cls = getattr(models, field.field_type)
            attrs[field.name] = model_field_cls()
        model_cls = type(
            makemodel.name,
            (models.Model, ),
            attrs,
        )

Every time we update MakeModel or MakeField, we need to reload our generated Models, which we can do using a signal:

@receiver(post_save, sender=MakeModel)
def model_save(sender, instance, created, **kwargs):
    reload_models()


@receiver(post_save, sender=MakeField)
def field_save(sender, instance, created, **kwargs):
    reload_models()

Remember signals need default_app_config and an import in "ready" in AppConfig.

Migrations

We've got the model in memory, but the next thing we need is the model in the database. If we start by looking at the source for the makemigrations command, we can figure out how to store migrations in the database.

The Migration model just needs to store the contents of what would usually be in a migration file:

class MakeMigration(models.Model):
    name = models.CharField(verbose_name='Migration Name', max_length=200)
    content = models.TextField()

    def __str__(self):
        return self.name

It takes a bit of hackery to generate migrations automatically: 1) We need to write changes to the database

def write_migration_rows(changes):
    for app_label, app_migrations in changes.items():
        for migration in app_migrations:
            writer = MigrationWriter(migration)
            migration_string = writer.as_string()
            MakeMigration.objects.create(
                name=writer.migration.name,
                content=migration_string
            )

class MyLoader(importlib.machinery.SourceFileLoader):
    def get_data(self, path):
        return MakeMigration.objects.get(name=path).content

def existing_migrations():
    existing = {}
    for makemigration in MakeMigration.objects.all():
        loader = MyLoader('a_b', makemigration.name)
        mod = types.ModuleType(loader.name)
        loader.exec_module(mod)
        existing[('meta', makemigration.name)] = mod.Migration(
            makemigration.name, 'meta')
    return existing

def write_migrations():
    import django.db.migrations.loader
    old_load_disk = django.db.migrations.loader.MigrationLoader.load_disk

    def load_disk(self):
        old_load_disk(self)
        self.disk_migrations.update(existing_migrations())

    django.db.migrations.loader.MigrationLoader.load_disk = load_disk

    loader = MigrationLoader(None, ignore_no_migrations=True)
    questioner = NonInteractiveMigrationQuestioner()
    autodetector = MigrationAutodetector(
        loader.project_state(),
        ProjectState.from_apps(apps),
        questioner,
    )
    app_label = 'meta'
    app_labels = {app_label, }
    changes = autodetector.changes(
        graph=loader.graph,
        trim_to_apps=app_labels,
        convert_apps=app_labels,
        migration_name=None,
    )
    write_migration_rows(changes)


def migration_progress_callback(action, migration=None, fake=False):
    pass

def migrate_migrations():
    app_label = 'meta'
    executor = MigrationExecutor(connection, migration_progress_callback)
    pre_migrate_state = executor._create_project_state(
        with_applied_migrations=True)

    targets = [key for key in executor.loader.graph.leaf_nodes(app_label)
               if key[0] == app_label]
    plan = executor.migration_plan(targets)
    if plan:
        post_migrate_state = executor.migrate(
            targets, plan=plan, state=pre_migrate_state.clone(), fake=False,
            fake_initial=False,
        )

To use Django Admin with new models, register them if they aren't already.

def reload_admin(model_cls):
    found_model = False
    for registered_model in admin.site._registry.copy():
        if registered_model.__name__ == model_cls.__name__:
            found_model = registered_model
    if found_model:
        admin.site.unregister(found_model)
        admin.site.register(model_cls)
    else:
        admin.site.register(model_cls)

URLs also need to be reloaded for Django Admin to work:

def reload_urls():
    importlib.reload(importlib.import_module(settings.ROOT_URLCONF))
    clear_url_caches()

The signals handle changes after the runserver / gunicorn or other server process has started. The models also need to be loaded when the process starts:

class MetaConfig(AppConfig):
    name = 'meta'

    def ready(self):
        import meta.signals
        meta.signals.reload_models()
        meta.signals.reload_urls()

A minimal example of this on github: https://github.com/jonatron/modelmodel

Permalink

Blog Archive