Safely dropping a column from production using the Django ORM

Problem

If you remove a field from a Django model and let Django generate the migration using python manage.py makemigrations, you most likely end up with a migration like this:

from django.db import migrations

class Migration(migrations.Migration):

    dependencies = [
        ('myapp', '0001_initial'),
    ]

    operations = [
        migrations.RemoveField(
            model_name='mymodel',
            name='myfield',
        ),
    ]

If we look at the resulting SQL, we see that the column is blindly dropped in one, atomic operation:

ALTER TABLE "myapp_mymodel" DROP COLUMN "myfield" CASCADE;

This can be a very dangerous operation for a production website. Typically, migrations are ran prior to releasing a new version of the code that stops using the code. As a result, when the migrations is applied there is code still querying the deleted field. This will result in failing queries until the new code has been released. This can happen even if you don’t explicitly use the field. Take this statement as an example:

MyModel.objects.all()

This generates a SQL statement such as:

SELECT id, somefield, myfield FROM myapp_mymodel;

As we can see, Django includes the column that we are deleting in the SELECT query. Django is including it because the code that is running in production is not aware that the field has been deleted. Any query to this table/model that doesn’t explicitly select fields will start failing after the migration applies until the new version of the code has been rolled out.

Depending on how long it takes for the new code to roll out, this can mean a non-sigificant amount of time in which there are failing queries. If the queries are in a view, it can mean those views are unavailable until the new version of the code rolls out.

Fix

To avoid this problem, we have to split the removal of the field in two roll-outs/releases:

1. Remove the field from the code base, but don’t drop the colunm.

   Remove the field from the model and modify Django’s auto-generated migration to only soft-delete the field and not delete the column from the underlying database:

   from django.db import migrations
   
   class Migration(migrations.Migration):
   
       dependencies = [
           ('myapp', '0001_initial'),
       ]
   
       operations = [
           migrations.SeparateDatabaseAndState(
               database_operations=[],
               state_operations=[
                   migrations.RemoveField(
                       model_name='mymodel',
                       name='myfield',
                   ),
               ],
           )
       ]

   This migration only executes the RemoveField operation on Django’s migration state. From now on, the field no longer exists as far as Django’s concerned. Since no database_operations have been passed, the column will not actually be deleted from the database.

   We can confirm it doesn’t actually delete the column by checking the SQL queries the migration generates:

   $ python manage.py sqlmigrate myapp 0002_drop_field
   BEGIN;
   COMMIT;

   We can also confirm that Django thinks the field+column has been deleted by running the migration detector:

   $ python manage.py makemigrations --dry
   No changes detected

2. Drop the column from the database.

   After the changes from step #1 have been deployed and the migration applied, the running code is no longer including the deleted column in any queries. We can now create a second migration that actually drops the column from the database:

   from django.db import migrations
   
   class Migration(migrations.Migration):
   
       dependencies = [
           ('myapp', '0002_drop_field'),
       ]
   
       operations = [
           migrations.RunPython(
               lambda (apps, schema_editor): schema_editor.remove_field('mymode', models.TextField(name='myfield')),
               lambda (apps, schema_editor): schema_editor.add_field('mymode', models.TextField(name='myfield')),
           )
       ]