How to auto-reload Celery worker on Code Change

Please check my course: The Definitive Guide to Celery and Django on testdriven.io

Introduction

This is #2 of my Django Celery Tutorial Series

Many newbie developers does not know Celery would not auto-reload worker like Django dev server so this might cause some weird problem when then learn Celery.

In this Django Celery tutorial, I would talk about how to auto-reload Celery worker on code change.

Here I will provide you two solutions to solve this problem.

Solution 1 (WatchDog)

We can use WatchDog to help us monitor the code files and rerun the command if necessary.

First, let's install the package, please note that PyYAML and argh also need to be installed as well.

$ pip install watchdog

Let's assume you run your Celery worker using command below

$ celery worker -A django_celery_example --loglevel=info

Now you can run the command in this way

$ watchmedo auto-restart -d django_celery_example/ -p '*.py' -- celery worker -A django_celery_example --loglevel=info

1. -d django_celery_example told watchmedo to watch files under django_celery_example directory

2. -p '*.py' told watchmedo only watch py files (so if you change js or scss files, the worker would not restart)

Another thing I want to say here is that if you press Ctrl + C twice to terminate above command, sometimes the Celery worker child process would not be closed, this might cause some weird problem in some cases.

So I prefer method below. Let's keep reading how it works.

Solution 2 (Django Autoreload)

As you know, if you change files in Djnago project, the dev server would restart because Django supports autoreload.

So here I would show you how to use Django autoreload code to help us solve this problem.

First, let's create a Django command

.
├── db.sqlite3
├── django_celery_example
│   ├── __init__.py
│   ├── celery.py
│   ├── management
│   │   ├── __init__.py
│   │   └── commands
│   │       ├── __init__.py
│   │       └── celery_worker.py
│   ├── settings.py
│   ├── urls.py
│   └── wsgi.py
├── manage.py
├── polls
│   ├── __init__.py
│   ├── admin.py
│   ├── apps.py
│   ├── migrations
│   │   └── __init__.py
│   ├── models.py
│   ├── tests.py
│   └── views.py
└── requirements.txt

As you can see, I created a Django command celery_worker

import shlex
import subprocess

from django.core.management.base import BaseCommand
from django.utils import autoreload


def restart_celery():
    cmd = 'pkill -f "celery worker"'
    subprocess.call(shlex.split(cmd))
    cmd = 'celery worker -A django_celery_example --loglevel=info'
    subprocess.call(shlex.split(cmd))


class Command(BaseCommand):

    def handle(self, *args, **options):
        print('Starting celery worker with autoreload...')

        # For Django>=2.2
        autoreload.run_with_reloader(restart_celery)

        # For django<2.1
        # autoreload.main(restart_celery)

Here we use Django autlreload function to run Celery worker for us, which is very clean and simple.

Conslution

In this Django Celery tutorial, I provide you 2 ways to solve the Celery worker autoreload problem.

Please check my course: The Definitive Guide to Celery and Django on testdriven.io