Celery

An asynchronous task queue

(not only) for Django
Markus Zapke-Gründemann
DjangoCon Europe 2011

Image (cc-by-sa): http://www.flickr.com/photos/artdrauglis/3747272595/

 Overview

• Why should I use a task queue?

• Celery
• Python Task
• Django Task
• Webhooks
Why should I use
a task queue?
Users have to wait…
Action Task

Add a comment Check for spam

Upload an image Create thumbnails

Process payment,
Order a product
send emails
Mark a favorite Update database
A task queue can help!

• Decoupling of information producers and

consumers
• Profit from asynchronous processing
• Improve scalability
• Replace cronjobs
Client Broker Worker
Client Worker

Client Broker

Client Worker
Client Worker

2
Client Broker

Client 1 Worker
Client Worker
3
4
2
Client Broker

Client 1 Worker
 Database
Filesystem
API

3
Client Worker
4
5
2
Client Broker

Client 1 Worker
Celery
 Celery
• Author: Ask Solem Hoel
• Written in Python
• Synchronous, asynchronous and scheduled
tasks
• Broker: RabbitMQ, NoSQL and Ghetto
Queue
 RabbitMQ

• Message Broker
• Written in Erlang
• AMQP (Advanced Message Queuing
Protocol)
• Clustering support
 Celery
Features
• Serialization (pickle, JSON,YAML or custom
code)
• Task sets and subtasks
• Retry failed tasks
• Routing (only AMQP brokers)
 Celery
Features
• Different result stores
• Easy logging from tasks
• Webhooks
• Integration with Python Web Frameworks
(Django, Flask, Pylons)
 Celery
History

• Until 1.0.6 the Django ORM was used

• Since 2.0 SQLAlchemy is used and Django
integration is provided by the „django-
celery“ package
 Install Celery

$ pip install celery

 Behind the scenes

• kombu - An AMQP messaging framework

for Python
• amqplib - Python AMQP client.
• anyjson - A uniform JSON API
 Setting up RabbitMQ

$ rabbitmqctl add_user myuser mypassword

$ rabbitmqctl add_vhost myvhost

$ rabbitmqctl set_permissions -p myvhost myuser ".*" ".*" ".*"

Python Task
 Python Task
# tasks.py
from celery.task import task

@task
def add(x, y):
return x + y
 Python Task
Configuration
# celeryconfig.py
BROKER_HOST = "localhost"
BROKER_PORT = 5672
BROKER_USER = "myuser"
BROKER_PASSWORD = "mypassword"
BROKER_VHOST = "myvhost"
CELERY_RESULT_BACKEND = "amqp"
CELERY_IMPORTS = ("tasks", )
Python Task Execution
>>> from tasks import add
>>> result = add.delay(1, 2)
>>> result.ready()
False

$ celeryd --loglevel=INFO

>>> result.get()
3
>>> result.result
3
>>> result.successful()
True
Django Task
 Install Celery
for Django

$ pip install django-celery

 Django Task

Client Broker Worker

 Django Task
blog.views.add_comment

Client Broker Worker

 Django Task
blog.views.add_comment blog.tasks.spam_filter

Client Broker Worker

 Django Task
blog.views.add_comment blog.tasks.spam_filter

Client Broker Worker

Comment
Model
 Django Task
blog.views.add_comment blog.tasks.spam_filter

Client Broker Worker

comment.save()
tasks.spam_filter.delay(comment.id,
remote_addr)

Comment
Model
 Django Task
blog.views.add_comment blog.tasks.spam_filter

Client Broker Worker

comment.save() comment.is_spam = True

tasks.spam_filter.delay(comment.id, comment.save()
remote_addr)

Comment
Model
 Django Task
# blog/tasks.py
@task
def spam_filter(comment_id, remote_addr=None):
logger = spam_filter.get_logger()
logger.info("Running spam filter for comment %s" % comment_id)

comment = Comment.objects.get(pk=comment_id)
current_domain = Site.objects.get_current().domain
akismet = Akismet(settings.AKISMET_KEY, "http://%s" % current_domain)
if not akismet.verify_key():
raise ImproperlyConfigured("Invalid AKISMET_KEY")

is_spam = akismet.comment_check(user_ip=remote_addr,
comment_content=comment.comment,
comment_author=comment.name,
comment_author_email=comment.email_address)
if is_spam:
comment.is_spam = True
comment.save()

return is_spam
 Django Task
# blog/tasks.py
@task
def spam_filter(comment_id, remote_addr=None):
logger = spam_filter.get_logger()
logger.info("Running spam filter for comment %s" % comment_id)

comment = Comment.objects.get(pk=comment_id)
current_domain = Site.objects.get_current().domain
akismet = Akismet(settings.AKISMET_KEY, "http://%s" % current_domain)
if not akismet.verify_key():
raise ImproperlyConfigured("Invalid AKISMET_KEY")

is_spam = akismet.comment_check(user_ip=remote_addr,
comment_content=comment.comment,
comment_author=comment.name,
comment_author_email=comment.email_address)
if is_spam:
comment.is_spam = True
comment.save()

return is_spam
 Django Task
Configuration
# settings.py
INSTALLED_APPS += ("djcelery", )

import djcelery
djcelery.setup_loader()

BROKER_HOST = "localhost"
BROKER_PORT = 5672
BROKER_USER = "myuser"
BROKER_PASSWORD = "mypassword"
BROKER_VHOST = "myvhost"

$ python manage.py syncdb

$ python manage.py celeryd -l info
Webhooks
 Webhooks
# POST
>>> from celery.task.http import URL
>>> res = URL("http://example.com/multiply").get_async(x=10, y=10)
>>> res.get() # {"status": "success", "retval": 100}
100

# GET
>>> from celery.task.http import HttpDispatchTask
>>> url = "http://example.com/multiply"
>>> res = HttpDispatchTask.delay(url, method="GET", x=10, y=10)
>>> res.get() # {"status": "success", "retval": 100}
100
 Links
• http://celeryproject.org/
• http://www.rabbitmq.com/
• http://github.com/ask
• http://pypi.python.org/pypi/celery
• http://pypi.python.org/pypi/django-celery
• http://pypi.python.org/pypi/celerymon
 Thanks!
@keimlink
http://www.keimlink.de
 License
This work is licensed under the Creative Commons
Attribution-ShareAlike 3.0 Unported License. To view a
copy of this license, visit http://creativecommons.org/
licenses/by-sa/3.0/ or send a letter to Creative Commons,
444 Castro Street, Suite 900, Mountain View, California,
94041, USA.