Asynchronous Tasks With Django and Celery

Asynchronous Tasks With Django and Celery

Django Celery Architecture 

When i was working on projects in  Django ,  one of the most frustrating thing I faced was need to run a bit of code periodically, i wrote my own function is for sending newsletter on the Monday morning 10 am , this time i faced lots of problem because some times my function did not working properly don't mean's syntactically , so i may think to start where i done wrong , is right no ? then how the problem is occurring , after that finally i found a solution for do some task periodically we can use Celery.

What is Celery ?

“Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well.” For this post, we will focus on the scheduling feature to periodically run a job/task.

Why is this useful?

• Think of all the times you have had to run a certain task in the future. Perhaps you needed to access an API every hour. Or maybe you needed to send a batch of emails at the end of the day. Large or small, Celery makes scheduling such periodic tasks easy.
• You never want end users to have to wait unnecessarily for pages to load or actions to complete. If a long process is part of your application’s workflow, you can use Celery to execute that process in the background, as resources become available, so that your application can continue to respond to client requests. This keeps the task out of the application’s context.

What you need ?

Celery requires a message transport to send and receive messages. The RabbitMQ and Redis broker transports are feature complete, but there’s also support for a myriad of other experimental solutions, including using SQLite for local development.

Celery can run on a single machine, on multiple machines, or even across data centers.

First Steps with Celery

Celery is a task queue with batteries included. It is easy to use so that you can get started without learning the full complexities of the problem it solves. It is designed around best practices so that your product can scale and integrate with other languages, and it comes with the tools and support you need to run such a system in production.

In this blog you will learn the absolute basics of using Celery. You will learn about;

• Choosing and installing a message transport (broker).
• Installing Celery and creating your first task.
• Starting the worker and calling tasks.
• Keeping track of tasks as they transition through different states, and inspecting return values.

     Choosing a Broker

         Celery requires a solution to send and receive messages; usually this comes in the form of          a separate service called a message broker.

         There are several choices available, including: (please search google for more details )

                  RabbitMQ
                  Redis
                  Using a database
                  Other brokers

 Installing Celery

 Celery is on the Python Package Index (PyPI), so it can be installed with standard Python tools like  pip or easy_install:

$ pip install celery

Example :

Let’s create the file tasks.py:

from celery import Celery

app = Celery('tasks', broker='amqp://guest@localhost//')

@app.task
def add(x, y):
    return x + y

Example Project :

Clone my project  url : git cloe https://github.com/renjithsraj/photogallery.git

Project Description :

This Project mainly looking for basic understand about the periodic task ( scheduling task ) in django. The scope of the project is collect the images from the flickr latest images and store in to data base in every two minute (change with your own time ) make it a gallery.
heroku url : https://flickercollection.herokuapp.com/

Configuration:

step 1 : Create Virtualenv using (virtualenv env)

step 2 : open terminal activate the env

step 3 : clone the project command ( git clone https://github.com/renjithsraj/photogallery.git )

step 4 : make the path to project ( cd photogallery in linux )

step 5 : install packages which we required for this project ( pip install -r requrements.txt )

step 6 : install broker here im used redis ( if install redis server also in your local machine )

step 7 : open new terminal start the redis server ( redis-server command )

step 8 : Running Locally
    Ready to run this thing?

    With your Django App and Redis running, open two new terminal windows/tabs. In each new window, navigate to your project directory, activate your virtualenv, and then run the following commands (one in each window):

    $ celery -A pincha worker -l info
    $ celery -A pincha beat -l info

    When you visit the site on http://127.0.0.1:8000/ you should now see one image. Our app gets one image from Flickr every 2 minutes: ( here i just take images freequent intervel please make your intervel )

Help !!

Please feel free to contact me : renjithsraj@live.com