Celery Vs Aws lambda for scheduled task
1 min readJun 21, 2024
I have researched throughly in the internet as I required a suggestion to choose between aws lambda to run the task or use celery to perform the task. It is quite amazing as for my use case lambda was a better choice.
If you are interested lets dive in. ::
Lets dive in only if are interested otherwise you can scroll
1) Amazon lambda service:
>> serverless compute service that runs code in response to events and automatically manages compute resources
Pros:
- no need to handle servers
- automatic scaling with respect to demand
- pay as you go
- aws friendly , event driven
Cons:
- slow latency for first task
- 15 minutes max exec time
use cases :
- real time stock price updates
- process tak, small, quick like data transformation, notification triggers.
2) Celery
open source , distributed task queue system that allows to run async tasks and manage task scheduling.
Pros:
- flexible [works with message brokers likes rabbitMQ, Redis, ]
- task with longer running execution time
- full control over task scheduling, retries
Cons:
- Require managing and scaling of infrastructure
- more complex setup
Suitable for:
- periodic data fetch, update
- complex workflows requiring task chaining and coordination