I am currently running several Airflow DAGs using LocalExecutor and it's been working fine. My server has plenty of resources. I am about to add a new DAG for a larger project and I am considering switching from LocalExecutor to CeleryExecutor.

My question is, what are some signs that I should switch to CeleryExecutor? Are there specific performance metrics I should be looking at to know when I need to start scaling out?

1 Answer

The LocalExecutor should be sufficient if you want to run a small number of concurrent tasks such that with your scheduling needs those tasks could run a single server. You should think of moving to CeleryExecutor when you feel that you need to distribute the jobs across multiple servers.

Your Answer

Sign up or log in

Sign up using Google Sign up using Facebook Sign up using Email and Password Submit

Post as a guest

Post Your Answer Discard

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy