I'm reading Prefect documentation and trying to understand how local deployment works. I can deploy a flow locally following the below steps.

First, I build the flow:

prefect deployment build ./log_flow.py:log_flow -n log-simple -q test

Where ./log_flow.py:log_flow are, respectively, the flow's location and entrypoint. log-simple is the name of deployment and test is the work queue

Second, I start the worker using:

prefect agent start -q 'test'

To apply the deployment, I use python running the below snippet:

from log_flow import log_flow
from prefect.deployments import Deployment
deployment = Deployment.build_from_flow( flow=log_flow, name="log-simple", parameters={"name": "Marvin"}, infra_overrides={"env": {"PREFECT_LOGGING_LEVEL": "DEBUG"}}, work_queue_name="test",
)
if __name__ == "__main__": deployment.apply()

Well, that works fine for a single flow. But how can I deploy several flows at once? I can repeat the above process for every flow, but it looks a bit unpractical to me since each build step generates another YAML file. I think would be more practical if my deployment generates a single YAML file for all flows.

Is there a way to deploy several flows at once in Prefect 2.0?

2 Answers

Yes, absolutely. You could loop over all flows from Python, bash or from CI/CD. Here are a couple of examples showing how you could approach it:

• CI/CD:
• bash script (there are several example bash scripts in this repo for various infra/storage options):

You can create a list of deployments and then, use method aply to each one of them. Something like this:

# deployment.py
from my_project.flows import my_flow, my_flow2
from prefect.deployments import Deployment
DEPLOYMENTS = [
Deployment.build_from_flow( flow=my_flow, name='deploy_1' ),
Deployment.build_from_flow( flow=my_flow2, name='deploy_2' )
...
]
if __name__ == '__main__': for deployment in DEPLOYMENTS: deployment.apply()

Then You can simply run this python script

Your Answer

Sign up or log in

Sign up using Google Sign up using Facebook Sign up using Email and Password Submit

Post as a guest

Post Your Answer Discard

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.