Virtually every part of modern life generates data, from using a credit card to shop for groceries to driving a car to work. Because of this huge volume of data, there needs to be a way to not only store the information but orchestrate and operationalize it into actionable business insights. Azure Data Factory is a managed cloud service that does just this. Using Azure DevOps, we can implement continuous deployment practices to automate Azure Data Factory deployment.
Getting Started
Before integrating Azure Data Factory with Azure DevOps for your automatic deployment, you must first ensure all prerequisites are met. First, you will need an Azure subscription linked to Azure DevOps Server or Azure Repos that uses the Azure Resource Manager (ARM) service endpoint. Next, you will need a data factory configured with Azure Repos Git integration. You will also need an Azure key vault that contains the secrets for each environment.
Azure Data Factory Integration with Azure Pipelines
With all prerequisites met, you’re now ready to set up your Azure Pipelines release. In Azure DevOps, open the project that holds your data factory. Then, open the tab for releases and select the option to create a new release pipeline. For this pipeline, you’ll choose the Empty job template. With the pipeline created, you’re going to modify it by adding an artifact. Here, that artifact is the Git repository configured with your data factory. Next, you’re going to add an ARM deployment task and configure it for this job.
If you have secrets to pass in an ARM template, it is recommended to use Azure Key Vault in your release. To do this, simply add an Azure Key Vault task before the ARM task in your pipeline. It is also recommended to keep separate key vaults for each environment.
Automating Deployment
Your release pipeline is complete, but now we want to make sure your deployment is automated. This can be done using release triggers. When the trigger conditions are met, the pipeline will automatically deploy your artifacts to the environment specified. To allow your release to move from environment to environment in your release pipeline, you’ll want to set up stage triggers. To set this up, click the lightning icon on your environments and set up your pre-deployment conditions, including what condition(s) specifically you want the trigger to be.
You now have an automated Azure Data Factory deployment! For more information, or to get started, contact our team of experts here at PRAKTIK.