In the previous post, we saw the basics of catching errors generated by activities during the flow and here we will see all the complexities involved in handling the same for execute pipeline activity. The first step of handling or say catching errors during a pipeline’s execution remains the same that you need to add a “Failure” path to the activity –
However, in order to catch the errors that it might have encountered, you need to remember that execute pipeline activity runs in its own context and generates a new pipeline run ID so you can not simply refer to its output to catch the error details. If you see the output of Execute Pipeline activity, in general, you will find the following two details only –
In order to get the details of underlying pipeline’s execution, you will have to make use of REST APIs. So, let us see how –
To make a call to REST API, you need to use web activity with GET method as we want to fetch the information about pipeline’s execution. The URL for API needs to be in the following format –
https://management.azure.com/subscriptions/<subscription id>/resourceGroups/<resource group name>/providers/Microsoft.DataFactory/factories/<data factory name>/pipelineruns/<pipeline run id>?api-version=2018-06-01
Subscription ID, Resource Group Name and Data Factory Name can be fairly easy to enter there. However, you will have to catch the pipeline run id from the output of Execute Pipeline Activity’s output as shown in the previous snapshot too. So, you can construct the API URL using concat as –
You can parameterize the subscription id, resource group name and data factory name too and get them concatenated in the concat function.
As a next step, you need to get authenticated yourself in order to pull the details from API execution as it’s not an open API. So, configure that web activity to use MSI as the authentication method and specify the resource as highlighted in the below snapshot –
Until now, you have just specified that you would like yourself to be authenticated using Managed identity (MSI) but haven’t provided any access to your ADF instance to get authenticated in order to retrieve the pipeline execution information. So, as a next step, you need to assign “Contributor” role to your ADF managed identity at the subscription level. To do so, you need to navigate to Identity and Access Management (IAM) at the subscription level and add new role assignment as (where you need to search for your data factory name) –
If you don’t do so, you will receive the following authorization error –
Once configured and authenticated successfully, you can read the error message generated by the Execute Pipeline Activity as –
activity(‘Web Activity’).output.message
So, this is how you can read the errors generated by Execute Pipeline Activity and send them over the emails or other communication channels for further actions.
One comment