Pipeline monitoring & sending email based notification using ADF and Logic apps

Logic apps are powerful services that are meant to provide useful APIs which are used for handling many popular use cases.

Consider an use case when complex Data factory pipelines are running with multiple dependencies. To monitor ADF pipeline proactively and send timely notification to the monitoring team, integration with logic apps can be very useful.

Pipeline monitoring using ADF & Logic apps

In the logic app section, choose when HTTP request is received and add a send mail(v2) task to it. If you carefully look into this image:

1 →It is a POST url which logic apps generates. This url is used in the web activity of the ADF. It's kind of POST API method in which all the parameters generated from the data factory are posted in this url which in turn triggers a mail.

2 →This is a json payload, consisting of schema that defines the pipeline,error name, datafactory and other necessary details that will come up through ADF.

3 →Once all the attributes are parameterized, results of the ADF are stored in this json payload. Next, a mail body along with subject is created with the source mail,i.e sender.

4 →To enable a source mail id, simply login from change connection and authorize yourself with any valid email from which you would like to send mail to the monitoring team.

{
    "properties": {
        "DataFactoryName": {
            "type": "string"
        },
        "EmailTo": {
            "type": "string"
        },
        "ErrorMessage": {
            "type": "string"
        },
        "PipelineName": {
            "type": "string"
        },
        "Subject": {
            "type": "string"
        }
}

Logic app send email(outlook)

So if you see the web activity settings, you can clearly see that the url that was generated by the logic apps is used here. The only thing that needs to be taken care is the proper way to parameterize the json elements in the body.

Here is a sample in which you can easily add the elements by passing the pipeline parameters.

{“DataFactoryName”:”@{pipeline().DataFactory}”,
“PipelineName”:”@{pipeline().Pipeline}”,
“Subject”:”Pipeline Processed Successfully!”,
“ErrorMessage”:””,
“EmailTo”:”sanajit@email.com”}

ADF web activity Body

Once everything is set, you can expect a mail in your outlook regarding the status of the pipeline that was under processing.

Mail Alert