How to Ensure Singleton Execution of ADF Pipeline

Many times we don’t think about it but in order to ensure the integrity and stability of our systems and data we pay attention to the fact that at any point of time there should be ONLY one instance of the pipeline running.

This can be done quite easily by setting up the concurrency property of the pipeline as shown below –

You can just set it to 1 and it will ensure that no other request to run the pipeline gets executed if the pipeline is already running. But this has a catch that it doesn’t completely ignore the incoming request to execute the pipeline while this is still underway. Instead, it pushes that request in the queue so as soon as the current execution of pipeline gets finished, it will start running it again.

If you want to completely ignore the request, you may have to go into some programming mode to control this in a more flexible way. There is no such option available on the GUI. However, this concurrency property ensures that integrity and stability of your system and data. We will explore the flexibility to control this via some programming in a future post.