Data Factory Data pipelines in Microsoft Fabric have rich support for building complex workflows and orchestration for data activities. With the latest feature announcements, we've taken things a step further based on our community’s feedback.
New enhancements to pipeline activities:
- Invoke Remote pipeline.
- Spark Notebook environment parameters.
Invoke Remote Pipeline Activity
We’ve been working diligently to make the very popular Data pipeline activity known as “Invoke Pipeline” better and more powerful. Based on customer feedback, we continue to iterate on the possibilities and have now added the exciting ability to call pipelines from Azure Data Factory (ADF) or Synapse Analytics pipelines as a public preview!
This creates countless possibilities to utilize your existing ADF or Synapse pipelines inside of a Fabric pipeline by calling it inline through this new Invoke Pipeline activity. Use cases that include calling Mapping Data Flows or SSIS pipelines from your Fabric Data pipeline will now be possible as well.
We will continue to support and include the previous Invoke Pipeline activity as “legacy”, without the support for ADF or Synapse remote pipeline invocation and without child pipeline monitoring in Fabric. But for the latest features like remote invocation and child pipeline monitoring, you can use the new Invoke Pipeline.
Spark Notebook Environment Parameters
One of the most popular use cases in Fabric Data Factory today is automating and orchestrating Fabric Spark Notebook executions from your Data pipelines. A common request has been to reuse existing Spark sessions to avoid any session cold-start delays. We’ve delivered on that requirement by enabling “Session tags” as an optional parameter under “Advanced settings” in the Fabric Spark Notebook activity! Now you can tag your Spark session and reuse the existing session using that same tag to reuse an existing session and greatly reduce the overall processing time of your Data pipelines.
Let us know your feedback!