Real-time Data Integration Options in Fabric: Salesforc , Custom API, and SharePoint as Data sources

justmehere — Wed, 30 Oct 2024 17:52:03 GMT

I need guidance on implementing real-time data integration in Microsoft Fabric for the following scenarios:

Automatically ingest data from my Salesforce data source whenever records change
Real-time data ingestion from a custom API or when data from my custom API changes
Trigger pipeline execution when files are added to a SharePoint folder

While I'm aware of scheduled pipeline runs, I need real-time/event-driven solutions.
- What are the recommended approaches in Fabric for these real-time integration requirements?
- Does Fabric support webhook triggers or change data capture for Salesforce?
- How can I set up SharePoint triggers for pipeline execution?
- Would Azure Data Factory be more suitable for these event-driven integration needs?
Any guidance on the best architecture approach would be appreciated.

Re: Real-time Data Integration Options in Fabric: Salesforc , Custom API, and SharePoint as Data sources

Anonymous — Thu, 31 Oct 2024 02:09:48 GMT

Here are some of my thoughts on the issues you raised:

1. What are the recommended approaches in Fabric for these real-time integration requirements?

(1) Eventstream: Ingests data from various sources, including change data feeds from operational databases.
(2) Real-Time Hub: Centralizes streaming and discrete event data, making it easily discoverable and actionable.
(3) Data Activator: Allows for setting up conditions to trigger notifications or processes based on real-time data.

You can look at this two documents:

New Real-Time Intelligence in Microsoft Fabric | Event-based actions and insights

Real-Time Intelligence in Microsoft Fabric documentation - Microsoft Fabric | Microsoft Learn

2. Does Fabric support webhook triggers or change data capture for Salesforce?

(1) Webhooks: Salesforce can send HTTP requests to a specified URL when certain events occur, triggering processes in Microsoft Fabric.
(2) Change Data Capture (CDC): Salesforce CDC mechanisms like Change Events or Platform Events can be streamed to Azure Event Hubs, triggering workflows in Fabric.

3. How can I set up SharePoint triggers for pipeline execution?

To set up SharePoint triggers for pipeline execution, you can use Azure Data Factory or Azure Synapse Analytics.
In the Azure Data Factory or Synapse UI, go to the Integrate tab, select Trigger, and then New/Edit.Select the type of trigger.Specify the conditions under which the trigger should fire, such as file creation or modification in a SharePoint library.

You can look at this document: Pipeline execution and triggers - Azure Data Factory & Azure Synapse | Microsoft Learn

4. Would Azure Data Factory be more suitable for these event-driven integration needs?

(1) Event-Based Triggers: ADF can trigger pipelines in response to events such as file creation or deletion in Azure Storage, leveraging Azure Event Grid.
(2) Custom Event Triggers: ADF can subscribe to custom topics in Event Grid, allowing for fine-grained control over which events trigger pipeline executions.

Here are the documents you can read:

Create event-based triggers - Azure Data Factory & Azure Synapse Analytics | Microsoft Learn

Event Driven Architecture with Custom Event Trigger and Advanced Filtering - Microsoft Community Hub

Best Regards

Yilong Zhou

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

topic Re: Real-time Data Integration Options in Fabric: Salesforc , Custom API, and SharePoint as Data sources in Pipelines

Real-time Data Integration Options in Fabric: Salesforc , Custom API, and SharePoint as Data sources

Re: Real-time Data Integration Options in Fabric: Salesforc , Custom API, and SharePoint as Data sources