Fabric Networking - Data Integration

[royclack]

Hi,

A while years ago while using the Power BI Data Gateway we'd envisaged potentially using Azure Data Factory and a Self-Hosted integration run-times for various reasons and since time has moved on and we do not use Azure Data Factory.

We still do have a legacy integration tool-kit based on VMs which we use to integrate source data with preperation and validation of ingested data, and we replicate this data into Fabric. For the replication we do use  Power BI data gateways and we do use Fabric Data Factory but are now looking at more varied source types and will be re-working our architecture if/when required to by-pass the legacy solution. 

In consideration of data source integration across different workloads does anyone have any recommendations to integrate data sources considering using item types for notebooks, data factory and data flows. The primary use cases would be to connect to back-end RDBMS or HTTPS external end-points. We do this internally with Data Factory COPY activities to VM for data replication via Power BI data gateway connections.

Our non-Fabric legacy solution which does not use the Power BI gateway has network routing from Azure to an on-premise network to be able to route network traffic so that an externally visible IP address could be white-listed when engaging with 3rd parties. We'd likely have to something similar from a networking perspective. Does anyone do similar with Fabric and/or have any recommendations?

https://learn.microsoft.com/en-us/fabric/security/security-managed-vnets-fabric-overview

Solved: Networking options Fabric - Microsoft Fabric Community

Azure Data Factory — Self Hosted Integration Runtime Overview, Usage Considerations and Best Practic...

[spencer_sa · Accepted solution]

We have a legacy 'on prem' RDBMS system that requires either a Gateway or a SHIR to connect (we're going with the former in the first instance).
We have a 'push' method from the RDBMS system that connects to Fabric/OneLake via API, uploads a parquet file, and writes/executes a notebook to convert to a Delta table, but will actually need a 'pull' form Fabric for our final state.

[v-venuppu]

Hi @royclack ,

I hope the information provided is helpful.I wanted to check whether you were able to resolve the issue with the provided solutions.Please let us know if you need any further assistance.

Thank you.

[v-venuppu]

Hi @royclack ,

May I ask if you have resolved this issue? Please let us know if you have any further issues, we are happy to help.

Thank you.

[v-venuppu]

Hi @royclack ,

I wanted to check if you had the opportunity to review the information provided and resolve the issue..?Please let us know if you need any further assistance.We are happy to help.

Thank you.

[v-venuppu · Accepted solution]

Hi @royclack ,

Thank you for reaching out to Microsoft Fabric Community.

Thank you @spencer_sa for the prompt response.

Use Fabric Data Factory to replicate data from RDBMS or HTTPS endpoints, then apply Dataflows or Notebooks for transformations. For secure network access, deploy a Managed VNET and self-hosted integration runtime (SHIR) to reach on-prem or private endpoints and maintain a known outbound IP for whitelisting. This modernizes your legacy VM setup while keeping security intact.

Here are some Microsoft Documentations for your reference:

Overview of managed virtual networks in Microsoft Fabric - Microsoft Fabric | Microsoft Learn

Create a self-hosted integration runtime - Azure Data Factory & Azure Synapse | Microsoft Learn

[spencer_sa · Accepted solution]

We have a legacy 'on prem' RDBMS system that requires either a Gateway or a SHIR to connect (we're going with the former in the first instance).
We have a 'push' method from the RDBMS system that connects to Fabric/OneLake via API, uploads a parquet file, and writes/executes a notebook to convert to a Delta table, but will actually need a 'pull' form Fabric for our final state.