Getting on premises data into Fabric

[joakimfenno]

I am wondering what the differemt alternatives to securely enable on premises data sources to Microsoft Fabric are.

I have used On Premise Datagateway which works for ingesting data using data pipelines. However for some of my sources I really need to use code-based ingestion (preferably notebooks).

Is it possbile to use/acess on premise datagateways from notebooks?

Are there alternatives to data gateways for enabling on premise data to Fabric?

[jwinchell40]

@joakimfenno - The gateway itself supports a variety of connections, including API's.  Have you tried setting up a Copy Activity that calls an API that runs On-Premise (assuming it is URL addressable) and can be added to the Gateway as a connection?

You could also look at:

- Using Azure Application Gateway (may be rebranded since I last used it) to broker the connection between an Azure Function or App Service and the On-Prem API.  Then you can call the AF from the Notebook.

- You could also look at setting up a front-end server (proxy) that has a public IP/URL that you could call from a Notebook

- Setup a Private Endpoint between an Azure Environment and the On-Prem resource, etc

I know that both of the above options require additional spend in Azure and additional management overhead.  Just throwing out options depending on how urgent the requirement is.

Jeremy

[joakimfenno]

Yes, I have tried data pipeline - copy data.

It works but it comes with a lot of overhead

I would prefer to use notebook

Azure Application Gateway/azure functions could be an option even though I would need to recode my ingestion - do you have any reference material?

we are not allowed to expose sources with public IP/URL

privcate endpoint might be an option, is it supported with fabric - do you have any reference material?

[NEJO]

The use of a notebook to ingets data would be very useful, certainly for connecting to API's, Odata, etc.  Not many votes at the moment - but you can vote for this here:  Microsoft Idea 

There are ways from I've read to force the connection within the notebook, but due to security restriction - we're all locked down and using the gateway is the only way. 

Get enough votes and Microsoft may look at it.

[Anonymous]

Hi @joakimfenno ,

Thanks for using Fabric Community.
Unfortunately it is not possible to use/acess on premise datagateways from notebooks. At present we can use on premise data gateway only with datapipelines and dataflow gen 2.

As an alternative, you can load your data into lakehouse from onprem using pipeline and you can perform transformation for data present in lakehouse using Notebook after loading successfully.

Docs to refer - Integrating On-Premises Data into Microsoft Fabric Using Data Pipelines in Data Factory | Microsoft ...

However, your suggestion is definitely valuable! We use customer feedback like yours to prioritize future features. The more users who request the ability to customize backgrounds, the higher it moves on our list.

Appreciate if you could share the feedback on our Microsoft Fabric Ideas. Which would be open for the user community to upvote & comment on. This allows our product teams to effectively prioritize your request against our existing feature backlog and gives insight into the potential impact of implementing the suggested feature.

I hope this information helps. If you have any further queries please do let us know.

[joakimfenno]

Thanks for getting back.

Apart from data gateways - are there other options for enabling on premise data sources (in my case web API, postgres db, kafka server etc.) for fabric or for azure in general. Is it possible to open up a on prem service for fabric in a secure way?

[Anonymous]

Hi @joakimfenno ,

Please refer below links - 
Loading on-prem DB to Fabric Lakehouse with Synapse Spark (youtube.com)
Microsoft Fabric: Load on-premise data file to Lakehouse using on-premise Python using OAuth Token (...

Hope this is helpful. Please do let me know incase of further queries.

[Anonymous]

Hi @joakimfenno ,

We haven’t heard from you on the last response and was just checking back to see if your query was answered.
Otherwise, will respond back with the more details and we will try to help .

Thanks

[joakimfenno]

I will discuss it with the team next week, thanks