topic Re: On-prem data gateways are NOT supported by the Azure Function activity in Pipelines

On-prem data gateways are NOT supported by the Azure Function activity

tsunloan — Thu, 18 Apr 2024 07:18:17 GMT

On-premisis data gateway support for data pipelines has been announced as "shipped" however it on-prem gateways are still not supported by Azure Function activities.

The promise of delivery in Q1 2024 was not made with limits or caveats and it's perfectly sensible to assume that support would extend to a Azure services such as function apps rather than being strictly limited to the copy activity.

When will support be provided for connecting to Azure Functions in private networks via the Azure function pipeline activity?

Relevant links:
https://learn.microsoft.com/en-us/fabric/release-plan/data-factory#opdg

https://blog.fabric.microsoft.com/en-au/blog/integrating-on-premises-data-into-microsoft-fabric-using-data-pipelines-in-data-factory?ft=All

Re: On-prem data gateways are NOT supported by the Azure Function activity

Anonymous — Thu, 18 Apr 2024 07:42:46 GMT

Hi tsunloan ,

Thanks for using Fabric Community.
As I can see we can now use on-prem data source in datapipeline - How to access on-premises data sources in Data Factory - Microsoft Fabric | Microsoft Learn

We can load the data from Onprem to cloud using Copy Activity.

Can you please help me understand with your specific requirment towards on-prem? What are you trying to achieve?
I will try to provide some help on your issue?

Re: On-prem data gateways are NOT supported by the Azure Function activity

tsunloan — Thu, 18 Apr 2024 08:05:14 GMT

My comment was very specific. On-prem gateways are not supported by the Azure Function activity in Fabric pipelines. Azure functions are a valid way of retrieving and transforming data.

From the Fabric roadmap:

The roadmap promised support for on-prem data in Q1 2024 and it does not say anywhere that on-prem support will only be available in the "Copy data" activity.

Re: On-prem data gateways are NOT supported by the Azure Function activity

tsunloan — Thu, 18 Apr 2024 08:08:53 GMT

In addition, on-prem asupport is not available in the "Web" pipeline activity.
API request are another perfectly valid and common way of retrieving data.

The declaration that on-prem data is supported in fabric pipelines should be accompanied by a strong disclaimer that on-prem support is strictly limited to the "Copy data" activity.

Re: On-prem data gateways are NOT supported by the Azure Function activity

Anonymous — Thu, 18 Apr 2024 08:13:52 GMT

Hi tsunloan ,

At this time, we are reaching out to the internal team to get some help on this .
We will update you once we hear back from them.

Re: On-prem data gateways are NOT supported by the Azure Function activity

Anonymous — Mon, 22 Apr 2024 09:26:57 GMT

Hi tsunloan ,

We have an update from internal team -

" On-prem is only supported and limited towards Copy Activity but not in any other activities. "

Incase if your looking for a specific feature or functionality, I request you to please raise a feature request.

Appreciate if you could share the feedback on our feedback channel. Which would be open for the user community to upvote & comment on. This allows our product teams to effectively prioritize your request against our existing feature backlog and gives insight into the potential impact of implementing the suggested feature.

Hope this helps. Please let me know if you have any further queries.

Re: On-prem data gateways are NOT supported by the Azure Function activity

tsunloan — Mon, 22 Apr 2024 11:17:21 GMT

That's bitterly disappointing.

I didn't think I needed to raise a feature request as the roadmap stated that support for on-prem data would be delivered in Q1 2024 and it didn't state anywhere that support was limited to the Copy activity.

I would have raised a feature request last year if they were honest about what they planned on delivering.

Re: On-prem data gateways are NOT supported by the Azure Function activity

Joshrodgers123 — Mon, 22 Apr 2024 16:00:39 GMT

It's extremely dissapointing. We waited a year for OPDG support and they released it with very limited functionality....

Re: On-prem data gateways are NOT supported by the Azure Function activity

tsunloan — Mon, 22 Apr 2024 22:23:57 GMT

And the functionality that was released doesn't even work.

Re: On-prem data gateways are NOT supported by the Azure Function activity

R1k91 — Tue, 23 Apr 2024 08:46:47 GMT

I didn't test it...

maybe you can workaround it using dataflows? I mean calling the APIs from dataflows (that supports gateways) and orchestrating the dataflows call inside Pipelines?

Re: On-prem data gateways are NOT supported by the Azure Function activity

tsunloan — Tue, 23 Apr 2024 08:58:29 GMT

That's such a heavy handed approach for something that the pipeline activity should provide out-of-the-box.

It seems that nothing in Fabric actually works on it's own without elaborate workarounds or hacks.

Perhaps some of the huge marketing budget should be re-directed to engineering so they can build a working product before shilling it to corporate execs.

Re: On-prem data gateways are NOT supported by the Azure Function activity

R1k91 — Tue, 23 Apr 2024 09:18:50 GMT

I see you.

That sensation of driving in a road construction site..

Other approach I saw (especially for already working ADF projects) is to continue using ADF and self-hosted IR to load data into OneLake

How to ingest data into Fabric using the Azure Data Factory Copy activity - Microsoft Fabric | Microsoft Learn

again... it's a workaround

Re: On-prem data gateways are NOT supported by the Azure Function activity

Anonymous — Tue, 23 Apr 2024 09:25:44 GMT

Hi tsunloan ,

I apologize for the issue, can you please help me understand what functionality is not working?
If you can share it with us, we will try to guide you.

Re: On-prem data gateways are NOT supported by the Azure Function activity

Anonymous — Thu, 25 Apr 2024 07:33:54 GMT

Hi tsunloan ,

We haven’t heard from you on the last response and was just checking back to see if you got a chance to look into my last response.
Can you please share the issue you are facing, we will try to help .

Thanks

Re: On-prem data gateways are NOT supported by the Azure Function activity

Anonymous — Fri, 26 Apr 2024 10:21:57 GMT