topic Re: fabric data factory pipeline access on-premise data source in Pipelines

fabric data factory pipeline access on-premise data source

Anonymous — Wed, 31 Jan 2024 01:01:08 GMT

if i am not mistaken, the pipeline in fabric data factory cannot access on-premise data source still, is it on the road map, when we can expect this will not be an obstable? data flow gen2 does support on-prem data source right now, only thing is from my testing, the performance of it is quite bad compared with if I use azure data factory. Hopefylly Fabric data factory pipeline will be a lot better later when it can connect to on-prem data source. Also, what about CI/CD ? I have ADF pipeline which copies data from on prem data source to data lake in fabric, next step is load it into fabric data warehouse, how do I connect these two step, since step 1 is ADF, it cannot see step 2 which is within Fabric.

Re: fabric data factory pipeline access on-premise data source

NandanHegde — Wed, 31 Jan 2024 04:22:24 GMT

Hey,

Couple of aspects :

1) Unlike PaaS version of ADF, Fabric Datapipeline and Dataflow gen 2 both use the Capacity as the support engine so depending on the capacity the performance might change.

The below link provides the roadmap :

https://learn.microsoft.com/en-us/fabric/release-plan/data-factory

and the plan seems to be in Q1 2024 for on prem support

2) Unfortunately the Data pipelines JSON code is read only as of today hence CICD is currently not possible but based on the road map w.r.t Git integrations, it would be available eventually

3) w.r.t 3rd ask, unfortuantly those 2 jobs would be in silos and you cannot directly integrate them because as of today Fabric doesnt have any REST API to trigger the pipelines externally and it doesnt support event/blob trigger as well.

Re: fabric data factory pipeline access on-premise data source

v-cboorla-msft — Thu, 01 Feb 2024 05:41:45 GMT

Hi Anonymous

Thanks for uisng Microsoft Fabric Community.

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others.
Otherwise, will respond back with the more details and we will try to help.

Thanks

Re: fabric data factory pipeline access on-premise data source

Anonymous — Sat, 03 Feb 2024 16:54:50 GMT

Thanks for checking, I accepted a solution

Re: fabric data factory pipeline access on-premise data source

v-cboorla-msft — Mon, 05 Feb 2024 05:38:38 GMT

Hi Anonymous

Glad that your query got resolved.

Please continue using Fabric Community for any help regarding your queries.