Forum Discussion
Dataflow Gen2 from on prem Oracle - burning CU like crazy
Hello @all
We need to transfer data from on-prem Oracle table to the Lakehouse. As of tuday, in Fabric, is it only possible using DF Gen2 (copy pipeline doesn't support on-prem gw yet).
We are using on prem db gatweay.
The problem is that we cannot import tables - with more then 2mln of rows, 150 columns (compressed parquet size is about 500mb) in DF2.
The flow run twice and finished with errors:
1 - timed out after ~ 1h 15m
2 - capacity exhausted 🙂
We are using F2 for testing.
The worst is, that DF2 burned almost 140k CU doing nothing. All work was done on on-prem gateway, not in the fabric.
Comparing with ADF v2 - we were able to import same table using self-hosted IR using only 0.2 hours data movement activities (0.02 eur) in 10 mins.
Is there any way to import on-prem data into fabric directly? Should I wait until Cp Pipelines support on prem gw/ir?
Do you have any experience with that?
This is my first post - so greeting Everyone 🙂
Best of all
prom
7 Replies
- AnonymousNot applicable
Hi prom ,
Thanks for using Fabric Community.
Unfortunately you can only connect to On-prem is via DataFlow Gen 2.
The feature - "Connecting to the On-premise gateways using Pipelines" is still on roadmap. I will keep you posted regarding the updates.Docs to refer -
What's new and planned for Data Factory in Microsoft Fabric - Microsoft Fabric | Microsoft Learn
How to access on-premises data sources in Data Factory - Microsoft Fabric | Microsoft Learn
Hope this helps. Please let me know if you have any further questions.- AnonymousNot applicable
Hello prom ,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .
Otherwise, will respond back with the more details and we will try to help .- promFrequent Visitor
Hello,
Thanks for quick answer. I'll probably wait for migration CP pipelines to Fabric. I'll try using ADF or custom spark cluster as a workaround.
Best Regards