Forum Discussion

prom's avatar
prom
Frequent Visitor
2 years ago

Dataflow Gen2 from on prem Oracle - burning CU like crazy

Hello @all

 

We need to transfer data from on-prem Oracle table to the Lakehouse. As of tuday, in Fabric, is it only possible using DF Gen2 (copy pipeline doesn't support on-prem gw yet). 

 

We are using on prem db gatweay.

 

The problem is that we cannot import tables - with more then 2mln of rows, 150 columns (compressed parquet size is about 500mb) in DF2.

 

The flow run twice and finished with errors:

 1 - timed out after ~ 1h 15m

 2  - capacity exhausted 🙂

 

We are using F2 for testing.

 

The worst is, that DF2 burned almost 140k CU doing nothing. All work was done on on-prem gateway, not in the fabric. 

 

Comparing with ADF v2 - we were able to import same table using self-hosted IR using only 0.2 hours data movement activities (0.02 eur) in 10 mins.

 

Is there any way to import on-prem data into fabric directly? Should I wait until Cp Pipelines support on prem gw/ir?

 

Do you have any experience with that?

 

This is my first post - so greeting Everyone 🙂

 

Best of all

prom

 

 

7 Replies