topic Re: Dataflow Gen2 from on prem Oracle - burning CU like crazy in Dataflow

Dataflow Gen2 from on prem Oracle - burning CU like crazy

prom — Tue, 12 Mar 2024 09:41:55 GMT

Hello @all

We need to transfer data from on-prem Oracle table to the Lakehouse. As of tuday, in Fabric, is it only possible using DF Gen2 (copy pipeline doesn't support on-prem gw yet).

We are using on prem db gatweay.

The problem is that we cannot import tables - with more then 2mln of rows, 150 columns (compressed parquet size is about 500mb) in DF2.

The flow run twice and finished with errors:

1 - timed out after ~ 1h 15m

2 - capacity exhausted 🙂

We are using F2 for testing.

The worst is, that DF2 burned almost 140k CU doing nothing. All work was done on on-prem gateway, not in the fabric.

Comparing with ADF v2 - we were able to import same table using self-hosted IR using only 0.2 hours data movement activities (0.02 eur) in 10 mins.

Is there any way to import on-prem data into fabric directly? Should I wait until Cp Pipelines support on prem gw/ir?

Do you have any experience with that?

This is my first post - so greeting Everyone 🙂

Best of all

prom

Re: Dataflow Gen2 from on prem Oracle - burning CU like crazy

Anonymous — Tue, 12 Mar 2024 10:59:30 GMT

Hi prom ,

Thanks for using Fabric Community.
Unfortunately you can only connect to On-prem is via DataFlow Gen 2.
The feature - "Connecting to the On-premise gateways using Pipelines" is still on roadmap. I will keep you posted regarding the updates.

Docs to refer -
What's new and planned for Data Factory in Microsoft Fabric - Microsoft Fabric | Microsoft Learn
How to access on-premises data sources in Data Factory - Microsoft Fabric | Microsoft Learn

Hope this helps. Please let me know if you have any further questions.

Re: Dataflow Gen2 from on prem Oracle - burning CU like crazy

Anonymous — Wed, 13 Mar 2024 12:04:04 GMT

Hello prom ,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .
Otherwise, will respond back with the more details and we will try to help .

Re: Dataflow Gen2 from on prem Oracle - burning CU like crazy

prom — Wed, 13 Mar 2024 14:50:24 GMT

Hello,

Thanks for quick answer. I'll probably wait for migration CP pipelines to Fabric. I'll try using ADF or custom spark cluster as a workaround.

Best Regards

Re: Dataflow Gen2 from on prem Oracle - burning CU like crazy

pqian_MSFT — Wed, 13 Mar 2024 18:12:45 GMT

prom one thing you could try is disable staging for your entities, it should at least complete without running twice.

We are aware of an issue that may slow down ingestion into LH if your data contains large strings (>4k in a cell) or column that is primarily nulls. If that is what you are running into, then the next GW patch will have this fix and it should speed it up by another 40%.

Re: Dataflow Gen2 from on prem Oracle - burning CU like crazy

Anonymous — Fri, 15 Mar 2024 06:18:06 GMT

Hi prom ,

We haven’t heard from you on the last response and was just checking back to see if we answered your query.
Otherwise, will respond back with the more details and we will try to help .

Re: Dataflow Gen2 from on prem Oracle - burning CU like crazy

Element115 — Tue, 19 Mar 2024 00:59:27 GMT

What about this approach?

Calling Microsoft Flow from your application | Power Automate Blog

Chris Webb's BI Blog: Calling Microsoft Flow From Power Query And Power BI (crossjoin.co.uk)

Re: Dataflow Gen2 from on prem Oracle - burning CU like crazy

prom — Tue, 19 Mar 2024 06:42:58 GMT

Hello,

I'm using ADF CP pipeline with lakehouse destination for now. It works pretty well with on-prem IR. But still waiting for native fabric functionality with equal througput and performance.

Regards

Prom