Forum Discussion

trebllaw's avatar
trebllaw
Frequent Visitor
1 year ago

Oracle Source to Lakehouse

What is the best way to load huge amount of data from an Oracle source to a Lakehouse? We are trying to load our FACT tables which has millions of rows onto a Fabric Lakehouse. Typically, these table...
data pipeline
trebllaw's avatar
trebllaw
Frequent Visitor
1 year ago

As much as possible, we'd like to avoid additional costs from using Azure since we are already paying for 3 P3 capacities as of the moment. That is why we didnt go the use staging route as that would consume Data Lake Gen 2 storage, which is an additional cost.

Anonymous's avatar
Anonymous
Not applicable
1 year ago

HI trebllaw,

Perhaps you can consider using sempy link which can used to getting data from Semantic model.

What is semantic link? - Microsoft Fabric | Microsoft Learn
You can design a PBI report on desktop with live connection model to the oracle data source and publish it to power bi service workspace that assigned with fabric capacity.
After you assign gateway to handle its refresh, you can use notebook to getting data from that model and loading to Lakehouse.

Semantic link and Power BI connectivity - Microsoft Fabric | Microsoft Learn

Regards,

Xiaoxin Sheng

  • trebllaw's avatar
    trebllaw
    Frequent Visitor
    1 year ago

    Unfortunately, this wouldnt work for us since we are going to be loading huge amount of data from multiple FACT tables.

     

    The main reason we are trying to use pipelines is that it is advertised to be the ideal option to move large amounts of data, being able to load petabytes of data pretty quick, but based from our tests, we are getting about 5Kb/s of throughput. For 3.5, we only managed to process and transfer about 1.5M rows. We may be doing something wrong here we are not sure. 

     

    Here are some of the things that might be of note:

    1. Oracle source system isnt partitioned.

    2. We've set the pipeline to dynamic partitioning and set one of the key fields as index.

    3. Parallel copies automatically set to 3