Forum Discussion

trebllaw's avatar
trebllaw
Frequent Visitor
1 year ago

Oracle Source to Lakehouse

What is the best way to load huge amount of data from an Oracle source to a Lakehouse? We are trying to load our FACT tables which has millions of rows onto a Fabric Lakehouse. Typically, these tables are terabytes in size. Just wanted to know the best way to approach this with a balance of speed and efficiency. Do we go using the Dataflow Gen2 route or Data Pipeline route?

 

We are currently testing the pipeline route, but its taking too long to copy over the data. We had to cancel the current run of this pipeline since its still running after 8 hours.

data pipeline

8 Replies

  • DataBard's avatar
    DataBard
    Icon for Most Valuable Professional rankMost Valuable Professional
    1 year ago

    Hi trebllaw,

    Some questions to clarify what you're looking for:

    1. Is this a one-time moving of data from Oracle to Fabric, or would this be on a routine schedule?
      1. If routine, will you need to process all historical data every time, or can you go incremental?
    2. Are any transformations occurring while moving the data to Fabric, or is it a basic 'copy and paste'?
  • trebllaw's avatar
    trebllaw
    Frequent Visitor
    1 year ago

    Currently with our testing, it would be a one time thing since we are trying to replicate what we have in the on prem oracle warehouse to the Fabric Lakehouse.

     

    In the future, we would also like do updates on the Lakehouse whenever the warehouse changes, maybe this would be scheduled like once a day or something.

     

    Its a direct copy and paste, no transformation whatsoever. Well, does changing of the detected datatype count? Sometimes Fabric detects the data type of a column as float when really its an integer, so we change that during the copy.

  • Anonymous's avatar
    Anonymous
    Not applicable
    1 year ago

    Hi trebllaw,

    Perhaps you can try to use shortcut feature to getting data. It will create a shortcut to load data from remote data source without data movement.
    Since current not support direct create oracle shortcut, you may need to mapping oracle data to supported data source(e.g. Azure, Dataverse…), then you can use shortcut to getting data from azure.

    Unify data sources with OneLake shortcuts - Microsoft Fabric | Microsoft Learn
    Regards,

    Xiaoxin Sheng

    • trebllaw's avatar
      trebllaw
      Frequent Visitor
      1 year ago

      As much as possible, we'd like to avoid additional costs from using Azure since we are already paying for 3 P3 capacities as of the moment. That is why we didnt go the use staging route as that would consume Data Lake Gen 2 storage, which is an additional cost.