Forum Discussion

devaj123's avatar
devaj123
Frequent Visitor
1 year ago
Solved

On Premises SQL Historical table taking long time to load into Fabric Lakehouse.

Hi Team,   I have a sql server historical table with milloins of records and table has nearly 150 columns i need to bring the entire data into lake house. I tried Dataflow Gen2, Copy job, pipeline b...
workspaces
  • v-prasare's avatar
    v-prasare
    1 year ago

    Hi devaj123,

    Using Notebooks with PySpark provides the best performance and flexibility, especially for large tables. If this is a historical table that changes over time, implement watermarking and load only new or updated rows using a timestamp or identity column.

     

    Repartition the data to spread it across multiple parallel tasks, especially helpful for performance when writing large volumes. Save the data in Delta format, which supports efficient queries, updates, and time travel versioning.

     

     

     


    Thanks,

    Prashanth Are

    MS Fabric community support


    If our resolved your issue, please mark it as "Accept as solution" and click "Yes" if you found it helpful.

v-prasare's avatar
v-prasare
Icon for Community Support rankCommunity Support
1 year ago

Hi devaj123,

As we haven’t heard back from you, we wanted to kindly follow up to check if there is any progress on above mentioned issue. if you got any updates i requet you to please post here and accept as solution. This will help other community members with similar issues find answers easily.

 

 

 

Thanks,

Prashanth Are

MS Fabric community support