Forum Discussion

Tanayraj's avatar
Tanayraj
Frequent Visitor
2 years ago
Solved

How can I load data into datamart form datawarehouse?

Hello!   I have created Medilon architecture. I completed it with a bronze layer and a silver layer.  Now to make one specific report I do not want to use all data that resides inside the sliver la...
  • Anonymous's avatar
    Anonymous
    2 years ago

    Hi Tanayraj 

     

    Instead of loading the entire dataset every time, consider implementing an incremental load strategy. This involves loading only the data that has changed since the last load. You can achieve this by using a watermark column (such as a last modified timestamp or an auto-incrementing ID) to identify new or updated records. Microsoft Fabric's Data Factory supports incremental loading, which can significantly reduce the volume of data transferred and the time required for each load.

    Incrementally load data from Data Warehouse to Lakehouse - Microsoft Fabric | Microsoft Learn
    Given the complexity of your SQL query, it's worth reviewing and optimizing it to reduce execution time. This could involve indexing key columns used in joins and where clauses, reducing the number of joins if possible, or breaking the query into smaller, more manageable parts.

     

    Or maybe you can consider using Data Factory pipelines to perform the data transformation and loading. Data Factory allows you to create data-driven workflows for orchestrating and automating data movement and data transformation. You can create a pipeline that executes your SQL script to transform the data as needed and then loads the result into your datamart. This approach offers more control over the execution and can be optimized for performance.

    What is Data Factory - Microsoft Fabric | Microsoft Learn

     

    Best Regards

    Zhengdong Xu
    If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.