Forum Discussion

P_Stavis's avatar
P_Stavis
Regular Visitor
1 year ago
Solved

Import multiple parquet files but with partial updates

Hello!   I extract data from SAP each month, generating multiple .xlsx files (one per month). To optimize the size of the database, I decided to convert each of these monthly files into separate .p...
Need Help
  • lbendlin's avatar
    lbendlin
    1 year ago

    The only way to avoid a full load is to use incremental refresh with a foldable source.  Doesn't seem to be an option in your scenario.

     

    Having the data in Parquet format should already give you the highest possible performance. If you put the Parquet files into a data lake you could also consider using the DirectLake connection (where the data is connected live).  Performance will be slightly inferior to import mode.

lbendlin's avatar
lbendlin
Icon for Super User rankSuper User
1 year ago

The only way to avoid a full load is to use incremental refresh with a foldable source.  Doesn't seem to be an option in your scenario.

 

Having the data in Parquet format should already give you the highest possible performance. If you put the Parquet files into a data lake you could also consider using the DirectLake connection (where the data is connected live).  Performance will be slightly inferior to import mode.