Forum Discussion

akkibuddy7's avatar
akkibuddy7
Icon for Helper I rankHelper I
1 year ago
Solved

Incremental refresh For S3 shortcut

I have parquet Partitioned data in S3 and using shortcut it only pulls the files not table and to solve this I'm using Notebook to convert that into delta tables in lakehouse. But S3 data gets refres...
data engineering
lakehouse
  • spark123's avatar
    spark123
    1 year ago

    If you are trying to run the notebook without a pipeline use schedule from inside the notebook 

     

hareshgowri's avatar
hareshgowri
Icon for Microsoft Employee rankMicrosoft Employee
2 days ago

Materialized Lake Views now have declarative experience to ingest files incrementally into Lakehouse as delta table. To start with, CSV and Parquet files are supported now with more formats being planned.  Folder or File shortcuts in OneLake could be a source to the Materialized Lake View. OneLake shortcuts support AWS s3 as source and so if you have an active folder shortcut created, then configuring a Materialized Lake View on top would automatically detect changes in the source folder and keep adding data from new files incrementally. Schema evolution is handled too with an option for users to configure Dynamic or Fixed schema. 

Please look at the documentation for more: Spark SQL Reference for Materialized Lake Views - Microsoft Fabric | Microsoft Learn