We need something low code close to autoloader of databricks in Fabric, which is built on top of structured streaming, that can load data incrementally from lake files in ADLS/One Lake into lakehouse as delta table, that can do automatic schema evolution checks as well.
Thanks,
Gaurav
3 Comments
tvilutis1Microsoft Employee
We are evaluating this idea and its timeline along with other priorities. Your comments and votes will help us plan and design the feature. Thank you for your feedback!- Status added:Needs Votes
hareshgowriMaterialized Lake Views now have declarative experience to ingest files incrementally into Lakehouse as delta table. To start with CSV and Parquet files supported now with more formats being planned. Schema evolution is handled too with an option for users to configure Dynamic or Fixed schema.
Please look at the documentation for more: Spark SQL Reference for Materialized Lake Views - Microsoft Fabric | Microsoft Learn
