Auto table loader

[cmaneu]

Today, you can easily transform a CSV file stored in the LakeHouse into a Delta Table. That's great for development and data exploration scenarios.

It would be great to have a way to go from this one-time data import to a recurring import. Here are some ideas about how this could be featured to the user:

View Code - Notebooks

Once the first load has successfully completed, a link could be inserted in the completion in-app notification to "View import code". When clicking on this link, you get redirected to a new notebook that contains two cells:

• One with the data frame loading code
• One with the cote to write to a delta table

Pros:

• From there, you can do your own manipulations if you need (change type/name mapping, etc...)
• Notebooks are easily scheduled

Cons:

• It only addresses users that write code.

Open as a Dataflow Gen2

An alternative option is to enable the use of a Data flow Gen2 to import that CSV to a delta table

Pros:

• It offers a Low-Code option
• Refresh Frequency schedule can be easily set

Cons:

• Depending on the file structure & name, it could be more difficult to read only new files with Data Flow

User scenarios for this idea

• Each week, I have a CSV file that gets updated with some reference data. I need to replace values in the delta table with the ones from the CSV
• Each night, each store drops a CSV file with the sale receipts of the previous day in a specific structure (eg \lakehouse\incoming_receipts\[STORE_ID]\YYYY\MM\DD\sales.csv. I need to load all CSV files corresponding to yesterday and append them in the existing Delta table