Currenly the only method to drop / truncate tables in a Lakehouse during a pipeline run is to start a spark session and initiate spark.sql to execute a sql statment.
In Data Factory the script activity does not support lakehouse as source and the sql endpoint of a lakehouse does not support drop.
Can an activity be created or the script activity updated in Data Factory pipelines to execute a sql statement on a lakehouse without having to start a notebook and spark session to do it.
Starting a spark session can take up to 5 mins so takes too long if we simply want drop/truncate a few lakehouse tables.
1 Comment
- A Copy Activity in a pipeline, with the Destination Table Action set to "Overwrite" will create a backup of the original table, then recreate that table. So every reload of that table gets you another copy of the table named "MyTable_backup_
" Come ON Microsoft, "Overwrite" means just that": Replace what is there. It does NOT mean, "make me a copy and start over". Give us an option to NOT create a backup table.
