Exclude All Null Column

Thanks.

So this is just my opinion on this, but you could write from the source database directly to the Lakehouse Files section (your raw/bronze layer).  That would mean that any new columns/altered columns in the source will just get automatically written to Parquet files in the lakehouse files section.

The issue as you've discovered is that there is no auto-update on the azure sql database table schema, you'd have to either:

• drop and recreate the table in the azure sql database using a pre-copy script in the copy data task and set the auto-create table option on (this will of course delete the data from the azure sql db)
• 

   

• for each table, pass in a sql statement that only included the columns you want into the copy data task
• 

   

You could go one step further and just write the source data to the lakehouse tables and use Overwrite (although it does "overwrite" the data in the destination table, although keeps history using delta versioning).