Forum Discussion
Exclude All Null Column
Hi There, I am building a pipeline that copies some tables from one Database and plonks it into an Azurew SQL Database, in which I can then save the tables into a Lakehouse as Parquet files. My p...
Just to confirm,
- Is this a Fabric Data Pipeline? (e.g. not a data factory pipeline)
- The first Copy Data task populates a list of available tables for import?
- The For Each contains the copy task to move the data from the source database to the azure sql database, and then from the Azure SQL db to the Lakehouse?
- Has the new column been added to the source database that you're loading from?
- The For Each is failing because the new column doesn't exist downstream in the Azure SQL database?
- Is there a reason why you're not writing directly to the Lakehouse from the source database (eg cut out the azure sql db)?
Just a thought, but if you use the Overwrite option when writing to a Lakehouse table (not Files as the schema could evolve anyway if you were just writing parquet files) then that will evolve the schema of the lakehouse table automatically when source columns get added to the source.
Not sure if that could help in your situation? Overwrite would replace the data downstream with the latest incoming so not sure if that's what you want