Forum Discussion
Exclude All Null Column
Hi There, I am building a pipeline that copies some tables from one Database and plonks it into an Azurew SQL Database, in which I can then save the tables into a Lakehouse as Parquet files. My p...
Just to confirm,
- Is this a Fabric Data Pipeline? (e.g. not a data factory pipeline)
- The first Copy Data task populates a list of available tables for import?
- The For Each contains the copy task to move the data from the source database to the azure sql database, and then from the Azure SQL db to the Lakehouse?
- Has the new column been added to the source database that you're loading from?
- The For Each is failing because the new column doesn't exist downstream in the Azure SQL database?
- Is there a reason why you're not writing directly to the Lakehouse from the source database (eg cut out the azure sql db)?