Forum Discussion
Exclude All Null Column
Hi There, I am building a pipeline that copies some tables from one Database and plonks it into an Azurew SQL Database, in which I can then save the tables into a Lakehouse as Parquet files. My p...
Just to confirm,
- Is this a Fabric Data Pipeline? (e.g. not a data factory pipeline)
- The first Copy Data task populates a list of available tables for import?
- The For Each contains the copy task to move the data from the source database to the azure sql database, and then from the Azure SQL db to the Lakehouse?
- Has the new column been added to the source database that you're loading from?
- The For Each is failing because the new column doesn't exist downstream in the Azure SQL database?
- Is there a reason why you're not writing directly to the Lakehouse from the source database (eg cut out the azure sql db)?
Hi AndyDDC , thanks for responding!
In answer to your bullet points:
- The first Copy Data task populates a list of available tables for import?
Yes, it runs through and lists out all of the tables for import. The table in question shows up in this step. - The For Each contains the copy task to move the data from the source database to the azure sql database, and then from the Azure SQL db to the Lakehouse?
Yes, it sends a copy of the data to the Azure database and then from there to the Lakehouse as Parquet files. - Has the new column been added to the source database that you're loading from?
Yes, the new column was added to the source database last week, but it wasn't communicated that it was being added so we only found out about it through the pipeline breaking. - The For Each is failing because the new column doesn't exist downstream in the Azure SQL database?
I believe so, but I'm not sure how to go about fixing this step as there is no explicit mapping to the source to be able to define the column or data type. I think this is the step that I'm just not getting my head around. I'm still getting to grasps with it all as I'm an analyst by trade but having to fill in until we get a proper engineer so trying my best until then 😊 - Is there a reason why you're not writing directly to the Lakehouse from the source database (eg cut out the azure sql db)?
We're building to a medallion architecture so the parquet files will give us some flexibility when applying transformations for the 'Silver Layer'. Definitely more than happy to listen to any suggestions if there are better ways of doing it.