Inconsistent Behavior When Using Lakehouse Tables in Warehouse-Scoped Queries in Copy Activity
Hi Fabric Team,
I’m writing to report a bug we've encountered while using Microsoft Fabric. we rely on Fabric for our ETL and ELT processes, leveraging both Warehouse and Lakehouse features.
As you know, when setting up a copy activity, we are typically allowed to use a query only on Warehouse, not on Lakehouse tables. However, we discovered that Fabric currently allows users to reference Lakehouse tables in a Warehouse-scoped query, which results in subtle and hard-to-detect issues.
This behavior caused a critical problem in one of our production pipelines: records from yesterday’s load were not copied from Fabric to Oracle. After thorough investigation, we found that the issue stemmed from a query referencing a Lakehouse table, even though the copy activity was configured to use a Warehouse. Fabric did not validate this inconsistency, and as a result, the copy activity silently failed to move the expected data.
This bug could potentially have significant consequences in production environments. I recommend implementing a validation step to ensure that all objects used in a query are consistent with the selected source type (Warehouse vs. Lakehouse), to avoid such hidden issues in the future.
Kind regards,
Yazdan
Data Engineer
Recent ideas
Auto-generate INSERT VALUES scripts for configuration data migration across environments
Hi Fabric Team, I would like to suggest a feature that could significantly simplify deployment and environment promotion activities. Today, when configuration tables contain reference or applicatio...todigen3 hours agoRegular VisitorNew10Views1like0CommentsAllow Non-Breaking Schema Changes in Fabric CI/CD Deployments for Data Warehouse Objects
Hi, currently, Microsoft Fabric deployment pipelines can fail when a Data Warehouse deployment contains schema changes such as adding, removing, renaming, or modifying columns in existing tables. I...todigen3 hours agoRegular VisitorNew8Views0likes0CommentsAllow Non-Breaking Schema Changes in Fabric CI/CD Deployments for Data Warehouse Objects
Hi, currently, Microsoft Fabric deployment pipelines can fail when a Data Warehouse deployment contains schema changes such as adding, removing, renaming, or modifying columns in existing tables. I...todigen3 hours agoRegular VisitorNew19Views1like0Comments