Inconsistent Behavior When Using Lakehouse Tables in Warehouse-Scoped Queries in Copy Activity
Hi Fabric Team,
I’m writing to report a bug we've encountered while using Microsoft Fabric. we rely on Fabric for our ETL and ELT processes, leveraging both Warehouse and Lakehouse features.
As you know, when setting up a copy activity, we are typically allowed to use a query only on Warehouse, not on Lakehouse tables. However, we discovered that Fabric currently allows users to reference Lakehouse tables in a Warehouse-scoped query, which results in subtle and hard-to-detect issues.
This behavior caused a critical problem in one of our production pipelines: records from yesterday’s load were not copied from Fabric to Oracle. After thorough investigation, we found that the issue stemmed from a query referencing a Lakehouse table, even though the copy activity was configured to use a Warehouse. Fabric did not validate this inconsistency, and as a result, the copy activity silently failed to move the expected data.
This bug could potentially have significant consequences in production environments. I recommend implementing a validation step to ensure that all objects used in a query are consistent with the selected source type (Warehouse vs. Lakehouse), to avoid such hidden issues in the future.
Kind regards,
Yazdan
Data Engineer
Recent ideas
Metadata-Driven Copy Job Control at Table and Column Level
We currently achieve metadata-driven table and column selection in Copy Data activities by using dynamic queries. This allows us to dynamically determine which tables and columns should be processed ...Motasim091 hour agoNew MemberNew2Views0likes0CommentsDynamic ForEach Parallel Batch Count Based on Runtime Capacity
We would like Microsoft Fabric Data Factory to support dynamic configuration of the ForEach activity's parallel batch count at runtime. Currently, the ForEach batchCount is effectively configured as...Motasim091 hour agoNew MemberNew2Views0likes0CommentsMake Workspace-Level Surge Protection Limit Visible to Workspace Users
Please make the applicable workspace-level surge protection limit visible in the workspace settings. Currently, when surge protection is applied by Capacity Admins, workspace users don't get visibil...frithjof_v3 hours agoCommunity ChampionNew23Views0likes0CommentsAllow Custom Recipient Settings for Gateway Expiry Notifications
Now, during the final 90 days before the gateway version end-of-support date, operations continue, but semantic model contacts users can receive warnings in email The warning message is: "The on-prem...YuanMao4 hours agoNew MemberNew8Views1like0Comments