Inconsistent Behavior When Using Lakehouse Tables in Warehouse-Scoped Queries in Copy Activity
Hi Fabric Team,
I’m writing to report a bug we've encountered while using Microsoft Fabric. we rely on Fabric for our ETL and ELT processes, leveraging both Warehouse and Lakehouse features.
As you know, when setting up a copy activity, we are typically allowed to use a query only on Warehouse, not on Lakehouse tables. However, we discovered that Fabric currently allows users to reference Lakehouse tables in a Warehouse-scoped query, which results in subtle and hard-to-detect issues.
This behavior caused a critical problem in one of our production pipelines: records from yesterday’s load were not copied from Fabric to Oracle. After thorough investigation, we found that the issue stemmed from a query referencing a Lakehouse table, even though the copy activity was configured to use a Warehouse. Fabric did not validate this inconsistency, and as a result, the copy activity silently failed to move the expected data.
This bug could potentially have significant consequences in production environments. I recommend implementing a validation step to ensure that all objects used in a query are consistent with the selected source type (Warehouse vs. Lakehouse), to avoid such hidden issues in the future.
Kind regards,
Yazdan
Data Engineer
Recent ideas
About "Download SQL database project" and fix the downloaded project
1. The sqlproj must be fixed, it is downloaded without the xml markup line <?xml version="1.0" encoding="utf-8"> 2. The sdk must be updated: Actual: <Sdk Name="Microsoft.Build.Sql" Version="0.1...Jarrinf2 days agoFrequent VisitorNew74Views3likes3CommentsAdd REST API to Take Ownership of Fabric Pipelines
When using service account to automatically deploy Fabric pipelines from Dev workspace to upper environments, the owner of pipelines in target envionment remain unchanged. In some situations, per...VickyPeng2 days agoRegular VisitorNew366Views5likes2CommentsUpdate the 'Users can set up email subscriptions' in Fabric Admin Portal to specific groups
I want to be able to allow a specific group of users to set up email subscriptions. In my admin portal view I only get the following Can this be updated to allow specific user groups? ...Chart682 days agoNew MemberNew138Views1like1Comment