Inconsistent Behavior When Using Lakehouse Tables in Warehouse-Scoped Queries in Copy Activity
Hi Fabric Team,
I’m writing to report a bug we've encountered while using Microsoft Fabric. we rely on Fabric for our ETL and ELT processes, leveraging both Warehouse and Lakehouse features.
As you know, when setting up a copy activity, we are typically allowed to use a query only on Warehouse, not on Lakehouse tables. However, we discovered that Fabric currently allows users to reference Lakehouse tables in a Warehouse-scoped query, which results in subtle and hard-to-detect issues.
This behavior caused a critical problem in one of our production pipelines: records from yesterday’s load were not copied from Fabric to Oracle. After thorough investigation, we found that the issue stemmed from a query referencing a Lakehouse table, even though the copy activity was configured to use a Warehouse. Fabric did not validate this inconsistency, and as a result, the copy activity silently failed to move the expected data.
This bug could potentially have significant consequences in production environments. I recommend implementing a validation step to ensure that all objects used in a query are consistent with the selected source type (Warehouse vs. Lakehouse), to avoid such hidden issues in the future.
Kind regards,
Yazdan
Data Engineer
Recent ideas
Allow NotebookUtils getSecret() to authenticate with Workspace Identity
Current behavior In Microsoft Fabric, notebookutils.credentials.getSecret() authenticates against Azure Key Vault using the identity of the user who executes the notebook. This behavior appli...tmihara3 hours agoNew MemberNew4Views0likes0CommentsSupport Synonyms in Fabric Warehouse
Microsoft SQL Server has a very powerful feature by the way of "synonyms." It allows users and DBAs to do all sorts of powerful magic such as rewiring objects under the hood (e.g. run the code agains...matthias-bi7 hours agoRegular VisitorNew1.3KViews18likes2CommentsExpose Refresh Warnings and Informational Messages via Notifications and API
When a Power BI semantic model refresh completes successfully, the status shows Completed, even when the refresh details contain warnings or informational messages that require attention. Please pro...Jashwanth_K9 hours agoMicrosoft EmployeeNew24Views6likes0CommentsInvoke Pipeline Task - Workspace Identity Authentication
Currently, the Fabric Data Factory Invoke Pipeline task uses the user's credentials who saved the pipeline to then authenticate to the Azure Data Factory to execute the ADF pipeline. When that user'...dzebrowitz13 hours agoAdvocate IPlanned1.8KViews61likes5CommentsFabric Pipeline should run as workspace identity
Currently, Microsoft Fabric pipelines run under the identity of the last user who modified them, which can cause disruptions when tenant administrators make changes to security policies, such as enab...pellitteris14 hours agoAdvocate IINew1.4KViews27likes3Comments