lakehouse
4 Topics
How can I save a delta table from lakehouse1 stored in wk1 to lakehouse2 in wk2 using a NB pyspark?
I have a notebook in wk1 that reads table1 from lakehouse1, performs a series of transformations, and then saves the new delta table in lakehouse2 in wk2, but I get an error saying that it cannot find the lakehouse in the catalogue when it is included in the notebook as an item. How can I make it visible? How can I use the save or saveAsTable statement to make it work? Lakehouses are without schemas.Solved2.3KViews1like5CommentsBug: Inconsistent Behavior When Using Lakehouse Tables in Warehouse-Scoped Queries in Copy Activity
Hi Fabric Team, I’m writing to report a bug we've encountered while using Microsoft Fabric. we rely on Fabric for our ETL and ELT processes, leveraging both Warehouse and Lakehouse features. As you know, when setting up a copy activity, we are typically allowed to use a query only on Warehouse, not on Lakehouse tables. However, we discovered that Fabric currently allows users to reference Lakehouse tables in a Warehouse-scoped query, which results in subtle and hard-to-detect issues. This behavior caused a critical problem in one of our production pipelines: records from yesterday’s load were not copied from Fabric to Oracle. After thorough investigation, we found that the issue stemmed from a query referencing a Lakehouse table, even though the copy activity was configured to use a Warehouse. Fabric did not validate this inconsistency, and as a result, the copy activity silently failed to move the expected data. This bug could potentially have significant consequences in production environments. I recommend implementing a validation step to ensure that all objects used in a query are consistent with the selected source type (Warehouse vs. Lakehouse), to avoid such hidden issues in the future. Kind regards, Yazdan Data EngineerSolved1.8KViews0likes7Comments
