Starting December 3, join live sessions with database experts and the Microsoft product team to learn just how easy it is to get started
Learn moreGet certified in Microsoft Fabric—for free! For a limited time, get a free DP-600 exam voucher to use by the end of 2024. Register now
I have a shortcut in a lakehouse pointing to a table in a warehouse. The record count I get when querying the lakehouse table is the same as when the shortcut was created. The warehouse table has had additional rows appended since the shortcut was created.
When I drill into the default data set >> Settings>>Refresh history, I see the error below. When I look at the files that underlie the shortcut in the lakehouse, I see additional parquet files that have created times that correspond with the refresh times.
An error has occurred while checking the latest version of the direct lake table Transfer, error System.InvalidOperationException: The requested operation requires an element of type 'Object', but the target element has type 'Null'. at System.Text.Json.JsonElement.EnumerateObject() at Azure.Core.Pipeline.StorageRequestFailedDetailsParser.TryParse(Response response, ResponseError& error, IDictionary`2& data) at Azure.RequestFailedException.GetRequestFailedExceptionContent(Response response, RequestFailedDetailsParser parser) at Azure.RequestFailedException..ctor(Response response, Exception innerException, RequestFailedDetailsParser detailsParser) at Azure.Storage.Files.DataLake.FileSystemRestClient.<ListPathsAsync>d__21.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Azure.Storage.Files.DataLake.DataLakeFileSystemClient.<GetPathsInternal>d__65.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Azure.Storage.Files.DataLake.Models.GetPathsAsyncCollection.<GetNextPageAsync>d__6.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Threading.Tasks.ValueTask`1.get_Result() at Azure.Storage.StorageCollectionEnumerator`1.StorageAsyncPageable.<GetAsyncEnumerator>d__5.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore`1.GetResult(Int16 token) at Microsoft.ASWL.Service.Engine.SeethruAutoSync.DirectLakeTable.<FetchTableVersionAsync>d__30.MoveNext() in C:\__w\1\s\ASWL.Service\Engine\SeethruAutoSync\DirectLakeTable.cs:line 107 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at Microsoft.ASWL.Service.Engine.SeethruAutoSync.DirectLakeTable.<FetchTableVersionAsync>d__30.MoveNext() in C:\__w\1\s\ASWL.Service\Engine\SeethruAutoSync\DirectLakeTable.cs:line 107.
I experienced the same issue. I created a shortcut pointing to a delta table. I checked the row counts in both databricks and OneLake environments. The counts were identical. I then deleted all the rows from the table in the databricks environment. I then checked for the counts in both environments. While the databricks table has 0 rows, the OneLake table still has all the rows, obviously still caching the old rows. I have tried multiple times and my shortcuts never seem to reflect the source data they depend on.
Hi @jon_clemens
Apologies for the delay in response.
Thanks for using Microsoft Fabric Community and posting your question here.
Just wanted to check whether your issue got resolved, please do let us know in case of any further queries.
Hi @jon_clemens
Following up to see whether your issue got resolved, please do let us know in case of any further queries.
@jon_clemens It seems listed in known issues with a workaround
https://learn.microsoft.com/en-us/fabric/get-started/known-issues/known-issue-453-data-warehouse-pub...
Thanks! I had not found this known issue. Do you (or anyone else) have any insight into plans to address this known issue?
FYI, in my case, I have tried the known issue. It will work "one time". If you then do another incremental upate to the warehouse table, it will once again break.
Here is an update on this question. I think I have arrived at the following conclusion. Shortcuts in a lakehouse pointed to a warehouse table do not like any sql operation that a lakehouse table (sql endpoint) does not allow. This means that any update or delete performed against the warehouse table, then renders the lakehouse shortcut broken, meaning that the shortcut does not display any of the changes after the update or delete. The rowcount is wrong and the data is wrong. And the error that I posted above, or something similar, appears in the dataset refresh history.
This does not seem like the intended behavior, does it? It would make using shortcuts in the lakehouse pointing to warehouse tables essentially useless.
I have experienced this as well and agree that this does not seem to be the correct behavior. I created a shortcut to a warehouse table to create a directlake dataset and the shortcut has old data and is not updated. Even after deleting the shortcut and recreating it I am still seeing old data. This seems like an oversite and not trully a shortcut but instead a snapshot.
Starting December 3, join live sessions with database experts and the Fabric product team to learn just how easy it is to get started.
Check out the November 2024 Fabric update to learn about new features.
User | Count |
---|---|
5 | |
4 | |
2 | |
1 | |
1 |
User | Count |
---|---|
14 | |
7 | |
5 | |
4 | |
3 |