Datalake shortcut is not showing the same data (i.e. row count) as the source datawarehouse table

[jon_clemens]

I have a shortcut in a lakehouse pointing to a table in a warehouse.  The record count I get when querying the lakehouse table is the same as when the shortcut was created.  The warehouse table has had additional rows appended since the shortcut was created.  

When I drill into the default data set >> Settings>>Refresh history, I see the error below.  When I look at the files that underlie the shortcut in the lakehouse, I see additional parquet files that have created times that correspond with the refresh times.

An error has occurred while checking the latest version of the direct lake table Transfer, error System.InvalidOperationException: The requested operation requires an element of type 'Object', but the target element has type 'Null'. at System.Text.Json.JsonElement.EnumerateObject() at Azure.Core.Pipeline.StorageRequestFailedDetailsParser.TryParse(Response response, ResponseError& error, IDictionary`2& data) at Azure.RequestFailedException.GetRequestFailedExceptionContent(Response response, RequestFailedDetailsParser parser) at Azure.RequestFailedException..ctor(Response response, Exception innerException, RequestFailedDetailsParser detailsParser) at Azure.Storage.Files.DataLake.FileSystemRestClient.<ListPathsAsync>d__21.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Azure.Storage.Files.DataLake.DataLakeFileSystemClient.<GetPathsInternal>d__65.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Azure.Storage.Files.DataLake.Models.GetPathsAsyncCollection.<GetNextPageAsync>d__6.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Threading.Tasks.ValueTask`1.get_Result() at Azure.Storage.StorageCollectionEnumerator`1.StorageAsyncPageable.<GetAsyncEnumerator>d__5.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore`1.GetResult(Int16 token) at Microsoft.ASWL.Service.Engine.SeethruAutoSync.DirectLakeTable.<FetchTableVersionAsync>d__30.MoveNext() in C:\__w\1\s\ASWL.Service\Engine\SeethruAutoSync\DirectLakeTable.cs:line 107 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at Microsoft.ASWL.Service.Engine.SeethruAutoSync.DirectLakeTable.<FetchTableVersionAsync>d__30.MoveNext() in C:\__w\1\s\ASWL.Service\Engine\SeethruAutoSync\DirectLakeTable.cs:line 107.

[Anonymous]

I experienced the same issue. I created a shortcut pointing to a delta table. I checked the row counts in both databricks and OneLake environments. The counts were identical. I then deleted all the rows from the table in the databricks environment. I then checked for the counts in both environments. While the databricks table has 0 rows, the OneLake table still has all the rows, obviously still caching the old rows. I have tried multiple times and my shortcuts never seem to reflect the source data they depend on. 

[v-cboorla-msft]

Hi @jon_clemens 

Apologies for the delay in response.

Thanks for using Microsoft Fabric Community and posting your question here.

Just wanted to check whether your issue got resolved, please do let us know in case of any further queries.

[v-cboorla-msft]

Hi @jon_clemens 

Following up to see whether your issue got resolved, please do let us know in case of any further queries.

[puneetvijwani]

@jon_clemens  It seems listed in known issues with a workaround 

https://learn.microsoft.com/en-us/fabric/get-started/known-issues/known-issue-453-data-warehouse-pub...

[jon_clemens]

Thanks!  I had not found this known issue.  Do you (or anyone else) have any insight into plans to address this known issue?  

FYI, in my case, I have tried the known issue.  It will work "one time".  If you then do another incremental upate to the warehouse table, it will once again break.  

[jon_clemens]

Here is an update on this question.  I think I have arrived at the following conclusion.  Shortcuts in a lakehouse pointed to a warehouse table do not like any sql operation that a lakehouse table (sql endpoint) does not allow.  This means that any update or delete performed against the warehouse table, then renders the lakehouse shortcut broken, meaning that the shortcut does not display any of the changes after the update or delete.  The rowcount is wrong and the data is wrong.  And the error that I posted above, or something similar, appears in the dataset refresh history.

This does not seem like the intended behavior, does it?  It would make using shortcuts in the lakehouse pointing to warehouse tables essentially useless.

[mwsmoley]

I have experienced this as well and agree that this does not seem to be the correct behavior. I created a shortcut to a warehouse table to create a directlake dataset and the shortcut has old data and is not updated. Even after deleting the shortcut and recreating it I am still seeing old data. This seems like an oversite and not trully a shortcut but instead a snapshot.