Dataverse shortcut does not show latest data - it worked once

[DanielAmbler]

Hi,

I am testing the Dataverse shortcut functionality for a single table. 

• I can setup the shortcut fine.
• I can use the same connection to query the dataverse via a pipeline.  (see attached screenshot - 8 records)
• Caching of shortcuts is not enabled on the workspace.
• The 'sync' or 'read' worked fine - once.  (Updated 5 records to 6)
• Both lakehouse and sql endpoint return. ( see attached screenshot - Only 6 records returned)
• This is the same after 2 hours.

I was under the impression the shorcut provided a direct read (unless caching is enabled) so am struggling to understand what is going on here or how to get this fixed.8 records in dataverse6 records in fabric

See attached screenshots

[v-jingzhan-msft]

Hi @DanielAmbler 

I researched some similar threads which reported the similar Dataverse shortcut data latency problem. It seems that there is still some synchronization delay with shortcuts. These delays may be caused by some reasons, leading to that the background metadata sync job is halted. 

One possible mitigation workaround is to manually refresh Lakehouse or shortcuts in the Web Lakehouse Explorer, which can trigger an on-demand refresh of metadata. Then the data shoud be updated. 

Another possible option is to scale up the capacity. This may shorten the latency, but I can't guarantee how short it will be. This requires you to observe and adjust yourself through testing.

I don't find a solid solution to this yet. If you get any insight or solution from the support ticket, can you share it with the community? Thank you in advance. 

Best Regards,
Jing
Community Support Team

[DanielAmbler]

Raised support ticket.

[Franck_SR]

Hello,

Have you solved the problem, I have the same behavior ?

Thank's

Best regards

[DanielAmbler]

Hi @Franck_SR ,  The problem has been solved with some confirmations from MS support.  I havent yet closed the ticket as I still have some internal testing to do but the short version of my findings so far is :-

• Dataverse shortcuts do not work the same as ADLS Gen2 shortcuts as they do not read directly from the underlying dataverse storage.   (This is where I went wrong and by my own admission bypassed some reading under the assumption that it was just another shortcut).  Instead the data is copied to OneLake. 
• The Link To Fabric or Link To Synapse functionality in Dataverse needs to be setup and working correctly, this is what provides the copy functionality.  If you have used this facility with Synapse it works the same.  If this isnt setup  - the shortcut will function but you will see no updates.
• There are two options with the Dataverse sync
  • 1 - Link To Fabric - This will sync all dataverse tables directly with onelake.  It creates a new lakehouse and you cannot select individual tables.
  • 2 - Link to Synapse - For this you dont need a Synapse workspace but you do need a separate ADLS Gen 2.  Using this method you can select individual tables for sync, select monthly or annual partitioning and an incremental feed (flag hard deletes as deleted records).
• When unlinking the Fabric/Synapse link from Dataverse, I would receive an error when trying to set it back up.  This error just went away after approx 24 hours after unlinking.

Hope this helps for now.  Feel free to ask more specifics if needed and if I can help I will.

[Franck_SR]

Thanks @DanielAmbler  for your quick reply !
In the end, what did you do on your side?
Did you destroy and remake the Fabric link?
Or did you go through SynapseLink?

Now you can see the changes via the shortcuts?

[DanielAmbler · Accepted solution]

Hi @Franck_SR 

I need to test only syncing a couple of tables so I (bearing in mind this is a POC system 🙂 )

• Unlinked everything in Dataverse
• Waited 24 hours (or thereabouts - dont have an answer as to why setting the link back up straight away failed several times)
• Setup a storage account
• Activated the Synapse Link for my test tables
• Setup a shortcut to ADLS Gen 2 in Fabric

I still have some testing to do but in principle, this is working for me. 

Before the use case for only a couple of tables presented itself, this also worked for me

• Setup the Fabric Link
• This created a new lakehouse in the target workspace and synchronised every (change enabled) table in Dataverse.

What I did to 'break' was

• Unlinked the Fabric link (see above - my bad :-))

I could still see the Lakehouse and synced data - but did not appreciate that this 'Dataverse shortcut' was actually a synchronisation.  I have not yet retried setting up the fabric link purely due to time pressure on other things.

TBH the way I now plan to structure Fabric for the teams data projects I probably will use the Fabric Link.

Update : Marking this as the solution.  N.B. My original case study was to synchronise 1 table.  After testing I think the fabric link to be a much more robust solution as it utilises the delta change feed.  My setup will be to use shortcuts via a second lakehouse item to surface individual tables.

Cheers

Dan

[Franck_SR]

Thank you very much Daniel 😉 I'll test it

Cheers

Franck

[DanielAmbler]

And interestingly - I cannot do this the other way using the Dataverse -> Fabric link - which worked yesterday. 

[hoanghiepng92]

Hi @DanielAmbler ,
I have a same issue as you. I has linked to Fabric successfully before and now I received this error when trying to un-link the Fabric Link and re-link again. Did you resolve this issue?

[DanielAmbler]

Hi @hoanghiepng92 - see my full answer in this thread.  Regarding the unlink/relink I had to wait for a period of approximately 24 hours before I was able to relink successfully.  I dont know if this is by design or I just got lucky.

Kind regards

[hoanghiepng92]

Dear @DanielAmbler 
In my case, after multiple attempts, the issue has been resolved. This is also solution from MS Support. The said that lots of their customers meet this error and after 2~3 tries the issue will resolve itself.

[DanielAmbler]

Tested the same in a new workspace/lakehouse/dataverse shortcut - get the same result of only 6 records.