Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
2 years ago

Shortcut table not showing in SQL Endpoint nor Dataset

Hello.  I am able to successfully establish a shortcut from one lakehouse to another.  I am the owner of both and both exist in the same workspace (also owner there).     I can see the data in my l...
admin
data engineering
security
FelixL's avatar
FelixL
Icon for Advocate II rankAdvocate II
2 years ago

I reported this to MS support, who verified the issues and patched it during the November 11 release. After this, it worked fine (again) for me for a few weeks. However, the isse returned last week, and shortcuts to external tables in other lakehouses are now missing in the SQL endpoint again. 

 

I have reported this to MS again (yesterday) and they are now investigating. I will update you here once I hear more. 

 

However, it appears to me that MS does not really know themselves if this should or should not work. I get conflicting communication back from support, sometimes referring to the fact that shourtcuts to external AGLS Gen 2 tables are not supported (at all), and sometimes saying this is a bug and that they will investigate. 

 

OneLake shortcuts - Microsoft Fabric | Microsoft Learn

 

 

My view on this is that it would be very unfotrunate if this is not supported/will not be supported. That would mean that using external tables (in my case DeltaLake tables created in Azure Synapse Analytics) would relegate me to a second grade fabric citizen, not able to use the most basic functionality of shortcuts throughout the business. 

 

Another issue related to this has recently been introduced; if a user tries to create the OneLake shortcut to an external lakehouse table (ADLS Gen 2 DeltaLake table) they must have the underlying cloud data connector shared with them, else they receive an error.

 

This also goes against the whole idea of using the OneLake security model for sharing data - since we really dont want to share the underlying data connectors (containing credentials and root access to data lake) with all business users, since they could then in practice bypass the entire security model by creating their own external shortcuts to whatever data they have access to through the cloud connectors. 

 

This was not the case until Fabric went GA - before GA all users could create shortcuts to tables in all lakeohuses shared with them, regardless if they had access to the underlying data connector. This more correlated with how I see users working with Fabric's security model; sharing of data sohuld be done on Lakehouse / object level - not also on underlying data connectors.

 

This has also been reported as a bug to MS support. 

 

This all goes to say that MS does not seem to really know what will work and what wont work with external shortcuts. And as it now stands the functionalit is at the very least bugged/broken... 

charrington's avatar
charrington
Icon for Advocate V rankAdvocate V
2 years ago

Thanks for the additional info!
The limitation you highlighted makes no sense to me, that article talks in detail on how ADLS shortcuts work, so how can they be not supported?? Looks more like a documentation not up to date from Microsoft. 

Yeah some of the Fabric stuff is still in the early stage, even though it's GA. I'm sure they'll improve it over time, we need to be patient.

  • 12angrymentiger's avatar
    12angrymentiger
    Icon for Advocate III rankAdvocate III
    2 years ago

    That's what I'm thinking. It doesn't make any sense to not support ADLS Gen 2 Delta Shortcuts. That's the whole point of the OneLake shortcuts. 

     

    I was informed by a Microsoft Support Engineer that the deployment for the fix went out last Friday. Anonymous  Mentioned there is a fix in place. Hopefully it either gets rolled out further or there is another issue that they are working on.

  • FelixL's avatar
    FelixL
    Icon for Advocate II rankAdvocate II
    2 years ago

    I just got verified from MS support who checked with the responsible persons in the product team (regarding the limitation of having to share the cloud connection); this is apparently intended behaviour - and wont be fixed. 

     

    I'm very dissapointed by this answer, as it shows that no real thought has been spent on the practical usage of external shortcuts.

     

    As mentioned above, we now find ourselves in a situation where we either have to create one cloud conneciton per external delta lake table, and manage sharing of these connections (thousands of connections) manually to allow users to create shortcuts to the data..

     

    ..or we use one top level cloud connection (as we do now), and share this with the entire organization, and live with the fact that all users in fabric can bypass any and all table security (object level and RLS) by simply mapping their own external shortcuts pointing directly to the ADLS Gen 2 delta lake tables.

     

    We faced similar security issues in Synapse serverless, where all users who had the Synapse workspace managed identity credential shared with them could actually bypass all table security if they knew the storage layer and wrote their own OPENROWSET queries. The response from MS 6 months back when pointing this out was that it wouldnt be fixed. When Fabric got released I was hopeful that the external shortcut support in Lakehouses would fix this issue - but the very same security and usability issues just seem to have been carried over 😞