Fabric Lakehouse Shortcut does not refresh Variable Library target after active value set change

I am also facing similar issues.

When I create shortcuts, let's suppose, lakehouse to lakehouse, I use Delegated Identity as that allows me to use Service Principle. The structure or architecture we have comprises of a workspace that brings in data and saves it to Bronze. We call it the Data Engineers Shared workspace. The data could  be sourced from mirrored databases, CSV files generated in Azure VMs, external SQL Server, Azure PostgreSQL, SharePoint. This workspace is therefore consistently running pipelines, notebooks, spark jobs, dataflows, etc. and contains a lot of raw data which we don't expose all at the same time. This workspace is deployed to TEST and PRODUCTION. So, in a way, there are 3 workspaces when taking into account environments but we refer to it as the Data Engineers Shared workspace. This allows us to keep minimal data in dev and test so that engineers can develop and test using actual data and add more data if they like. The Shared Production Workspace has scheduled and streaming live data being saved and the data size is much, much bigger. 

When any department wants to build reports or apps, we create a new set of workspaces for them, for e.g. Finance workspace (also deployed). I won't go into more detail and just mention that Finance Dev reads from Shared Dev, Finance Test reads from Shared Test and Finance Prod reads from Shared Prod. Hence, we arrive at the lakehouse to lakehouse shortcuts. Each shortcut connection uses Delegated Identity via service principle credentials. Shared {env} to Finance {env} get their own connection: "{sharedWS}/{env}/{Lakehouse}" which is helpful as it only allows a "subset" of your data to be visible to those working in the Finance workspaces. We are healthcare tech therefore the data is very sensitive. 

The three connections are saved in the variable library. I then click on Manage Shortcut to assign variable to Target connection. 

Issue 1: The UI is buggy. When I open the Manage Shortcut pane and then click on Edit Target, a new small window opens. Clicking any where other than the Manage Shortcut pane closes the pane and once it's closed, applying the variable doesn't do anything. I've been using my keyboard (tab + spacebar) to find the correct variable from my variable library.

Issue 2: When I look at shortcuts.json of the lakehouse, I do not see a variable assigned to the target connection. When I use variables for Target Item or Target Path, it works fine and the JSON file registers the variables.

This is what the JSON looks like right after setting the variables for Target Item or Target Path. I see no difference in the JSON when I set a variable for target connection. .

{"name": "extract_log",
"path": "/Tables/bronze_temp",
"target": {
"type": "OneLake",
"oneLake": {
"path": "Tables/logs/extract_log",
"itemReference": "$(/**/VL_DE_shared_environment/item_test_1)"}}},

{"name": "extract_config",
"path": "/Tables/bronze_temp",
"target": {
"type": "OneLake",
"oneLake": {
"path": "$(/**/VL_DE_shared_environment/table_test_1)",
"itemId": "3f385684-349c-4fe0-9717-c90771db8e24",
"workspaceId": "00000000-0000-0000-0000-000000000000",
"artifactType": "Lakehouse"}}}

The items, paths and variables in the above JSON are real, but I've made them just to show what the issue looks like.

Issue 3: This one is not that important but is still an issue I've observed. Target Item and Target path shortcuts cannot be set at the same time for the same table shortcut. For some reason it does not register it. So, if I wanted to switch item and path per environment, it won't let me.

This has forced us to changing every shortcut's Connection method to Passthrough Identity instead of Delegated. 

At this point, I just want to know whether Fabric has changed something for a justifiable reason or am I making a mistake. I just want my workflows to resume in a stable manner without having to spend days in POC mode for items and features that are not in Preview phase anymore.

Issue 4: Around the same time this happened, our one and only PostgreSQL mirrored database has stopped functioning. It gives the following error:

Internal error: CDC_ERR_SYS_ONELAKE_COMM_FAILED. Certificate or private key file not found. 

We have tried over 20 different troubleshooting steps except restarting the source DB server as it's a production server. We presume that restarting Fabric mirroring in the Azure PostgreSQL Flexible Server will resolve this, but it isn't something we want to do at the moment.

Feel free to ask any questions.

Hi paulv ,

Thanks you for reaching out to fabric community.

Thanks for testing the feature and letting us know about the issue with the variable library. Since this appears to be a problem, please submit it in the issues forum (https://community.fabric.microsoft.com/t5/Issues/idb-p/Issues) and post it here as well so others facing a similar issue can vote on it and help get it considered for a fix.

Thanks!!

Hi paulv 

this certainly sounds like a bug to me, because when you're clicking on it, it appears to resolve the issue