Cross Lakehouse/warehouse connections

[DennesTorres]

Hi,

I noticed some curious behaviours in relation to cross objects queries, the main questions I have about these behaviours are:

Are they issues, which I should register under issues?
Are they missing features, which should be registered as ideas?
Am I missing some technical detail for this to work?
Am I with a wrong point of view about these features?

Everything starts when clicking on the button  "+warehouse" either in a data warehouse or in a lakehouse sql endpoint.

Why the cross lakehouse/datawarehouse/workspace feature is important ? (in my personal opinion)

The domains feature is already in preview. This seems linked to Data Mesh architecture. On Data Mesh architecture, each domain exposes an "API" for consumption by other domains.

"exposing" may be done with permissions over schemas, while "consumption" may be done with cross lakehouse/data warehouse/workspace queries, making the Fabric the perfect environment for Data Mesh implementation, except for the problems/questions I mention below.

First problem: The filters are different if the workspace is set as part of a domain or not.

These are the filters when the workspace is not part of a domain. Mind how it's not possible to filter by domain:

These are the filters when the workspace is part of a domain. From this point, it's possible to filter by domain.

Should the possibility to filter by domain be available either if the current workspace is part of a domain or not? We are looking for other data on the enterprise, maybe we are starting to build something, it may not be set to a domain yet, but we would like to consume other domains.

So, in relation to the initial questions, what about this?

Second Problem: The "All domain" filter is default when we open this window, clicking on "+warehouse", but the list of objects is not from all domain. It requires to click the dropdown, select some other domain and select "All domain" again to really see the correct result.

This is the starting result of the "All domain" when opening the window:

The correct result from "All domains" filter, which only appears after changing the dropdown:

This seems a small UI issue, is it?

Third Problem: The list of domains is not correct. It only shows the domain linked to the current workspace. In my opinion, missing the purpose of this.

The filter, showing only the domain of the current workspace:

The actual list of domains:

Fourth Problem: It's possible to add lakehouses/data warehouses from different workspaces in the same explorer window, but the queries don't work.

The image below shows 3 lakehouses from different workspaces in the same query window:

The intellisense in the query window shows the cross lakehouse/workspace objects:

But the query doesn't work:

I'm not sure if this is an issue or if I'm writting the query wrong. (P.S. I know about the shortcut workaround, I mention it later in the question).

I tried to include the workspace name in the query, the error message changes:

I also tried:

• Opening the explorer window from a data warehouse and starting from a data warehouse
• Making reference to a lakehouse in a workspace without a space in the name

No success.

Known Workaround (to the 4th problem): It's possible to create shortcuts between lakehouses in different workspaces, allowing the cross lakehouse/workspace query. However, this brings many other problems (which I understand as limitations) which I will mention below.

Fifth Problem: Shortcuts can't be done to views. They can only be done to tables.

A possible easy way to create an "API" layer from one domain to another, is to create a schema, create views in the schema and give permissions to users to only access this schema.

However, considering the cross workspace queries are not working and shortcuts can't be done to views, this easy "API Creation" can't be done. 

While the shortcuts can only be done to tables, and all the tables are in the DBO schema, this would open permissions to all the tables across domains.

Another Workaround: We could create a lakehouse only to contain shortcuts to the tables we would like to expose. This lakehouse could work as a cross-domain API layer.

Sixth Problem: The API created with the above workaround would be limited

Using a lakehouse as shortcuts container/API layer would still be limited, because would require us to always expose materialized tables between domains. If we would like to make filters (vertical or horizontal) or something else, we couldn't, because views are not supported in shortcuts.

Seventh Problem: An additional middle object between lakehouses.

Considering the use of a lakehouse as a shortcuts container/API, this creates a middle object between the domains. A query executed in one domain, needs to cross two lakehouses. (Executed on domain A, reaches the "API" lakehouse, reaches the lakehouse with data).

Eighth Problem: Data Warehouses don't support shortcuts

Considering the Data Warehouses don't support shortcuts, they require to create a lakehouse in the same workspace to contain shortcuts to other lakehouses. This lakehouse will work as a bridge. This leads to a scenario with a lot of middle objects:

DW -> Bridge Lakehouse -> API lakehouse -> source lakehouse

All because the cross workspace queries is not working.

Does this excessive number of middle objects affect the performance? If not, how ?

So, back to the main questions, how these 8 problems fit with my initial questions? Issue? Idea? doing wrong? 

Thank you in advance !

Kind Regards,

Dennes

[gtb]

Same , seems pointless not being able to query outside the boundery of a workspace(cross workspace) whilest the data all sits in one lake 

[HimanshuS-msft]

Hello @DennesTorres 
We do appreciate writing in this detail about all the points . I do not think that you are doing anything wrong , I will put this under the issues buckets .
Thanks
HImanshu