topic Lookup Activity Refencing file in different workspace in Pipelines

Lookup Activity Refencing file in different workspace

MartinMason — Thu, 25 Apr 2024 22:49:41 GMT

We have a medallion architecture. Our bronze layer consists of two workspaces for the different source systems being combined. We have a third workspace for the gold layer combining conformed data from the two separate source systems. We would like to create a single Master pipeline that orchestrates the entire ETL process in the Gold workspace. However, a Pipeline Activity cannot call a pipeline in a different workspace. So no go there. So we thought we'd move the master pipelines from the Bronze workspaces to the Gold workspace to workaround that limitation. Now when a Lookup activity is executed for the bronze master pipeline which reads a configuration file from the bronze lakehouse, it blows up. Is this a temporary glitch? All lakehouses in our tenant are listed as options in the initial configuration of the Lookup Activity so one would think that you'd be able to reference a lakehouse file.

Re: Lookup Activity Refencing file in different workspace

Anonymous — Fri, 26 Apr 2024 13:19:26 GMT

Hi MartinMason ,

Thanks for using Fabric Community.
Can you please check this similar thread: execute activities in another workspace? - Microsoft Fabric Community
I hope you might some insights while looking this thread.

Please let me know if you have further queries.

Re: Lookup Activity Refencing file in different workspace

MartinMason — Fri, 26 Apr 2024 23:40:59 GMT

Not useful in my case. Microsoft needs to gather around and get on the same page. Best practices as I understand them, is that medallion layers should be placed in separate workspaces. Bronze layer, or layers in my case, and gold in different workspaces.

However, Pipelines do not enable that to be accomplished, at least by orchestrating load of Bronze layer followed by Gold in a single master pipeline. When creating an Invoke Pipeline activity, only Pipelines in the current workspace can be referenced. Come on, Pipeline team, get your #### together.

Workaround, have Pipelines all be in the same workspace and call Lakehouses in other workspaces to populate Bronze. Though a Copy Data activity can reference a Lakehouse in a different workspace, and Validation will succeed, execution of that Copy Data activity will fail with the following.

Failure happened on 'destination' side. ErrorCode=LakehouseOperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Lakehouse operation failed for: Operation returned an invalid status code 'NotFound'. Workspace:

I'm interpreting this as the Copy Data activity is looking for the specified Lakehouse in the current workspace as nowhere in the Copy Data activity is a workspace able to be specified. This "feature" of the Copy Data activity is truly an epic fail.

Re: Lookup Activity Refencing file in different workspace

Anonymous — Tue, 30 Apr 2024 09:11:44 GMT

Hi MartinMason ,

I would like to understand how are you connecting to the lakehouse present in different workspace via Copy Activity? If you can share these details, I can guide you better!

Re: Lookup Activity Refencing file in different workspace

Anonymous — Thu, 02 May 2024 09:20:40 GMT

Hi MartinMason ,

We haven’t heard from you on the last response and was just checking back to see if you got a chance to check my last response.

Thanks

Re: Lookup Activity Refencing file in different workspace

Anonymous — Mon, 06 May 2024 09:27:14 GMT

Hi MartinMason ,

We haven’t heard from you on the last response and was just checking back to see if you got a chance to check my last response.

Thanks