Datapipeline-LookUp-Lakehouse

[Mathan1787]

Hi,

Look up activty in data pipeline has query option only for external data store type (for azure sql database) but when i select the delta table i cant able to query. what are the other option we could use it to run a query against lakehouse tables in pipeline.

[Mathan1787]

Thanks sql end point worked in this case. it would be great to have a query option for lakehouse.

[GraceGu]

Could you vote on Fabric idea OneLake Connector to work across workspaces in Data Factory in Fabric ?

[DennesTorres]

Hi,

Query the lakehouse is possible. The lookup activity has two options : Workspace and External.

The workspace option allow you to query the lakehouse.

What seems to be a limitation, and may be you could add this into the "Ideas" area, is that "Workspace" requires the lakehouse to be in the same workspace as the pipeline. 

In some cases, if you would like to access a different lakehouse, but you have one in the same workspace, maybe you could create a view in the one on the same workspace making a query to the external one?

Kind Regards,

Dennes

[DennesTorres]

Hi,

One possible workaround is to use a SQL Database connection and point to the lakehouse SQL Endpoint. It's not a perfect solution, but probably will work.

Kind Regards,

Dennes

[ajarora]

Even the cross workspace limitation should be addressable by creating a shortcut to the remote lakehouse, in the current workspace.

[DennesTorres]

Hi,

This would only work if there is a lakehouse in both workspaces and it would only work as a source, not as a destination.

I understand that if I would like to break down the pipelines in different workspaces for organization purposes, I couldn't.

Kind Regards,

Dennes

[ajarora]

You basically have a LH1 in Workspace1, and Pipeline with Lookup activity in Workspace2. Can you create a LH2 in Workspace2, which points to the LH1 in Workspace1, and use lookup activity against LH2 ?