topic Re: Datapipeline-LookUp-Lakehouse in Pipelines

Datapipeline-LookUp-Lakehouse

Mathan1787 — Fri, 11 Aug 2023 10:31:00 GMT

Hi,

Look up activty in data pipeline has query option only for external data store type (for azure sql database) but when i select the delta table i cant able to query. what are the other option we could use it to run a query against lakehouse tables in pipeline.

Re: Datapipeline-LookUp-Lakehouse

DennesTorres — Fri, 11 Aug 2023 21:06:30 GMT

Hi,

Query the lakehouse is possible. The lookup activity has two options : Workspace and External.

The workspace option allow you to query the lakehouse.

What seems to be a limitation, and may be you could add this into the "Ideas" area, is that "Workspace" requires the lakehouse to be in the same workspace as the pipeline.

In some cases, if you would like to access a different lakehouse, but you have one in the same workspace, maybe you could create a view in the one on the same workspace making a query to the external one?

Kind Regards,

Dennes

Re: Datapipeline-LookUp-Lakehouse

DennesTorres — Sun, 13 Aug 2023 10:26:09 GMT

Hi,

One possible workaround is to use a SQL Database connection and point to the lakehouse SQL Endpoint. It's not a perfect solution, but probably will work.

Kind Regards,

Dennes

Re: Datapipeline-LookUp-Lakehouse

GraceGu — Mon, 14 Aug 2023 03:29:42 GMT

Could you vote on Fabric idea OneLake Connector to work across workspaces in Data Factory in Fabric ?

Re: Datapipeline-LookUp-Lakehouse

Mathan1787 — Mon, 14 Aug 2023 07:31:25 GMT

Thanks sql end point worked in this case. it would be great to have a query option for lakehouse.

Re: Datapipeline-LookUp-Lakehouse

ajarora — Mon, 14 Aug 2023 20:27:40 GMT

Even the cross workspace limitation should be addressable by creating a shortcut to the remote lakehouse, in the current workspace.

Re: Datapipeline-LookUp-Lakehouse

DennesTorres — Tue, 15 Aug 2023 13:07:43 GMT

Hi, This would only work if there is a lakehouse in both workspaces and it would only work as a source, not as a destination. I understand that if I would like to break down the pipelines in different workspaces for organization purposes, I couldn't. Kind Regards, Dennes

Re: Datapipeline-LookUp-Lakehouse

ajarora — Tue, 15 Aug 2023 23:10:10 GMT

You basically have a LH1 in Workspace1, and Pipeline with Lookup activity in Workspace2. Can you create a LH2 in Workspace2, which points to the LH1 in Workspace1, and use lookup activity against LH2 ?