Forum Discussion

Mathan1787's avatar
Mathan1787
Icon for Helper I rankHelper I
3 years ago

Datapipeline-LookUp-Lakehouse

Hi,   Look up activty in data pipeline has query option only for external data store type (for azure sql database) but when i select the delta table i cant able to query. what are the other option ...
pipelines
DennesTorres's avatar
DennesTorres
Icon for Power Participant rankPower Participant
3 years ago

Hi,

Query the lakehouse is possible. The lookup activity has two options : Workspace and External.

The workspace option allow you to query the lakehouse.


What seems to be a limitation, and may be you could add this into the "Ideas" area, is that "Workspace" requires the lakehouse to be in the same workspace as the pipeline. 

In some cases, if you would like to access a different lakehouse, but you have one in the same workspace, maybe you could create a view in the one on the same workspace making a query to the external one?

Kind Regards,

Dennes

  • DennesTorres's avatar
    DennesTorres
    Icon for Power Participant rankPower Participant
    3 years ago

    Hi,

    One possible workaround is to use a SQL Database connection and point to the lakehouse SQL Endpoint. It's not a perfect solution, but probably will work.

     

    Kind Regards,

     

    Dennes

    • ajarora's avatar
      ajarora
      Icon for Microsoft Employee rankMicrosoft Employee
      3 years ago

      Even the cross workspace limitation should be addressable by creating a shortcut to the remote lakehouse, in the current workspace.

      • DennesTorres's avatar
        DennesTorres
        Icon for Power Participant rankPower Participant
        3 years ago
        Hi,

This would only work if there is a lakehouse in both workspaces and it would only work as a source, not as a destination.

I understand that if I would like to break down the pipelines in different workspaces for organization purposes, I couldn't.

Kind Regards,

Dennes