Microsoft Fabric - Data Factory - Data Pipeline - No Option to extract set of records from Lakehouse

[Sathish133]

Hi Team,

I am trying to extract the set of records using filter and load it into another Lakehouse/Warehouse using Copy Activity. Because we have the SQL Endpoint where we can write SQL, apply filter and check the data from Lakehouse, sameway I am trying to achieve from Copy Activity.

But Copy Activity not have Query or Filter option in the Source to apply filter and extract the set of records from Lakehouse.

Is this limitation of Microsoft Fabric or Am I missing anything ?

Regards,

Sathish Kumar Mathivanan

[Sathish133]

Hi Team,

Thanks for the approach.

I am trying to create the Azure SQL Database connection, using the basic Username & Password and but it showing invalid credentials.

UserName & Password using for connection is same as using for connecting Microsoft Fabric.

Then tried the option Organizational Account then it connected and I can able to Query the data.

Regards,

Sathish Kumar

[v-veshwara-msft]

Hi @Sathish133 ,

Just checking in to see if you query is resolved and if any responses were helpful. If so, kindly consider marking the helpful reply as 'Accepted Solution' to help others with similar queries. 

Otherwise, feel free to reach out for further assistance.

Thank you.

[v-veshwara-msft]

Hi @Sathish133 ,

Thanks for reaching out to the Microsoft Fabric Community.

Currently, to query a Lakehouse within a Data Pipeline, you can consider the following options:

1. As suggested by @NandanHegde , you can use the Azure SQL Database connector and point it to the SQL Analytics endpoint of your Lakehouse. This enables the query option in the GUI, allowing you to extract a filtered set of records and copy them into another Lakehouse or a Warehouse.

2. Alternatively, you can use a Script activity to run a query against the Lakehouse. To enable query support in the Script activity, again use the Azure SQL Database connector targeting the Lakehouse SQL Analytics endpoint.
   Please follow this link for step by step guidance:Solved: Re: MISSING QUERY FEATURE from Lookup activity - Microsoft Fabric Community

Hope this helps. Please reach out for further assistance.
If this post helps, then please consider to give a kudos and Accept as the solution to help the other members find it more quickly.

Thank you.

[NandanHegde]

You can use the Azure SQL database connection with the SQL analytics end point post which you would get Query option where you can write your SQL query with filter aspect