Forum Discussion
Unable to Add Dynamic Query Conditions in Get Metadata for Lakehouse Table
I am working with Microsoft Fabric Data Pipelines and using the Get Metadata activity to access a Lakehouse table. Currently, I need to add query conditions (for example, filtering by the latest batch date or remark column) directly in the field list → dynamic content argument.
However, the issue is that:
Direct queries are not supported inside the Get Metadata activity for Lakehouse tables.
When I try to add dynamic expressions (e.g., to filter on latest batch date or conditional remarks), the activity does not allow applying conditions.
Because of this limitation, I cannot dynamically restrict the metadata read to the required subset of rows.
Yes, It's alimitation of Get Metadat activity. It is desiged for retriveing the metdata such as schema.
But not used to query/filter the data rows. So, Whatever query you are using inside the get metadata activity it will not work. That activity looking for static table reference not query result.
As a work around you mau use notebook and inside the notebook you can apply this logic using spark SQL, then store this results in a new table of lakehouse and later you can add get metadata activity.
Please let me know if it helps you.
Thank you!
Did I answer your question? Mark my post as a solution!
Proud to be a Super User!Hi NuwanAthukorala,
Thank you for posting your query in the Microsoft Fabric Community Forum, and thanks to suparnababu8 & tayloramy for sharing valuable insights.
Could you please confirm if your query has been resolved by the provided solutions? This would be helpful for other members who may encounter similar issues.
Thank you for being part of the Microsoft Fabric Community.
3 Replies
Yes, It's alimitation of Get Metadat activity. It is desiged for retriveing the metdata such as schema.
But not used to query/filter the data rows. So, Whatever query you are using inside the get metadata activity it will not work. That activity looking for static table reference not query result.
As a work around you mau use notebook and inside the notebook you can apply this logic using spark SQL, then store this results in a new table of lakehouse and later you can add get metadata activity.
Please let me know if it helps you.
Thank you!
Did I answer your question? Mark my post as a solution!
Proud to be a Super User!Hi NuwanAthukorala,
To add to suparnababu8's great answer, you can also return data from a notebook back to the pipeline that ran it, so you can run the query in a notebook, and then return the data back to the pipeline.If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.
Hi NuwanAthukorala,
Thank you for posting your query in the Microsoft Fabric Community Forum, and thanks to suparnababu8 & tayloramy for sharing valuable insights.
Could you please confirm if your query has been resolved by the provided solutions? This would be helpful for other members who may encounter similar issues.
Thank you for being part of the Microsoft Fabric Community.
