topic Re: How do we use the Metadata feature for Copy activity feature? in Pipelines

How do we use the Metadata feature for Copy activity feature?

arpost — Fri, 07 Feb 2025 21:07:30 GMT

Greetings, community. I recently noticed the Metadata option on a Copy activity's Destination tab.

I found some Microsoft documentation for this feature (scroll down in this MS doc section), but I cannot figure out how to actually view or pull the metadata applied to a file.

I set this up to write to a Fabric lakehouse. This doesn't seem to show up in any Fabric UIs, and I haven't seemed to be able to tap into this with either a Get metadata activity. Anyone have ideas?

Re: How do we use the Metadata feature for Copy activity feature?

Anonymous — Mon, 10 Feb 2025 01:22:57 GMT

Hi arpost ,

Here are some of my personal thoughts:

1. You can use the Get Metadata activity in a Fabric pipeline to retrieve metadata from your data source. This activity allows you to look up various metadata fields such as column count, structure, and more.

2. To add a Get Metadata activity, create a new pipeline, search for Get Metadata in the Activities pane, and add it to the canvas. Configure the settings by selecting an existing connection or creating a new one, and specify the table or file path.

3. You can read this two documents below:

Solved: data factory get metadata activity - Microsoft Fabric Community

Get Metadata activity - Microsoft Fabric | Microsoft Learn

Best Regards

Yilong Zhou

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Re: How do we use the Metadata feature for Copy activity feature?

arpost — Mon, 10 Feb 2025 15:15:47 GMT

Anonymous, thank you for the reply. I am familiar with the Get metadata activity; the issue is the custom metadata I add doesn't get returned by the Get metadata activity.

Here you can see I added multiple metadata fields to a file I was inserting to Lakehouse.

When I use Get metadata activity, none of that custom metadata appears.

I even tried did another Get metadata inside a ForEach to get the metadata for the indiviual files and add the custom metadata fields:

Still not seeing custom metadata.

That leads to some questions:

Does this mean lakehouse doesn't support this feature in which case why does it show up as an option for use?
Does data factory not yet return all custom metadata from an object?
Is there any way to access that custom metadata?

Re: How do we use the Metadata feature for Copy activity feature?

Anonymous — Thu, 20 Feb 2025 07:28:12 GMT

Hi arpost ,

Here are my personal thoughts on your three questions:

1. Yes, Lakehouse does support custom metadata. However, the support for custom metadata might depend on the specific implementation and tools you are using.

For example, Lakehouse schemas allow you to group tables and manage metadata more effectively.

Here is the document you can read: Lakehouse schemas (Preview) - Microsoft Fabric | Microsoft Learn

2. All custom metadata for an object is not currently returned in Fabric Data Factory.

3. To access custom metadata, you might need to use a different approach or tool. For instance, using the Lakehouse management API could help you retrieve and manage custom metadata more effectively.

Best Regards

Yilong Zhou

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.