topic How to use the get metadata output json in a notebook in Data Engineering

How to use the get metadata output json in a notebook

Bendelyon — Wed, 01 May 2024 14:20:28 GMT

Hello everyones

I need to use the json output folder of a get metadata activity in a notebook.

How Can i do ?

Thanks for your help

Ben

Re: How to use the get metadata output json in a notebook

Expiscornovus — Wed, 01 May 2024 14:37:45 GMT

Hi @Bendelyon,

Do you want to reuse the output in another activity of your pipeline, if so what are you trying to do with the output? Can you share a bit more details about this?

Normally you should be able to reuse the output as dynamic content:

https://learn.microsoft.com/en-us/fabric/data-factory/get-metadata-activity#using-the-get-metadata-activity

Or by using expressions:

@activity('Get Metadata1').output

https://learn.microsoft.com/en-us/azure/data-factory/control-flow-expression-language-functions

Re: How to use the get metadata output json in a notebook

Bendelyon — Wed, 01 May 2024 15:00:44 GMT

Hi @Expiscornovus

I want to use the @activity@activity('Get Metadata1').output.childitems to make a python list and reuse it in a notebook.

The easier gonna be to use all of the activities in the same pipeline.

Do not know if i could do this ?

Thanks

@activity('Get Metadata1').output

Re: How to use the get metadata output json in a notebook

Bendelyon — Wed, 01 May 2024 16:04:23 GMT

For more details

This IS ma schema

Parent folder A1

Child folder a

Csv

Parent folder A2

Child folder a

Csv

Parent folder A3

Child folder a

Csv

Etc

...

My Idea IS to use 'get metadata' to get the name of the folder (A1, A2, A3) and make a python list in a notebook to iterate and consolidate the csv files.

Hope you could understand.

Re: How to use the get metadata output json in a notebook

Anonymous — Thu, 02 May 2024 08:13:45 GMT

Hi @Bendelyon ,

Thanks for using Fabric Community.
Did you tried checking this - Microsoft Spark Utilities (MSSparkUtils) for Fabric - Microsoft Fabric | Microsoft Learn

We can get the list of files present inside lakehouse using Code -

files = mssparkutils.fs.ls("path")

Please let me know if you have further queries.

Re: How to use the get metadata output json in a notebook

Anonymous — Fri, 03 May 2024 10:08:57 GMT

Hello @Bendelyon ,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .
In case if you have any resolution please do share that same with the community as it can be helpful to others .
Otherwise, will respond back with the more details and we will try to help .

Re: How to use the get metadata output json in a notebook

Anonymous — Mon, 06 May 2024 08:15:08 GMT

Hi @Bendelyon ,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .
In case if you have any resolution please do share that same with the community as it can be helpful to others .
Otherwise, will respond back with the more details and we will try to help .

Re: How to use the get metadata output json in a notebook

Anonymous — Mon, 06 May 2024 09:47:31 GMT

Hi @Bendelyon ,

Glad to know that your query got resolved. Please continue using Fabric Community on your further queries.