topic Re: Ingest data in a lakehouse's table with the metadata activity output in Pipelines

Ingest data in a lakehouse's table with the metadata activity output

Toi69___ — Wed, 18 Dec 2024 16:10:34 GMT

Hello,

I have a get metadata activty wicth return this metadata :

{ "lastModified": "2024-12-17T09:10:43Z", "itemName": "Ventes_allemagne.csv", "executionDuration": 1 }

I need to make a table in my lakehouse with two columns : itemName and lastModified with the values from my metadata.

How should i do this ?

Thank you

Re: Ingest data in a lakehouse's table with the metadata activity output

spencer_sa — Thu, 19 Dec 2024 01:35:23 GMT

Possibly not the wisest option but assuming your Get Metadata is returning a single object (which I think it has to to return 'last modified')
1 - Create a notebook that accepts a couple of parameters - item name and last modified - into variables
2 - Set the pipeline to pass these two fields from the Get metadata step into parameters of a Notebook activity
3 - Create a dataframe containing these two variables
4 - Write to a table in Append mode.

The code I used to test - first cell is a parameter cell

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly

Re: Ingest data in a lakehouse's table with the metadata activity output

Anonymous — Thu, 19 Dec 2024 02:07:47 GMT

Hi Toi69___

In the data pipeline, add a new Copy Data activity.

Get Metadata

Copy data

Run

You can get the newmetadata table in lakehouse.

Regards,

Nono Chen

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Re: Ingest data in a lakehouse's table with the metadata activity output

BenLyon — Thu, 19 Dec 2024 08:14:25 GMT

Hello

Great example but in my case i am looking for the metadata from a csv file, not a table.

Do you think you could modified your answer in this case ?

Thank so much

Re: Ingest data in a lakehouse's table with the metadata activity output

Anonymous — Mon, 23 Dec 2024 02:44:08 GMT

Hi BenLyon

You can import the CSV file into Lakehouse, convert it into a table, and then use the above method.

Regards,

Nono Chen

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.