Forum Discussion
Solved
Store Activity Data into a Table
I am wanting to store the data collected from the Get Metadata activity into a lakehouse table, along with the Copy Data activity output for each table loaded in the ForEach activity. I have tried us...
The following works;
1) Feed the output of the Get MetaData activity into a parameter of a Notebook Activity@string(activity('Get Metadata1').output.childItems)
2) Have a Notebook similiar to the following - attach it to a lakehousemetadata_json = json.loads(metadata)
df = spark.createDataFrame(metadata_json)
df.write.mode('overwrite').saveAsTable('get_metadata')3) When the pipeline is run, this is what gets output;
If this helps, please consider Accepting as a Solution to help others find it more easily.
If you explicitly want the Get Metadata activity data in a table, then it is possible to have a notebook with a string input parameter, feed the activity output into the parameter of the Notebook Activity, and then parse the json in python and write the subsequent output to a delta table using pyspark.