Forum Discussion
Ka13
9 months agoHelper II
Copy Activity Fabric
How to extract the values of rowsCopied and rowsRead from the below Copy activity output in Fabric and store in Fabric Warehouse to maintain the log table of rowsCopied and rowsRead in fabric warehou...
Vinodh247
9 months agoSuper User
use dataflow/notebook after the copy activity to extract metrics from the output JSON. I have copied some example (from web, hope this will show you the path) in a notebook using Pipeline().activity() output, this stores the rowsRead and rowsCopied values into your Fabric warehouse log table.
import json
from notebookutils import mssparkutils
# Get Copy activity output
output = mssparkutils.pipeline.getOutput("CopyActivityName")
data = json.loads(output)
rows_read = data["rowsRead"]
rows_copied = data["rowsCopied"]
# Write to fabric warehouse tbl
spark.sql(f"""
INSERT INTO log_table (activity_name, rows_read, rows_copied, load_time)
VALUES ('CopyActivityName', {rows_read}, {rows_copied}, CURRENT_TIMESTAMP)
""")Make sure to customize/modify to suit your req.