Forum Discussion
Ka13
Helper II
10 months agoCopy Activity Fabric
How to extract the values of rowsCopied and rowsRead from the below Copy activity output in Fabric and store in Fabric Warehouse to maintain the log table of rowsCopied and rowsRead in fabric warehou...
Vinodh247
Super User
10 months agouse dataflow/notebook after the copy activity to extract metrics from the output JSON. I have copied some example (from web, hope this will show you the path) in a notebook using Pipeline().activity() output, this stores the rowsRead and rowsCopied values into your Fabric warehouse log table.
import json
from notebookutils import mssparkutils
# Get Copy activity output
output = mssparkutils.pipeline.getOutput("CopyActivityName")
data = json.loads(output)
rows_read = data["rowsRead"]
rows_copied = data["rowsCopied"]
# Write to fabric warehouse tbl
spark.sql(f"""
INSERT INTO log_table (activity_name, rows_read, rows_copied, load_time)
VALUES ('CopyActivityName', {rows_read}, {rows_copied}, CURRENT_TIMESTAMP)
""")Make sure to customize/modify to suit your req.