Copy Activity Fabric

use dataflow/notebook after the copy activity to extract metrics from the output JSON. I have copied some example (from web, hope this will show you the path) in a notebook using Pipeline().activity() output, this stores the rowsRead and rowsCopied values into your Fabric warehouse log table.

import json
from notebookutils import mssparkutils

# Get Copy activity output
output = mssparkutils.pipeline.getOutput("CopyActivityName")
data = json.loads(output)

rows_read = data["rowsRead"]
rows_copied = data["rowsCopied"]

# Write to fabric warehouse tbl
spark.sql(f"""
INSERT INTO log_table (activity_name, rows_read, rows_copied, load_time)
VALUES ('CopyActivityName', {rows_read}, {rows_copied}, CURRENT_TIMESTAMP)
""")

Make sure to customize/modify to suit your req.

Hi Ka13, 

As mentioned in the Fabric Discord, you can add a script task that references the output of a previous step. 

Something along the lines of: @activity(Copy-activity).output.rowsRead should work for you

If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution. 

Ka13 

Create a Log Table in Fabric Warehouse

CREATE TABLE CopyActivityLog ( ActivityDate DATETIME, RowsRead INT, RowsCopied INT );

2. Use a Data Pipeline with Script Activity

• After the Copy activity, add a Script activity in the pipeline.
• Use dynamic content to pass rowsRead and rowsCopied from the Copy activity output.

3. Insert Values Using SQL Script

Example SQL script using pipeline parameters:

INSERT INTO CopyActivityLog (ActivityDate, RowsRead, RowsCopied) VALUES (GETDATE(), @{activity('CopyData').output.rowsRead}, @{activity('CopyData').output.rowsCopied});



Replace 'CopyData' with the actual name of your Copy activity.

4. Validate Pipeline Execution

• Run the pipeline and check the log table.
• Ensure the Script activity executes after the Copy activity.

Hi Ka13, 

Let's go through this step by step. 

First, I created a metadata table in my warehouse: 

CREATE TABLE metadata  (
runid UNIQUEIDENTIFIER, 
rowsRead int, 
rowsCopied int, 
CopyDuration int

)

Next I add a script activity after the copy activity: 

Next, in the expression builder, I will build the expression: 

@concat('INSERT INTO dbo.metadata (runid, rowsRead, rowsCopied, CopyDuration) VALUES
(''',pipeline().RunId,''', '''
,activity('Copy data1').output.rowsRead, ''','''
,activity('Copy data1').output.rowsCopied, ''','''
,activity('Copy data1').output.copyDuration, ''');' )

THis runs successfully: 

 and wehn I look at the warehouse, I see my run recorded: 

Hope this helps. 

If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.