Forum Discussion
BUG::PIPELINE::EXPRESSION BUILDER::EDITS NOT SAVED
- 2 years ago
I figured out what the issue was:
when rows are copied, this is what the JSON output from the Copy data activity looks like:
{ "dataRead": 33, "dataWritten": 1685, "filesWritten": 1, "sourcePeakConnections": 1, "sinkPeakConnections": 1, "rowsRead": 1, "rowsCopied": 1, "copyDuration": 10, "throughput": 0.01, "errors": [], "usedParallelCopies": 1, "executionDetails": [ { "source": { "type": "SqlServer" }, "sink": { "type": "Lakehouse" }, "status": "Succeeded", "start": "4/18/2024, 11:29:47 PM", "duration": 10, "usedParallelCopies": 1, "profile": { "queue": { "status": "Completed", "duration": 0 }, "transfer": { "status": "Completed", "duration": 6, "details": { "readingFromSource": { "type": "SqlServer", "workingDuration": 0, "timeToFirstByte": 0 }, "writingToSink": { "type": "Lakehouse", "workingDuration": 0 } } } }, "detailedDurations": { "queuingDuration": 0, "timeToFirstByte": 0, "transferDuration": 6 } } ], "dataConsistencyVerification": { "VerificationResult": "NotVerified" } }but when there is no new data to be copied, the rowsCopied is not there in the JSON even though it should be equal to 0 and still be in the JSON:
{ "dataRead": 0, "dataWritten": 0, "filesWritten": 0, "sourcePeakConnections": 1, "sinkPeakConnections": 1, "rowsRead": 0, "copyDuration": 7, "throughput": 0, "errors": [], "usedParallelCopies": 1, "executionDetails": [ { "source": { "type": "SqlServer" }, "sink": { "type": "Lakehouse" }, "status": "Succeeded", "start": "4/19/2024, 1:29:48 AM", "duration": 7, "usedParallelCopies": 1, "profile": { "queue": { "status": "Completed", "duration": 0 }, "transfer": { "status": "Completed", "duration": 4, "details": { "readingFromSource": { "type": "SqlServer", "workingDuration": 0, "timeToFirstByte": 2 }, "writingToSink": { "type": "Lakehouse", "workingDuration": 0 } } } }, "detailedDurations": { "queuingDuration": 0, "timeToFirstByte": 2, "transferDuration": 2 } } ], "dataConsistencyVerification": { "VerificationResult": "NotVerified" } }So in other words, when rowsRead == 0, rowsCopied property is removed from the output JSON instead of being left there with value 0.
And this causes the flow to break, or rather forces the user to add some code like 'if rowsRead ==0 then... else rowsCopied...', which is less user friendly than just leave rowsCopied in the JSON. For logging purpose, that is fine to just print rowsCopied == 0, it's the point after all.
Hi Element115
Did you face any issue again? Please let me know.
Thanks
I figured out what the issue was:
when rows are copied, this is what the JSON output from the Copy data activity looks like:
{
"dataRead": 33,
"dataWritten": 1685,
"filesWritten": 1,
"sourcePeakConnections": 1,
"sinkPeakConnections": 1,
"rowsRead": 1,
"rowsCopied": 1,
"copyDuration": 10,
"throughput": 0.01,
"errors": [],
"usedParallelCopies": 1,
"executionDetails": [
{
"source": {
"type": "SqlServer"
},
"sink": {
"type": "Lakehouse"
},
"status": "Succeeded",
"start": "4/18/2024, 11:29:47 PM",
"duration": 10,
"usedParallelCopies": 1,
"profile": {
"queue": {
"status": "Completed",
"duration": 0
},
"transfer": {
"status": "Completed",
"duration": 6,
"details": {
"readingFromSource": {
"type": "SqlServer",
"workingDuration": 0,
"timeToFirstByte": 0
},
"writingToSink": {
"type": "Lakehouse",
"workingDuration": 0
}
}
}
},
"detailedDurations": {
"queuingDuration": 0,
"timeToFirstByte": 0,
"transferDuration": 6
}
}
],
"dataConsistencyVerification": {
"VerificationResult": "NotVerified"
}
}
but when there is no new data to be copied, the rowsCopied is not there in the JSON even though it should be equal to 0 and still be in the JSON:
{
"dataRead": 0,
"dataWritten": 0,
"filesWritten": 0,
"sourcePeakConnections": 1,
"sinkPeakConnections": 1,
"rowsRead": 0,
"copyDuration": 7,
"throughput": 0,
"errors": [],
"usedParallelCopies": 1,
"executionDetails": [
{
"source": {
"type": "SqlServer"
},
"sink": {
"type": "Lakehouse"
},
"status": "Succeeded",
"start": "4/19/2024, 1:29:48 AM",
"duration": 7,
"usedParallelCopies": 1,
"profile": {
"queue": {
"status": "Completed",
"duration": 0
},
"transfer": {
"status": "Completed",
"duration": 4,
"details": {
"readingFromSource": {
"type": "SqlServer",
"workingDuration": 0,
"timeToFirstByte": 2
},
"writingToSink": {
"type": "Lakehouse",
"workingDuration": 0
}
}
}
},
"detailedDurations": {
"queuingDuration": 0,
"timeToFirstByte": 2,
"transferDuration": 2
}
}
],
"dataConsistencyVerification": {
"VerificationResult": "NotVerified"
}
}
So in other words, when rowsRead == 0, rowsCopied property is removed from the output JSON instead of being left there with value 0.
And this causes the flow to break, or rather forces the user to add some code like 'if rowsRead ==0 then... else rowsCopied...', which is less user friendly than just leave rowsCopied in the JSON. For logging purpose, that is fine to just print rowsCopied == 0, it's the point after all.
- Anonymous2 years agoNot applicable
Thanks for providing the detailed information here Element115 . I will communicate the same with our team and check if this is a bug. If yes, then we will prioritize this and solve it as soon as possible.
Thank you for the feedback.