Forum Discussion
Memorable MemberBUG::PIPELINE::EXPRESSION BUILDER::EDITS NOT SAVED
I figured out what the issue was:
when rows are copied, this is what the JSON output from the Copy data activity looks like:
{ "dataRead": 33, "dataWritten": 1685, "filesWritten": 1, "sourcePeakConnections": 1, "sinkPeakConnections": 1, "rowsRead": 1, "rowsCopied": 1, "copyDuration": 10, "throughput": 0.01, "errors": [], "usedParallelCopies": 1, "executionDetails": [ { "source": { "type": "SqlServer" }, "sink": { "type": "Lakehouse" }, "status": "Succeeded", "start": "4/18/2024, 11:29:47 PM", "duration": 10, "usedParallelCopies": 1, "profile": { "queue": { "status": "Completed", "duration": 0 }, "transfer": { "status": "Completed", "duration": 6, "details": { "readingFromSource": { "type": "SqlServer", "workingDuration": 0, "timeToFirstByte": 0 }, "writingToSink": { "type": "Lakehouse", "workingDuration": 0 } } } }, "detailedDurations": { "queuingDuration": 0, "timeToFirstByte": 0, "transferDuration": 6 } } ], "dataConsistencyVerification": { "VerificationResult": "NotVerified" } }but when there is no new data to be copied, the rowsCopied is not there in the JSON even though it should be equal to 0 and still be in the JSON:
{ "dataRead": 0, "dataWritten": 0, "filesWritten": 0, "sourcePeakConnections": 1, "sinkPeakConnections": 1, "rowsRead": 0, "copyDuration": 7, "throughput": 0, "errors": [], "usedParallelCopies": 1, "executionDetails": [ { "source": { "type": "SqlServer" }, "sink": { "type": "Lakehouse" }, "status": "Succeeded", "start": "4/19/2024, 1:29:48 AM", "duration": 7, "usedParallelCopies": 1, "profile": { "queue": { "status": "Completed", "duration": 0 }, "transfer": { "status": "Completed", "duration": 4, "details": { "readingFromSource": { "type": "SqlServer", "workingDuration": 0, "timeToFirstByte": 2 }, "writingToSink": { "type": "Lakehouse", "workingDuration": 0 } } } }, "detailedDurations": { "queuingDuration": 0, "timeToFirstByte": 2, "transferDuration": 2 } } ], "dataConsistencyVerification": { "VerificationResult": "NotVerified" } }So in other words, when rowsRead == 0, rowsCopied property is removed from the output JSON instead of being left there with value 0.
And this causes the flow to break, or rather forces the user to add some code like 'if rowsRead ==0 then... else rowsCopied...', which is less user friendly than just leave rowsCopied in the JSON. For logging purpose, that is fine to just print rowsCopied == 0, it's the point after all.
Memorable MemberNow it is the 3rd time I recreate all the Settings and once done, I get these 4 errors, which is less than all the Teams activity I got, so there was some retention of some Settings. Then, after redoing all 4:
I was able to save and exit the pipeline. I reopened the pipeline and clicked Validate just to make sure all Settings were still preserved, and it was OK. Now I will run it to see if it works or if somehow Settings disappear again.
Hi Element115
Did you face any issue again? Please let me know.
Thanks
- Element115
atm the pipeline is scheduled and running correctly. For now. So eventually, the Settings editor started to persist the config I was inputing as mentioned in my previous post.
Definitely something is not right with the Javascript client-side code or its interaction with the Edge Javascript execution environment. Unless every single time you input something the Expression builder (for the Teams activity settings) , the code does a roundtrip to the backend and comes back with the wrong session or authentication token or something.
- Element115
I figured out what the issue was:
when rows are copied, this is what the JSON output from the Copy data activity looks like:
{ "dataRead": 33, "dataWritten": 1685, "filesWritten": 1, "sourcePeakConnections": 1, "sinkPeakConnections": 1, "rowsRead": 1, "rowsCopied": 1, "copyDuration": 10, "throughput": 0.01, "errors": [], "usedParallelCopies": 1, "executionDetails": [ { "source": { "type": "SqlServer" }, "sink": { "type": "Lakehouse" }, "status": "Succeeded", "start": "4/18/2024, 11:29:47 PM", "duration": 10, "usedParallelCopies": 1, "profile": { "queue": { "status": "Completed", "duration": 0 }, "transfer": { "status": "Completed", "duration": 6, "details": { "readingFromSource": { "type": "SqlServer", "workingDuration": 0, "timeToFirstByte": 0 }, "writingToSink": { "type": "Lakehouse", "workingDuration": 0 } } } }, "detailedDurations": { "queuingDuration": 0, "timeToFirstByte": 0, "transferDuration": 6 } } ], "dataConsistencyVerification": { "VerificationResult": "NotVerified" } }but when there is no new data to be copied, the rowsCopied is not there in the JSON even though it should be equal to 0 and still be in the JSON:
{ "dataRead": 0, "dataWritten": 0, "filesWritten": 0, "sourcePeakConnections": 1, "sinkPeakConnections": 1, "rowsRead": 0, "copyDuration": 7, "throughput": 0, "errors": [], "usedParallelCopies": 1, "executionDetails": [ { "source": { "type": "SqlServer" }, "sink": { "type": "Lakehouse" }, "status": "Succeeded", "start": "4/19/2024, 1:29:48 AM", "duration": 7, "usedParallelCopies": 1, "profile": { "queue": { "status": "Completed", "duration": 0 }, "transfer": { "status": "Completed", "duration": 4, "details": { "readingFromSource": { "type": "SqlServer", "workingDuration": 0, "timeToFirstByte": 2 }, "writingToSink": { "type": "Lakehouse", "workingDuration": 0 } } } }, "detailedDurations": { "queuingDuration": 0, "timeToFirstByte": 2, "transferDuration": 2 } } ], "dataConsistencyVerification": { "VerificationResult": "NotVerified" } }So in other words, when rowsRead == 0, rowsCopied property is removed from the output JSON instead of being left there with value 0.
And this causes the flow to break, or rather forces the user to add some code like 'if rowsRead ==0 then... else rowsCopied...', which is less user friendly than just leave rowsCopied in the JSON. For logging purpose, that is fine to just print rowsCopied == 0, it's the point after all.
Thanks for providing the detailed information here Element115 . I will communicate the same with our team and check if this is a bug. If yes, then we will prioritize this and solve it as soon as possible.
Thank you for the feedback.- Element115
Something new happened during the last pipeline run:
Now, that rowsCopied property is exactly the same that is referenced from another Teams activity and the execution of this one succeeded using the exact same code, mind you!
Both activities in the green and red boxes use the same code, and yet, one succeeds while the other fails even though they reference the same property both from a Copy data activity.