topic Re: BUG::PIPELINE::EXPRESSION BUILDER::EDITS NOT SAVED in Data Engineering

BUG::PIPELINE::EXPRESSION BUILDER::EDITS NOT SAVED

Element115 — Wed, 17 Apr 2024 03:05:19 GMT

Using Edge Version 123.0.2420.97 (Official build) (64-bit).

ISSUE:

0__drop a Teams activity onto the design surface.

1__enter some text string in Message section.

2__click View in expression builder and insert System variables wherever.

3__click OK to dismiss the expression builder.

4__the Message text field only show the text you manually entered at step 1; all variables inserted in expression builder are nowhere to be seen.

5__open Expression builder again and only the text from step 1 is shown, all variables inserted at step 2 are gone!

TBH, this is really starting to get on my nerves. It used to work fine. Why all of a sudden this basic functionality is totally gone and broken?!

Re: BUG::PIPELINE::EXPRESSION BUILDER::EDITS NOT SAVED

Element115 — Wed, 17 Apr 2024 03:16:17 GMT

Even crazier, after deleting all the text from the Message field and then opening the expression builder, all the original modifs show up in the expression builder, not in the Message field!!!

I am starting to wonder... I never had this happen before, but before Edge was always running with only 1 user profile. Today, just before starting this Fabric session, I created 2 new profiles, so now I have 3 Edge windows open, each under a different user profile, ie different Microsoft professional account. The profile I use to do all the Fabric work is the one with the same email account as the Fabric user. Perhaps relevant?

Re: BUG::PIPELINE::EXPRESSION BUILDER::EDITS NOT SAVED

Anonymous — Wed, 17 Apr 2024 16:30:43 GMT

Hi @Element115
Thanks for using Fabric Community.
I tried to follow the steps you mentioned above, but did not face any problem.

As per the above screenshot you are trying to concatenate a string and system variable. So I did that in below step.

After clicking on OK, the changes are still there.

If I open the dynamic expression builder again, there is no change.

Hence this is not a bug. Hope this helps. Please let me know if you have any further questions.

Re: BUG::PIPELINE::EXPRESSION BUILDER::EDITS NOT SAVED

Element115 — Wed, 17 Apr 2024 19:27:09 GMT

Yeah maybe so, but have you tried staying connect to Fabric for days on end and leaving the pipeline design surface open for a couple of days? I think this might be explained by some Javascript nonsense going on under the hood client-side.

What browser did you use? I am using Edge Version 123.0.2420.97 (Official build) (64-bit).

Re: BUG::PIPELINE::EXPRESSION BUILDER::EDITS NOT SAVED

Anonymous — Wed, 17 Apr 2024 19:29:01 GMT

@Element115
Even I am using Edge Version 123.0.2420.97 (Official build) (64-bit)

Re: BUG::PIPELINE::EXPRESSION BUILDER::EDITS NOT SAVED

Element115 — Wed, 17 Apr 2024 19:32:46 GMT

Ok don't know what else to tell you. JS cache issue?

Re: BUG::PIPELINE::EXPRESSION BUILDER::EDITS NOT SAVED

Anonymous — Thu, 18 Apr 2024 17:07:13 GMT

Apologies @Element115
I am not able to repro this, hence cannot find the root cause here. Please try again and let me know.

Re: BUG::PIPELINE::EXPRESSION BUILDER::EDITS NOT SAVED

Element115 — Thu, 18 Apr 2024 19:59:39 GMT

Now it is the 3rd time I recreate all the Settings and once done, I get these 4 errors, which is less than all the Teams activity I got, so there was some retention of some Settings. Then, after redoing all 4:

I was able to save and exit the pipeline. I reopened the pipeline and clicked Validate just to make sure all Settings were still preserved, and it was OK. Now I will run it to see if it works or if somehow Settings disappear again.

Re: BUG::PIPELINE::EXPRESSION BUILDER::EDITS NOT SAVED

Anonymous — Thu, 18 Apr 2024 20:44:34 GMT

Hi @Element115
Did you face any issue again? Please let me know.
Thanks

Re: BUG::PIPELINE::EXPRESSION BUILDER::EDITS NOT SAVED

Element115 — Thu, 18 Apr 2024 21:59:29 GMT

atm the pipeline is scheduled and running correctly. For now. So eventually, the Settings editor started to persist the config I was inputing as mentioned in my previous post.

Definitely something is not right with the Javascript client-side code or its interaction with the Edge Javascript execution environment. Unless every single time you input something the Expression builder (for the Teams activity settings) , the code does a roundtrip to the backend and comes back with the wrong session or authentication token or something.

Re: BUG::PIPELINE::EXPRESSION BUILDER::EDITS NOT SAVED

Element115 — Fri, 19 Apr 2024 02:16:47 GMT

Something new happened during the last pipeline run:

Now, that rowsCopied property is exactly the same that is referenced from another Teams activity and the execution of this one succeeded using the exact same code, mind you!

Both activities in the green and red boxes use the same code, and yet, one succeeds while the other fails even though they reference the same property both from a Copy data activity.

Re: BUG::PIPELINE::EXPRESSION BUILDER::EDITS NOT SAVED

Element115 — Fri, 19 Apr 2024 13:10:47 GMT

I figured out what the issue was:

when rows are copied, this is what the JSON output from the Copy data activity looks like:

{ "dataRead": 33, "dataWritten": 1685, "filesWritten": 1, "sourcePeakConnections": 1, "sinkPeakConnections": 1, "rowsRead": 1, "rowsCopied": 1, "copyDuration": 10, "throughput": 0.01, "errors": [], "usedParallelCopies": 1, "executionDetails": [ { "source": { "type": "SqlServer" }, "sink": { "type": "Lakehouse" }, "status": "Succeeded", "start": "4/18/2024, 11:29:47 PM", "duration": 10, "usedParallelCopies": 1, "profile": { "queue": { "status": "Completed", "duration": 0 }, "transfer": { "status": "Completed", "duration": 6, "details": { "readingFromSource": { "type": "SqlServer", "workingDuration": 0, "timeToFirstByte": 0 }, "writingToSink": { "type": "Lakehouse", "workingDuration": 0 } } } }, "detailedDurations": { "queuingDuration": 0, "timeToFirstByte": 0, "transferDuration": 6 } } ], "dataConsistencyVerification": { "VerificationResult": "NotVerified" } }

but when there is no new data to be copied, the rowsCopied is not there in the JSON even though it should be equal to 0 and still be in the JSON:

{ "dataRead": 0, "dataWritten": 0, "filesWritten": 0, "sourcePeakConnections": 1, "sinkPeakConnections": 1, "rowsRead": 0, "copyDuration": 7, "throughput": 0, "errors": [], "usedParallelCopies": 1, "executionDetails": [ { "source": { "type": "SqlServer" }, "sink": { "type": "Lakehouse" }, "status": "Succeeded", "start": "4/19/2024, 1:29:48 AM", "duration": 7, "usedParallelCopies": 1, "profile": { "queue": { "status": "Completed", "duration": 0 }, "transfer": { "status": "Completed", "duration": 4, "details": { "readingFromSource": { "type": "SqlServer", "workingDuration": 0, "timeToFirstByte": 2 }, "writingToSink": { "type": "Lakehouse", "workingDuration": 0 } } } }, "detailedDurations": { "queuingDuration": 0, "timeToFirstByte": 2, "transferDuration": 2 } } ], "dataConsistencyVerification": { "VerificationResult": "NotVerified" } }

So in other words, when rowsRead == 0, rowsCopied property is removed from the output JSON instead of being left there with value 0.

And this causes the flow to break, or rather forces the user to add some code like 'if rowsRead ==0 then... else rowsCopied...', which is less user friendly than just leave rowsCopied in the JSON. For logging purpose, that is fine to just print rowsCopied == 0, it's the point after all.

Re: BUG::PIPELINE::EXPRESSION BUILDER::EDITS NOT SAVED

Anonymous — Fri, 19 Apr 2024 06:04:55 GMT

Thanks for providing the detailed information here @Element115 . I will communicate the same with our team and check if this is a bug. If yes, then we will prioritize this and solve it as soon as possible.
Thank you for the feedback.