topic Re: Item snapshots in spark session monitoring are not working properly in Data Engineering

Item snapshots in spark session monitoring are not working properly

Ostrzak — Mon, 18 Nov 2024 21:08:35 GMT

Hi everyone,

I have a pipeline combining several notebooks. Sometimes when I get an error, I would like to access the item snapshot to see exactly what went wrong and where. For some reason though, only the first notebook's snapshot is loading. I can choose others in the left-hand menu, but it still loads the snapshot of the first one.

Could someone help me with it? Is it a known issue maybe? It's a pretty small nuisance for opening the support ticket right now.

Thanks

Re: Item snapshots in spark session monitoring are not working properly

FabianSchut — Mon, 18 Nov 2024 21:41:53 GMT

I've come across this issue with running pipelines with child activities too. For me it worked to open one acitivity in the monitor and change the url to the notebook and activity you want to watch. You can find these ID's often in the monitor. I haven't checked this for notebook snapshots, but it may work there too.

Re: Item snapshots in spark session monitoring are not working properly

Anonymous — Tue, 19 Nov 2024 02:09:15 GMT

Hi @Ostrzak ,

Can you please share some more detail information about this issue? They should help us clarify your scenario and test to troubleshoot.

Regards,

Xiaoxin Sheng

Re: Item snapshots in spark session monitoring are not working properly

jankuc — Wed, 20 Nov 2024 09:58:57 GMT

We are seeing this problem for about a month now. When we run multiple child notebooks from a data pipeline and try to see the resulting snapshots from the monitoring page, we always see the snapshot only of the first notebook.

Re: Item snapshots in spark session monitoring are not working properly

jwinchell40 — Wed, 20 Nov 2024 11:52:04 GMT

@Ostrzak - I moved away from chaining together multiple notebooks in a Pipeline. If I have to run more than 1 or 2 notebooks, I am using a DAG and the RunMultiple() option to orchestrate Notebook processing. That has seemed to help with being able to dig into item snapshots because there is 1 Snapshot for the DAG but then it allows me to navigate to each individual notebook. Seems like it works better or at least in my head it is working better.

Re: Item snapshots in spark session monitoring are not working properly

SivaReddy24680 — Mon, 25 Nov 2024 12:03:43 GMT

Hi @Anonymous

We also came across the same issue. Hope these will help in understading the issue.

We are running multiple notebooks in a single Fabric Pipeline. If one of the notebook is failed and when we try to look at the snapshot to find the where the issue is, it is showing someother table snapshot.

Issue Table(MARC) :

Showing someother tables snapshot(EKKO) :

Note: It is opening the same EKKO snapshot for all the tables irrespective of the table status(success/fail)

Re: Item snapshots in spark session monitoring are not working properly

Ostrzak — Fri, 29 Nov 2024 13:05:17 GMT

Hi, Thank you for answering @jankuc and @jwinchell40

I will raise a ticket with MS Support, as I currently cannot step away from chaining notebooks in a pipeline. Will see how it goes.

Re: Item snapshots in spark session monitoring are not working properly

Anonymous — Mon, 02 Dec 2024 06:53:19 GMT

HI @Ostrzak ,

Any update on this? Did this issue fix or still appeared on your side?

BTW, this issue seems more related to azure pipeline snapshot. If fabric support not helps, you can also report them on azure support side.

Regards,

Xiaoxin Sheng

Re: Item snapshots in spark session monitoring are not working properly

anuyogam — Mon, 16 Dec 2024 20:57:46 GMT

Are you running on high concurrency mode?

Re: Item snapshots in spark session monitoring are not working properly

SivaReddy24680 — Tue, 21 Jan 2025 09:32:06 GMT

It is not working from the activites aswell. I can see some notebooks were repeated for some tables