Deneb Visual - Vegalite - Bad Plotting with "Page Refresh"

[el-bachaohaql]

Hi all,

I am using a Deneb visual to plot time-based data vertically: Y-axis is time and I have two other fields in the chart plotted independently, each on its own x-axis.

This is a simple line/area chart visual.

Data source: SQL Direct Query; limited to 900 data rows. The data is in the same table

Page Refresh is: Between 5-15 seconds

Platform: PowerBI desktop and PowerBI service

Between and around the time the refresh completes, Deneb displays the points and removes the line/area charts. I have tried a line chart separately, an area chart separately and then I combined them thinking that if one object is not displayed the other will be. 

I have attached a video that shows what is happening.

The following google drive link contains:
-Sample data dataset
-Deneb Code
-Video showing the error referred to above: see second 3 and second 34.

https://drive.google.com/drive/folders/1D9n02VRopAempKRvJDat7qZO-mZlvxDp?usp=sharing 

Not a vegalite expert, but perhaps I'm missing something that enables the Deneb visual to plot data normally.

Another very important thing:

-After some time of running it, memory usage goes off the roof and PowerBi desktop crashes with error "WebView2 out of memory"; also leads to PowerBi service crashing

Thanks in advance

[lbendlin]

How many rows do you have? It's not crashing on my side with the sample data.   Can you do some data aggregation or sampling (like, every 15 minutes?)

[el-bachaohaql]

@lbendlin 

I have millions, downsampled to the few thousands in the file... downsampling further will lead to loss of critical information.

 The behavior I'm seeing is that the memory consumed by powerbi when this visual is in use keeps growing ~50MB after every refresh then after few minutes it reaches the limit and crashes.  

When you say it's not crashing on your end, can you confirm that you simulated this with direct SQL and enabled page auto-refresh? If you did then the issue is definitely related to my environment.

Thanks a lot

[lbendlin]

I have millions, downsampled to the few thousands in the file... downsampling further will lead to loss of critical information.

allow me to challenge you on that.  Keep in mind that a visual only has so many pixels to work with.  Realistically you won't get more than 1000 rows of data rendered anyway.  Since you are using APR you are also likely not expecting end users to interact with the visual.  Maybe a streaming dataset and a dashboard is a better option in that scenario?

[el-bachaohaql]

@lbendlin 

By few thousands I mean mostly less than 5000... I'll let you win that challenge 😎.

I did actully test with 900 rows before posting this, same thing... the memory leaks then that issue with rendering; I wonder if there's a way to get this guy to take a look at it "Daniel Marsh-Patrick".

To answer your suggestion, I was hoping I can build some interactivity into this which theoretically should be possible.

Please explain what you mean by dashboard and a streaming dataset.

Thanks

[lbendlin]

@dm-p  for info

Streaming dataset:  Real-time streaming in Power BI - Power BI | Microsoft Learn

[dm-p]

Some thoughts, as it's the weekend and I'm short on free time.

First thing I'd consider is reducing the number of visuals and looking at concetenating your charts into a single visual (if your dataset can allow it). Having four visuals on the same page that are refreshing as frequently as this has a number of moving parts that could point to the memory issues (query frequency, plus Power BI having to send the data to each visual instance). This would result in a single query to your dataset and a single visual to render.

How does this perform if you take Deneb out of the equation and just use core line charts? If the performance here is good, then we can take a look at what might be wrong with Deneb.

The visual changing from a line to individual dots on updates suggests an ordering issue with y-axis for the the new data coming in. Can you confirm that the data is ordered correctly on each update? I would suggest ensuring using the order channel in your encodings to enforce this.

Another thing I noticed: you can specify the line type in your area mark, which will reduce on layers and encodings. I could remove two layers, by changing the areas as follows:

{
    ...
    {
      "mark": {
        "type": "area",
        "line": {
          "color": "#00DCFE",
          "strokeWidth": 1.5
        },
        "color": "#00DCFE",
        "opacity": 0.37,
        "fill": "#00DCFE"
      },
      "encoding": {
        "x": {
          "field": "Param (1) (1)",
          "type": "quantitative",
          "axis": {
            "labelColor": "#00DCFE",
            "labelPadding": 10,
            "title": null
          }
        }
      }
    },
    {
      "mark": {
        "type": "area",
        "color": "#45AB89",
        "opacity": 0.37,
        "fill": "#45AB89",
        "line": {
          "color": "#45AB89",
          "strokeWidth": 1.5
        }
      },
      "encoding": {
        "x": {
          "field": "Param (1) (2)",
          "type": "quantitative",
          "axis": {
            "labelColor": "#45AB89",
            "labelPadding": 10,
            "title": null
          }
        }
      }
    },
    ...
}

I do not have the resources to reproduce your setup, so if you have further issues, I'd suggest creating an issue in Deneb's GitHub repository with the details, as per instructions on the support page. As I develop and support Deneb in my free time, if you are able to provide access to your environment (in confidence if necessary) so I can debug that would go a great way towards helping diagnose and solve any further issues.

[el-bachaohaql]

@dm-p  thanks for the prompt response and apologies for taking long to respond.

To answer your questions:

-Using one single Deneb visual and not several is leading to the crash. This means I'm only doing one query

-Taking Deneb out or stopping the "Page Refresh" does not lead to PowerBI memory usage increasing until it crashes

-The column is ordered in the data source and in PowerBI. I've also followed your recommendation about force ordering the column but the display issue persists.

-I'll open a ticket on the Github page and can surely grant you access.

Thanks

@lbendlin thanks a lot. Should I mark this as solved or would that be misleading to community members?

[el-bachaohaql]

@lbendlin thanks for taking the time to take a look at this.

The data does not have the same order of magnitude because the intention is to plot data with different magnitudes against the same time axis; say i'm plotting distance in 1000s of meters on one axis and temperature on another axis. An analogy to the chart you shared would be putting one parameter on the primary axis and the second on the secondary axis.

The code I provided does the following:
-Reverses the time axis

-Puts each parameter on its own x-axis

-Displays each parameter in an area chart then embelishes it with a thick line at the end

-Configures the tooltip

Questions are:

-Why is it leaking memory and leading to Webview2 crashing?

-Why does it have this issue with rendering on refreshes?

[lbendlin]

Try without the embellishments first.  Does the below code also result in a crash?

{
  "data": {"name": "dataset"},
  "encoding": {
    "y": {
      "field": "Local Time",
      "type": "temporal",
      "sort":"descending",
      "axis": null
    }
  },
  "layer": [
    {
      "mark": "area",
      "encoding": {
        "x": {
          "field": "Sum of Param2",
          "type": "quantitative",
          "axis":{"title": null }
        }
      }
    },
    {
      "mark": "area",
      "encoding": {
        "x": {
          "field": "Sum of Param1",
          "type": "quantitative",
          "axis":{"title": null }
        }
      }
    }
  ],
  "resolve": {"scale": {"x": "independent"}}
}

[lbendlin]

This seems to work but still throws syntax warnings.

{
  "data": {"name": "dataset"},
  "encoding": {
    "y": {
      "field": "Local Time",
      "type": "temporal",
      "sort": "descending",
      "axis": null
    }
  },
  "layer": [
    {
      "mark": {"type": "area"},
      "encoding": {
        "x": {
          "field": "Param2",
          "type": "quantitative",
          "axis": {"title": null}
        }
      }
    },
    {
      "mark": {"type": "area"},
      "encoding": {
        "x": {
          "field": "Param1",
          "type": "quantitative",
          "axis": {"title": null}
        }
      }
    },
    {
      "mark": {
        "type": "rule",
        "color": "#BAC1CB",
        "strokeWidth": 1
      },
      "encoding": {
        "opacity": {
          "condition": {
            "value": 0.3,
            "selection": "hover"
          },
          "value": 0
        },
        "tooltip": [
          {
            "field": "Local Time",
            "type": "temporal",
            "format": "%Y-%m-%d %H:%M:%S"
          },
          {
            "field": "Param1",
            "type": "quantitative",
            "format": ".2f"
          },
          {
            "field": "Param2",
            "type": "quantitative",
            "format": ".2f"
          }
        ]
      },
      "selection": {
        "hover": {
          "type": "single",
          "fields": ["Local Time"],
          "nearest": true,
          "on": "mouseover",
          "empty": "none",
          "clear": "mouseout"
        }
      }
    }
  ],
  "resolve": {
    "scale": {"x": "independent"}
  }
}

[el-bachaohaql]

@lbendlin Yes it throws these errors but works fine in terms of functionality; perhaps that's what is causing the issue... not sure who can advise.

Before posting online, I have tried the following, note with all of them the visual has the rendering issue when page auto-refresh takes place (shown in the picture):

-One line/area chart: no memory leaks/ no crashing

-Two line/area charts same axis: no memory leaks/ no crashing

-Separate axes for line, area and combined line and area charts;: memory leaks/ crashing

Thanks

[lbendlin]

Your sample data has different orders of magnitude

even with a logarithmic scale it doesn't resemble your screenshots.

Here is the basic Vega Lite code.

{
  "data": {"name": "dataset"},
  "layer":[
  {"mark": "area",
  "encoding": {
    "y": {
      "field": "Local Time",
      "type": "temporal"
    },
    "x": {
      "field": "Sum of Param2",
      "type": "quantitative"
    }
  }},
  {"mark": "area",
  "encoding": {
    "y": {
      "field": "Local Time",
      "type": "temporal"
    },
    "x": {
      "field": "Sum of Param1",
      "type": "quantitative"
    }
  }}
  ]
}

Probably the Y axis sort order needs to be reversed.