Developing a custom visual but not getting all the rows in the dataset

[angelflight]

I'm building a custom visual. The dataset is a csv file with 18,100 rows and 6 columns. I'm loading the data using a table data row mapping as shown below. When I look at the options.dataViews[0].table.rows.length in the code, there are only 1,000 rows. I have already (I think) applied the dataReductionAlgorithm parameter. There are no segments listed in options.dataViews[0].metadata.segment so fetchMoreData has no effect. I can create a basic table in this report and I see all the rows in the table.

Any help would be appreciated...

"dataViewMappings": [
{
"table": {
"rows": {
"select": [
{
"for": {
"in": "column"
},
"dataReductionAlgorithm": {
"window": {"count": 100000}
}
},
{
"for": {
"in": "value"
},
"dataReductionAlgorithm": {
"window": { "count": 100000 }
}
}
]
}
}
}
]

Stephan

[stefanmmarek]

Hi,

i had a similar issue and found out that depending on the mapping you need to set the supported datalength:

    "dataViewMappings": [
        {
            "table": {
                "rows": {
                    "select": [
                        {
                            "for": {
                                "in": "parameter"
                            }
                        },
                        {
                            "for": {
                                "in": "value"
                            }
                        }
                    ],
                    "dataReductionAlgorithm": {
                        "top": { "count": 5000000 }
                    }
                }, "rowCount": { "supported": {"max": 5000000}}
            }
        }
 

now i always get all rows since it will never exceed 5 mio 😉

[dm-p]

Hi @angelflight,

The dataReductionAlgorithm has a max value of 30,000, even (if you want to use the window algorithm with fetchMoreData) and will likely be reverting to the default of 1,000 as the value is out of range.

I'm not near my dev machine to validate this suggestion for you but I would start by setting this to 30K or lower and seeing if this gets you over 1,000 rows. In the case of using window, the 30K limit still applies for this but the presence of the segment property in your dataView can then be used to load in the additional data.

If this doesn't work for you, let me know and I'll see if I can provide you with a valid configuration when I can get back to my dev machine (likely tomorrow), but I'm hoping that's all you need to get this working.

Good luck!

Daniel

EDIT: completely forgot I made this repo a while ago to demo this exact situation, even down to using the table dataViewMapping... you hopefully might find it useful

[angelflight]

Thank you so much for your response. I had actually already found your repo and had integrated that code, but since there are no segments in the metadata object, it doesn't do the fetch. I tried removing the check to see if the segments exist:

if (options.dataViews[0].metadata.segment) {

When I do that, this.host.fetchMoreData() evaluates to true, but I still only get 1,000 rows.

What's weird is that I only have 18,100 rows. I have tried setting the dataReductionAlgorithm count to 20,000 and also to 100,000 and then, based on your suggestion, to 6,000 to see if it will force the creation of the segments, but it does not.

Any additional thoughts you may have would be appreciated.

[ThibaudGD]

Have you found the solution ? Because I'm facing the same issue