Forum Discussion
Prep Data for AI increasing dataset size and truncating all responses?
It's import mode with incremental refresh.
I read the post you mentioned but since the underlaying data, the relationships, number of tables, etc. is exactly the same I don't think that is the issue. On top of that, with prep data for AI the second dataset should be even smaller.
Super Userok, intetesting scenario. I've only Copilot over Import use cases not copilot over Import+Incremental.
Are you testing both datasets in the same place? I mean both on power bi desktop or both published on the service.
Docs says that indexing for Import happens when you publish it or when you process the model. but in your case you're not processing the whole model, just some partitions (on the service), therefore I'm wondering:
1. will indexing be fired anyway? I hope so.
2. will indexing consider the whole data or just what has been processed in that partition? I guess it's the former but who knows...
I'm wondering if it could be a strange indexing behaviour caused by the refresh policy that doesn't trigger the indexing accordingly.
have you tried to monitor the indexing queries to see if they happen on regular basis according to your refreshing setup?
obviously, if the models are both import+incremental and published with the exact amount of data and the only difference is the prep for ai it's definetly strange but, as webb said, it could depend on how much data copilot thinks it has to index
"
Remember that it only appears when Copilot thinks it needs to index more than 5 million text values, so what is causing it to go over that limit?
"
to me this seems to be a big "it depends" on how copilot interprets the question and how it thinks to resolve it (indexing or not indexing) and with prep for ai maybe it always goes for indexed data that are over 5milion in your case I guess.
Indeed, both datasets are uploaded in service and both use incremental refresh. I haven't checked the indexed queries, I'll have a look now, thanks.
Do you think the issue could be in the metadata? Since I changed measure names to be more legible (i.e. "cnx bkg this" to "cancelled bookings CY") and added descriptions and synonims that were not there before, could that increase in metadata characters be causing the issue?
- R1k91
I don't think so. According to docs and Webb posts only actual data affects indexing not metadata.
you have more than 5MLN distinct text values in your dataset haven't you?
Webb also said that copilots may think to have to scan +5MLN rows.
adding more clarity with AI instructions may trigger copilot to know he has to scan +5MLN rows and if data you're requests are outside the boundaries scanned it simply say "nothing to show".
unfortunately I'm just guessing and I think you should fire a ticket to microsoft.
Thanks, will probably do that.
I understand the 5M limitation and yes, probably over that, but I keep wondering why the original one with the same data returns correct values and doesn't trigger the issue while the copy does. For example if I ask 'Average lead time for Europe for May 2026', on the original dataset I get the correct value (checked myself in connecting to the dataset to do a report) and the prepped dataset returns incorrect values (because of the warning).I'll open a ticket, thanks a lot for your help.