Forum Discussion
Prep Data for AI increasing dataset size and truncating all responses?
Super Userprep data for ai doesn't increase the size of the model. Not in a way that should affect the indexing process.
is it an Import or Direct Query model?
Import, with incremental refresh.
I've already read the post you mention, but the amount of data, tables, relationships, etc. Is exactly the same between the original and the copilot copy, and the original is not getting that error so I don't think the issue is there...
It's import mode with incremental refresh.
I read the post you mentioned but since the underlaying data, the relationships, number of tables, etc. is exactly the same I don't think that is the issue. On top of that, with prep data for AI the second dataset should be even smaller.- R1k91
ok, intetesting scenario. I've only Copilot over Import use cases not copilot over Import+Incremental.
Are you testing both datasets in the same place? I mean both on power bi desktop or both published on the service.
Docs says that indexing for Import happens when you publish it or when you process the model. but in your case you're not processing the whole model, just some partitions (on the service), therefore I'm wondering:
1. will indexing be fired anyway? I hope so.
2. will indexing consider the whole data or just what has been processed in that partition? I guess it's the former but who knows...
I'm wondering if it could be a strange indexing behaviour caused by the refresh policy that doesn't trigger the indexing accordingly.
have you tried to monitor the indexing queries to see if they happen on regular basis according to your refreshing setup?
obviously, if the models are both import+incremental and published with the exact amount of data and the only difference is the prep for ai it's definetly strange but, as webb said, it could depend on how much data copilot thinks it has to index
"
Remember that it only appears when Copilot thinks it needs to index more than 5 million text values, so what is causing it to go over that limit?
"
to me this seems to be a big "it depends" on how copilot interprets the question and how it thinks to resolve it (indexing or not indexing) and with prep for ai maybe it always goes for indexed data that are over 5milion in your case I guess.
Indeed, both datasets are uploaded in service and both use incremental refresh. I haven't checked the indexed queries, I'll have a look now, thanks.
Do you think the issue could be in the metadata? Since I changed measure names to be more legible (i.e. "cnx bkg this" to "cancelled bookings CY") and added descriptions and synonims that were not there before, could that increase in metadata characters be causing the issue?
