Forum Discussion
Prep Data for AI increasing dataset size and truncating all responses?
Hi everyone, I am hopping to get some advice on how to handle the following issue.
We have a dataset connected to several reports that we want to allow users to use in copilot for sending prompts. To do some testing and clean the dataset I cloned it and did the following in the new one:
- Changed names for tables, fields and measures to make them more clear and standarise them
- Added descriptions to most unhidden fields and measures, and also some synonims
- Hide fields and measures that copilot shouldn't use
On top of that, I also used 'Prep for AI' to add instructions due to some complex logic (e.g. "always use UTC region.timezone unless an explicit timezone is in the prompt" or "use comparative id 1 for current calculations and comparative id 2 for last year calculations"), without these instructions the logic was not correct in the responses. Finally, I also simplified the data schema for copilot, removing some fields.
When testing the new dataset, the logic is always correct but for most answers I get the warning "Copilot can only analyze part of xxx due to its size, which may affect these results." and get empty answers. Now here's the weird thing: If I ask the same to the original dataset it works.
Example of prompts I am trying, that return data (incorrect logic) in the original dataset and come up blank in the new one:
- "Which are the top 100 clients in sales between 1st of May and 16th of May and what region do they belong to?"
- "What are the sales in the last 3 days vs last year?"
So I guess what I am asking is: does prep data for AI increase the size of the dataset or its complexity? how can i remove the 'prep data for AI' configuration?
11 Replies
R1k91Super User
prep data for ai doesn't increase the size of the model. Not in a way that should affect the indexing process.
is it an Import or Direct Query model?
Import, with incremental refresh.
I've already read the post you mention, but the amount of data, tables, relationships, etc. Is exactly the same between the original and the copilot copy, and the original is not getting that error so I don't think the issue is there...
It's import mode with incremental refresh.
I read the post you mentioned but since the underlaying data, the relationships, number of tables, etc. is exactly the same I don't think that is the issue. On top of that, with prep data for AI the second dataset should be even smaller.- R1k91
ok, intetesting scenario. I've only Copilot over Import use cases not copilot over Import+Incremental.
Are you testing both datasets in the same place? I mean both on power bi desktop or both published on the service.
Docs says that indexing for Import happens when you publish it or when you process the model. but in your case you're not processing the whole model, just some partitions (on the service), therefore I'm wondering:
1. will indexing be fired anyway? I hope so.
2. will indexing consider the whole data or just what has been processed in that partition? I guess it's the former but who knows...
I'm wondering if it could be a strange indexing behaviour caused by the refresh policy that doesn't trigger the indexing accordingly.
have you tried to monitor the indexing queries to see if they happen on regular basis according to your refreshing setup?
obviously, if the models are both import+incremental and published with the exact amount of data and the only difference is the prep for ai it's definetly strange but, as webb said, it could depend on how much data copilot thinks it has to index
"
Remember that it only appears when Copilot thinks it needs to index more than 5 million text values, so what is causing it to go over that limit?
"
to me this seems to be a big "it depends" on how copilot interprets the question and how it thinks to resolve it (indexing or not indexing) and with prep for ai maybe it always goes for indexed data that are over 5milion in your case I guess.
- GilbertQ
Hi tdponcelas
The only thing that I could think of that is causing this issue is the prep for AI was done when those previous columns and parts of your data were there. So what I would recommend doing is to remove the prep for AI and recreate it again and see if that resolves your issue.
Thanks for your reply, I don't see where to delete de 'prep data' config, do you know how i can remove it to try again?
