Copilot - theory versus practice

[sajLGBI]

There is a lot of focus from Microsoft around what Copilot could do within the context of Power BI and Fabric.
As part of our organisation's assessment we're following the news and posts with interest. We've got a good number of users / consumers who prefer the conversational approach to understanding their business rather than Power BI dashboards and more 'traditional' BI/MI interfaces.

But are Microsoft practicing what they preach?

General Copilot constantly gives incorrect and outdated information around Microsoft's own products.

On the one hand it really shows the importance of good structure within semantic models and the likes but on the other hand it's frustrating and to a point MS shoot themselves in the foot.

[v-kathullac]

Hi @sajLGBI ,

I hope the information provided has addressed your query. If you require any further assistance or have additional questions, please feel free to reachout to fabric community.

Thanks & Regards,

Chaithanya.

[v-kathullac]

Hi @sajLGBI ,

I hope the information provided has addressed your query. If you require any further assistance or have additional questions, please feel free to reachout to fabric community.

Thanks & Regards,

Chaithanya.

[v-kathullac]

Hi @sajLGBI  , 

Thank you for sharing your thoughts. I understand where you're coming from, and I agree that trust is one of the most important factors for AI adoption.

• It's reasonable to expect users to first validate Copilot with questions where they already know the answers before relying on it for more complex scenarios. Accuracy, particularly for Microsoft products and services, plays a significant role in building that confidence.
• You're also right that Copilot's responses are influenced by both the quality and structure of the underlying data and the way questions are phrased. While Microsoft continues to improve the models and product capabilities, user feedback like yours is invaluable in identifying areas where the experience can be improved.
• I appreciate you taking the time to share these observations. I'll ensure your feedback is documented and shared with the appropriate product team, as it highlights important considerations around accuracy, trust, and the overall Copilot experience.

Thanks,

Chaithanya.

[DataTako]

Hi,

Your second point is the one I'd take seriously, and it's not the same as the first.

General Copilot being wrong about Microsoft's own docs is an open-domain grounding problem. Copilot on a semantic model is a text-to-DAX problem. They fail for different reasons, so the first one isn't really evidence about the second. But the conclusion you land on is right anyway, just for a different reason: the risk isn't the model hallucinating a fact, it's the model quietly computing something slightly different from what the user meant, and presenting it with the same confidence as a correct answer. A wrong number with no visual context around it is worse than no number.

What actually helped us, and Riccardo is right that it's real effort:

Don't let the agent write free-form DAX. Restrict it to the measures that already exist in the model. If the report calculates revenue a certain way, the chat answer uses that same measure, and there's no second definition of revenue floating around. That single constraint removed most of the misinterpretation risk for us, because the business logic stays where it always was.

Whitelist the tables and columns it may touch. A narrow, curated model beats a complete one every time here.

Show the work. We return the numbers and let people see how they were produced, so an answer can be checked instead of trusted.

Let it say no. An agent that refuses when it isn't sure is far more useful in a regulated setting than one that always produces something.

For context, we do this at our Saas DataTako, where you can embed reports so viewers can see reports more easiliy without the PPU/PRO license. Built-in Copilot isn't available in that setup at all, so we built a custom agent on the semantic model with Azure AI (and OpenAI of Anthrophic). That forced us to make all of the above explicit rather than hope the built-in experience handles it, which turned out to be the more useful position anyway.

So I'd agree it's not production ready as a switch you flip. As a scoped thing on a curated model, with the agent constrained to existing measures, it holds up. Happy to go into the setup if it's useful for your assessment.

[sajLGBI]

Thanks for that response, it's very interesting.

I've not come across open domain grounding so that was a very interesting quick read.

We've not yet got a specific use case to pursue yet but getting feedback is really helpful.

Thanks again.

[sajLGBI]

Thanks. I appreciate the response.

For me it's all intertwined and not siloed into General copilot and 'flavours' of Copilot.

You start by asking an AI questions that you know the answer to.

Then you move onto asking questions where you don't know the answer but either have experience to know whether the answer is reasonable or can be easily validated via internet searches.

If Copilot regularly gives incorrect answers around Microsoft's products then that's going to be a blocker.

Kind of "if Microsoft can't get Copilot to answer correctly within an area completely within its control then that demonstrates just the complexity and resource requirements of Copilot connecting to internal data"

It's also caveat emptor, let the buyer beware. In my industry (and I suspect many others) there's a regular comment that "users will misinterprete the data". Fixed visuals and context help reduce this. Nuances in how the user asks a question in Copilot and the underlying structure of the data mean it is going to be interesting.

Thanks

[R1k91]

eheh.. this is a biiig discussion I think.

from the official power bi docs: 

"

Some Copilot experiences are generally available, and others are in preview. The report agent Copilot pane available on the right side of reports is generally available. The Power BI agent available as a standalone, full-screen experience accessible from the Power BI left navigation is in preview....

"

this sentence basically means that this feature can change without warning and it may fail or act in a different way.
even Prep-for-AI in power bi desktop is in preview. 

we built POCs on customers integrating Power BI Copilot and Data Agents and results are good as long as your model pefectly fits with best practices, you followed preparation for AI and you filled up both agent and power bi model with comprehensive instructions and sample (the limited amount of text most of the time is not enough) and, probably the most important one, the user exactly knows what he wants to see and masters the domain.

in other words, IMHO, this feature is not production ready yet and opens to something that most of the time companies don't want to do: take a sit with users and carefullly understand needs and double check outcomes after the implementation. 

during an implementation a customer said: "this would add so much effort compared to a standard power bi report" and that's true. on the other side, if you manage to fit copilot and agent 100% to your users' needs you'll lower close to zero the need of a new report because they'll ask everything to copilot.

for now this seems uthopia according to my experience.

[v-kathullac]

Hi @sajLGBI  , 

We understand your observations regarding Copilot and appreciate you taking the time to share them.

A few points that may help clarify the current experience.

• Microsoft offers different Copilot experiences across its products, and their capabilities may vary depending on the underlying data source and service.
• General Microsoft Copilot is designed to answer a broad range of questions using publicly available information and other available sources. As a result, responses may occasionally be outdated, incomplete.
• Copilot in Power BI and Microsoft Fabric is designed to work with your organization's governed data, semantic models, and user permissions rather than relying solely on public information.
• The quality of responses generated by Copilot in Power BI/Fabric depends significantly on the quality of the semantic model, including well-defined relationships, meaningful measure names, business-friendly metadata, and proper data governance.
• Microsoft recommends using well-structured semantic models and curated datasets to achieve the best results when using Copilot in Power BI and Fabric.
• Copilot is an AI-assisted capability and should be considered a productivity aid. Microsoft recommends validating AI-generated responses, especially when they are used for business-critical decisions.
• Microsoft continues to improve Copilot through regular feature updates and model enhancements, so you may notice improvements over time.
• Customer feedback plays an important role in shaping future improvements. If you encounter scenarios where Copilot provides incorrect or misleading responses, we encourage you to submit feedback through the product or share the details with us for further review.

Below is the refrence links :

https://learn.microsoft.com/en-us/power-bi/create-reports/copilot-introduction

https://learn.microsoft.com/en-us/fabric/fundamentals/copilot-fabric-overview

Thanks,

Chaithanya.