Incremental Refresh Failing

Hi DougalR, 

The scary bit in your message is DataFormat.Error: We reached the end of the buffer. That almost always points to a malformed CSV somewhere (unclosed quote, rogue line break, wrong delimiter/encoding, or an inconsistent column count). The “clusterdetails” part is just the Service wrapper call that bubbled up the error; it’s not the true cause. See: Csv.Document documentation and examples of quoted line-break handling.

• In Power Query, open the Transform Sample File for the folder combine and make it error-tolerant:
  1. In the Source step for the CSV, set QuoteStyle=QuoteStyle.Csv, specify Encoding=65001 (UTF-8), and add ExtraValues=ExtraValues.Ignore. This handles quoted line breaks and extra trailing columns gracefully. Docs - Connector options
  2. Avoid hard-coding a Columns= count unless every file is guaranteed identical.
  3. Wrap the per-file parse in try ... otherwise so you can identify the bad file(s) instead of failing the whole refresh (sample snippet below).
• After you stabilize the CSV parsing, re-enable incremental refresh. Be aware that with file sources (SharePoint/CSV), query folding is limited; IR still creates partitions, but it may have to scan more data than a relational source. If refresh time matters, stage the files into a Dataflow Gen2 or Lakehouse table with a clean schema and an ingestion date, and point your semantic model’s IR at that curated table. IR overview • IR troubleshooting • Text/CSV connector tips

If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.