Forum Discussion
Power BI REST API Import Dataflow Overwrite error (DuplicatePackageNotFoundError issue in 2023)
Hi lbendlin
Do you understand any part of the language from this message (DuplicatePackageNotFoundError)?
What is a "package"? Why and when is it "duplicated"? Why would it matter if a duplicated package couldn't be found?
Nothing about this message seems to intersect with my understanding of dataflows.
The only thing I can guess is that it is an internal/technical message that Microsoft never intended to be seen by customers. Also the language is so strange that I'm guessing that it is related to some internal dependency that the dataflow is relying on (like an azure storage account for the storage of the dataflow results or something).
Your guess is as good as mine (wich is that this call requires the dataflow JSON to exist to be overwritable, but sometimes cannot find it) but my suggestion is to ask the Microsoft technical support team to explain it .
Here are two examples of powershell libraries that are buggy, as a result of the problems with Microsoft's underlying REST api (for dataflow imports):
- https://github.com/microsoft/powerbi-powershell/blob/master/examples/dataflows/
- https://github.com/marclelijveld/Power-BI-Automation
I've been working with this Microsoft PG for over a month now and they are NOT willing to fix their bugs so far. They claim the underlying buggy behavior is "by design" and that using any permutation of "overwrite" for the conflict handler is "not supported". In other words, the only thing that Microsoft want you to do with this API is upload brand new dataflows. It seems totally pointless since the first time you upload a dataflow you must configure gateway, data source connections, and refresh schedules, etc. Not all of these operations appear like they can be automated. So it defeats the entire purpose of this REST API since the real payout isn't attained until multiple updates can be performed in an automated way by REST (eg. by way of CI/CD deployments for example).
Given the fact that there are two github repros with examples of using "overwrite", it seems possible/likely that this worked in the past and then was subsequently broken.
I haven't closed my support case yet. And I did get them to share the fact that other customers have reported the same bug. There is even a bug tracking ID. (below)
The title of the bug is "REST API - imports with parameter CreateOrOverwrite throws PackagenotfoundError" and it was created on 5th Sept 2022.
Here is the Bug ID: 875909
Sadly Microsoft doesn't seem to write KB articles about their bugs nowadays (am I a graybeard?) So I don't think you will find this information in an authoritative format from Microsoft. Nor is it likely that they will give you a workaround for your occurrences of DuplicatePackageNotFoundError.Another thing I should mention is that the error ("DuplicatePackageNotFoundError") is known by another name in the code itself. I think it is known as a "metadata mismatch exception". Meaning that if the Power BI service doesn't recognize the updated dataflow definition, when it compares to the original, then there is a mismatch that causes failure.
I tried to ask the PG what characteristics of the dataflow are used for comparing dataflow-metadata during the REST API import operation. So far they have refused to share that information. From what I can tell, there are some pretty arbitrary & silly reasons why a modified dataflow would "mismatch" the target. Here are some possible reasons:
- Number of tables changed, or schema of tables
- Dataflow has changed ownership
- Possible timing issue - dataflow changes ownership since CSV files were generated. If dataflow changes ownership prior to refresh then perhaps there is not a metadata mismatch.
- The list of dataflow parameters was changed
- Other arbitrary types of mismatches.
I also want to point out that the team creating the dataflow REST API for imports is very likely to be a different team than the one creating the dataset REST API for imports. I think there is potentially a high degree of inconsistency because they have different philosophies for building REST API's. It appears they must cohabitate the "imports" portion of the REST API, for better or worse. And it appears that the dataflow team doesn't integrate that portion of the API in a very useful or consistent way.
Apart from being ugly, what is the other debt of a process where you remove the dataflow and re-upload it each time you want to push out a change? Will it get a new GUID and thus make all the dependent data sinks fall over? Is it unfeasible because of the manual configuration that you mentioned?
How often are you planning to modify the dataflow?