Forum Discussion
Power BI REST API Import Dataflow Overwrite error (DuplicatePackageNotFoundError issue in 2023)
Your guess is as good as mine (wich is that this call requires the dataflow JSON to exist to be overwritable, but sometimes cannot find it) but my suggestion is to ask the Microsoft technical support team to explain it .
Here are two examples of powershell libraries that are buggy, as a result of the problems with Microsoft's underlying REST api (for dataflow imports):
- https://github.com/microsoft/powerbi-powershell/blob/master/examples/dataflows/
- https://github.com/marclelijveld/Power-BI-Automation
I've been working with this Microsoft PG for over a month now and they are NOT willing to fix their bugs so far. They claim the underlying buggy behavior is "by design" and that using any permutation of "overwrite" for the conflict handler is "not supported". In other words, the only thing that Microsoft want you to do with this API is upload brand new dataflows. It seems totally pointless since the first time you upload a dataflow you must configure gateway, data source connections, and refresh schedules, etc. Not all of these operations appear like they can be automated. So it defeats the entire purpose of this REST API since the real payout isn't attained until multiple updates can be performed in an automated way by REST (eg. by way of CI/CD deployments for example).
Given the fact that there are two github repros with examples of using "overwrite", it seems possible/likely that this worked in the past and then was subsequently broken.
I haven't closed my support case yet. And I did get them to share the fact that other customers have reported the same bug. There is even a bug tracking ID. (below)
The title of the bug is "REST API - imports with parameter CreateOrOverwrite throws PackagenotfoundError" and it was created on 5th Sept 2022.
Here is the Bug ID: 875909
Sadly Microsoft doesn't seem to write KB articles about their bugs nowadays (am I a graybeard?) So I don't think you will find this information in an authoritative format from Microsoft. Nor is it likely that they will give you a workaround for your occurrences of DuplicatePackageNotFoundError.
Another thing I should mention is that the error ("DuplicatePackageNotFoundError") is known by another name in the code itself. I think it is known as a "metadata mismatch exception". Meaning that if the Power BI service doesn't recognize the updated dataflow definition, when it compares to the original, then there is a mismatch that causes failure.
I tried to ask the PG what characteristics of the dataflow are used for comparing dataflow-metadata during the REST API import operation. So far they have refused to share that information. From what I can tell, there are some pretty arbitrary & silly reasons why a modified dataflow would "mismatch" the target. Here are some possible reasons:
- Number of tables changed, or schema of tables
- Dataflow has changed ownership
- Possible timing issue - dataflow changes ownership since CSV files were generated. If dataflow changes ownership prior to refresh then perhaps there is not a metadata mismatch.
- The list of dataflow parameters was changed
- Other arbitrary types of mismatches.
I also want to point out that the team creating the dataflow REST API for imports is very likely to be a different team than the one creating the dataset REST API for imports. I think there is potentially a high degree of inconsistency because they have different philosophies for building REST API's. It appears they must cohabitate the "imports" portion of the REST API, for better or worse. And it appears that the dataflow team doesn't integrate that portion of the API in a very useful or consistent way.
Apart from being ugly, what is the other debt of a process where you remove the dataflow and re-upload it each time you want to push out a change? Will it get a new GUID and thus make all the dependent data sinks fall over? Is it unfeasible because of the manual configuration that you mentioned?
How often are you planning to modify the dataflow?
lbendlin
I wouldn't mind removing the dataflow from the workspace and re-uploading. The problem with this is related to the gateway and data sources. I recall that there is no way to configure that without doing it manually (on the first deployment of the GUID-identified-dataflow)
There is at least a way to re-initialize the schedule. So we have one piece of the puzzle, but this is the least of my concerns. Here is the API for updating the refresh schedule:
https://learn.microsoft.com/en-us/rest/api/power-bi/dataflowsIf it helps to clarify, here is another blog that discusses the types of issues we encounter when attempting to deploy dataflows:
See https://data-marc.com/2019/10/22/move-dataflows-across-workspaces-with-the-power-bi-rest-api/
>> How often are you planning to modify the dataflow?
We redeploy our dataflows often enough (potentially on a monthly release cycle). It happens to four different workspaces. The amount of effort isn't the only problem, but also ensuring reliability and consistency. Microsoft doesn't seem to understand the concepts of source control, or CI/CD. As-of today I heard confirmation from CSS that PM's (Matthew Roache, etc) have decided to deprecate this REST API for dataflows and designate it "unsupported". Oddly that particular PM has a blog encouraging people to use it, and there are samples available as well. See "overwrite" being used for dataflows here:
https://github.com/microsoft/powerbi-powershell/blob/master/examples/dataflows/Hope this is clear. I don't think the team will change the REST API to throw more exceptions for the "unsupported" features. They will probably just leave them there like an easter egg for people to find. I don't mind easter eggs, except that it takes a month to get in contact with these folks and discover that they are not supported.