Power BI REST API Import Dataflow Overwrite error (DuplicatePackageNotFoundError issue in 2023)

[dbeavon3]

Hi PBI Team,

It seems there is an issue with the Power BI REST API when attempting to Import a dataflow with an ImportConflictHandlerMode of "Overwrite".

Steps:

Create the following request (Overwrite)

curl -X POST \
  'https://api.powerbi.com/v1.0/myorg/groups/{{workspaceId}}/imports?datasetDisplayName=model.json&nameConflict=Overwrite' \
  -H 'Accept: */*' \
  -H 'Accept-Encoding: gzip, deflate' \
  -H 'Authorization: Bearer {{authorizationToken}}' \
  -H 'Cache-Control: no-cache' \
  -H 'Connection: keep-alive' \
  -H 'Content-Length: 1522' \
  -H 'Content-Type: application/json' \
  -H 'Host: api.powerbi.com' \
  -H 'Postman-Token: 98346372-e9c3-4c82-afdd-50050023c55f,ad5ca53d-35bf-4258-b487-b9e328e0baa6' \
  -H 'User-Agent: PostmanRuntime/7.15.2' \
  -H 'cache-control: no-cache' \
  -H 'content-type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW' \
  -F =@/{{pathToDataflowJson}}

Expected behavior is for the request to successfully:

- overwrite the dataflow if a dataflow with the same name already exists

However, I often see a received response 404 Not Found with a body of:

{
    "error": {
        "code": "DuplicatePackageNotFoundError",
        "pbi.error": {
            "code": "DuplicatePackageNotFoundError",
            "parameters": {},
            "details": []
        }
    }
}

Other observed behavior:

1. Creating the dataflow first by hitting 

POST https://api.powerbi.com/v1.0/myorg/groups/{{workspaceId}}/imports?datasetDisplayName=model.json

...first ... and then executing either of these will work :

POST  https://api.powerbi.com/v1.0/myorg/groups/{{workspaceId}}/imports?datasetDisplayName=model.json&nameConflict=CreateOrOverwrite

POST https://api.powerbi.com/v1.0/myorg/groups/{{workspaceId}}/imports?datasetDisplayName=model.json&nameConflict=Overwrite 

These work as expected i.e. the dataflow is overwritten.

In other words, I can't overwrite with an ImportConflictHandlerMode of "Overwrite".  I can't do that.... except when I can. 
Please help.

[lbendlin]

You can raise an issue at https://issues.powerbi.com . If you have a Pro license you can consider raising a Pro ticket at https://powerbi.microsoft.com/en-us/support/pro/

[dbeavon3]

Hi @lbendlin 

Do you understand any part of the language from this message (DuplicatePackageNotFoundError)?

What is a "package"?  Why and when is it "duplicated"?  Why would it matter if a duplicated package couldn't be found?

Nothing about this message seems to intersect with my understanding of dataflows.

The only thing I can guess is that it is an internal/technical message that Microsoft never intended to be seen by customers.  Also the language is so strange that I'm guessing that it is related to some internal dependency that the dataflow is relying on (like an azure storage account for the storage of the dataflow results or something).

[lbendlin]

Your guess is as good as mine (wich is that this call requires the dataflow JSON to exist to be overwritable, but sometimes cannot find it) but my suggestion is to ask the Microsoft technical support team to explain it .

[dbeavon3]

Here are two examples of powershell libraries that are buggy, as a result of the problems with Microsoft's underlying REST api (for dataflow imports):

• https://github.com/microsoft/powerbi-powershell/blob/master/examples/dataflows/
• https://github.com/marclelijveld/Power-BI-Automation

I've been working with this Microsoft PG for over a month now and they are NOT willing to fix their bugs so far.  They claim the underlying buggy behavior is "by design" and that using any permutation of "overwrite" for the conflict handler is "not supported".  In other words, the only thing that Microsoft want you to do with this API is upload brand new dataflows.  It seems totally pointless since the first time you upload a dataflow you must configure gateway, data source connections, and refresh schedules, etc.  Not all of these operations appear like they can be automated.  So it defeats the entire purpose of this REST API since the real payout isn't attained until multiple updates can be performed in an automated way by REST (eg. by way of CI/CD deployments for example).

Given the fact that there are two github repros with examples of using "overwrite", it seems possible/likely that this worked in the past and then was subsequently broken.

 I haven't closed my support case yet.  And I did get them to share the fact that other customers have reported the same bug.  There is even a bug tracking ID.  (below)

The title of the bug is "REST API - imports with parameter CreateOrOverwrite throws PackagenotfoundError" and it was created on 5th Sept 2022.

Here is the Bug ID: 875909


Sadly Microsoft doesn't seem to write KB articles about their bugs nowadays (am I a graybeard?)  So I don't think you will find this information in an authoritative format from Microsoft.  Nor is it likely that they will give you a workaround for your occurrences of DuplicatePackageNotFoundError. 

[dbeavon3]

Another thing I should mention is that the error ("DuplicatePackageNotFoundError") is known by another name in the code itself.  I think it is known as a "metadata mismatch exception".  Meaning that if the Power BI service doesn't recognize the updated dataflow definition, when it compares to the original, then there is a mismatch that causes failure.

I tried to ask the PG what characteristics of the dataflow are used for comparing dataflow-metadata during the REST API import operation.  So far they have refused to share that information.  From what I can tell, there are some pretty arbitrary & silly reasons why a modified dataflow would "mismatch" the target.  Here are some possible reasons:

• Number of tables changed, or schema of tables
• Dataflow has changed ownership
  • Possible timing issue - dataflow changes ownership since CSV files were generated.  If dataflow changes ownership prior to refresh then perhaps there is not a metadata mismatch.
• The list of dataflow parameters was changed
• Other arbitrary types of mismatches.

I also want to point out that the team creating the dataflow REST API for imports is very likely to be a different team than the one creating the dataset REST API for imports.  I think there is potentially a high degree of inconsistency because they have different philosophies for building REST API's.  It appears they must cohabitate the "imports" portion of the REST API, for better or worse.  And it appears that the dataflow team doesn't integrate that portion of the API in a very useful or consistent way.

[lbendlin]

Apart from being ugly, what is the other debt of a process where you remove the dataflow and re-upload it each time you want to push out a change? Will it get a new GUID and thus make all the dependent data sinks fall over? Is it unfeasible because of the manual configuration that you mentioned?

How often are you planning to modify the dataflow?

[dbeavon3]

@lbendlin 

I wouldn't mind removing the dataflow from the workspace and re-uploading.  The problem with this is related to the gateway and data sources.  I recall that there is no way to configure that without doing it manually (on the first deployment of the GUID-identified-dataflow)

There is at least a way to re-initialize the schedule.  So we have one piece of the puzzle, but this is the least of my concerns.  Here is the API for updating the refresh schedule:
https://learn.microsoft.com/en-us/rest/api/power-bi/dataflows

If it helps to clarify, here is another blog that discusses the types of issues we encounter when attempting to deploy dataflows:

See https://data-marc.com/2019/10/22/move-dataflows-across-workspaces-with-the-power-bi-rest-api/

>> How often are you planning to modify the dataflow?

We redeploy our dataflows often enough (potentially on a monthly release cycle).  It happens to four different workspaces.  The amount of effort isn't the only problem, but also ensuring reliability and consistency.  Microsoft doesn't seem to understand the concepts of source control, or CI/CD.  As-of today I heard confirmation from CSS that PM's (Matthew Roache, etc) have decided to deprecate this REST API for dataflows and designate it "unsupported".   Oddly that particular PM has a blog encouraging people to use it, and there are samples available as well.   See "overwrite" being used for dataflows here:
https://github.com/microsoft/powerbi-powershell/blob/master/examples/dataflows/

Hope this is clear.  I don't think the team will change the REST API to throw more exceptions for the "unsupported" features.  They will probably just leave them there like an easter egg for people to find.  I don't mind easter eggs, except that it takes a month to get in contact with these folks and discover that they are not supported.