Power BI REST API Import Dataflow Overwrite error (DuplicatePackageNotFoundError issue in 2023)

Question

Hi PBI Team,&nbsp;It seems there is an issue with the Power BI REST API when attempting to Import a dataflow with an&nbsp;ImportConflictHandlerMode&nbsp;of "Overwrite".&nbsp;Steps:Create the following request (Overwrite)&nbsp;curl -X POST \
  'https://api.powerbi.com/v1.0/myorg/groups/{{workspaceId}}/imports?datasetDisplayName=model.json&amp;nameConflict=Overwrite' \
  -H 'Accept: */*' \
  -H 'Accept-Encoding: gzip, deflate' \
  -H 'Authorization: Bearer {{authorizationToken}}' \
  -H 'Cache-Control: no-cache' \
  -H 'Connection: keep-alive' \
  -H 'Content-Length: 1522' \
  -H 'Content-Type: application/json' \
  -H 'Host: api.powerbi.com' \
  -H 'Postman-Token: 98346372-e9c3-4c82-afdd-50050023c55f,ad5ca53d-35bf-4258-b487-b9e328e0baa6' \
  -H 'User-Agent: PostmanRuntime/7.15.2' \
  -H 'cache-control: no-cache' \
  -H 'content-type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW' \
  -F =@/{{pathToDataflowJson}}Expected behavior is for the request to successfully:- overwrite the dataflow if a dataflow with the same name already exists&nbsp;&nbsp;However, I often see a received response 404 Not Found&nbsp;with a body of:{
    "error": {
        "code": "DuplicatePackageNotFoundError",
        "pbi.error": {
            "code": "DuplicatePackageNotFoundError",
            "parameters": {},
            "details": []
        }
    }
}&nbsp;&nbsp;Other observed behavior:&nbsp;&nbsp;1. Creating the dataflow first by hitting&nbsp;POST https://api.powerbi.com/v1.0/myorg/groups/{{workspaceId}}/imports?datasetDisplayName=model.json...first ... and then executing either of these will work :POST  https://api.powerbi.com/v1.0/myorg/groups/{{workspaceId}}/imports?datasetDisplayName=model.json&amp;nameConflict=CreateOrOverwritePOST https://api.powerbi.com/v1.0/myorg/groups/{{workspaceId}}/imports?datasetDisplayName=model.json&amp;nameConflict=Overwrite These&nbsp;work as expected&nbsp;i.e. the dataflow is overwritten.&nbsp;&nbsp;In other words, I can't overwrite with&nbsp;an&nbsp;ImportConflictHandlerMode&nbsp;of "Overwrite".&nbsp; I can't do that.... except when I can.&nbsp;Please help.&nbsp;&nbsp;

lbendlin · Answer

You can raise an issue at https://issues.powerbi.com . If you have a Pro license you can consider raising a Pro ticket at https://powerbi.microsoft.com/en-us/support/pro/

dbeavon3 · Answer

Hi&nbsp;lbendlin&nbsp;Do you understand any part of the language from this message (DuplicatePackageNotFoundError)?&nbsp;What is a "package"?&nbsp; Why and when is it "duplicated"?&nbsp; Why would it matter if a duplicated package couldn't be found?&nbsp;Nothing about this message seems to intersect with my understanding of dataflows.&nbsp;The only thing I can guess is that it is an internal/technical message that Microsoft never intended to be seen by customers.&nbsp; Also the language is so strange that I'm guessing that it is related to some internal dependency that the dataflow is relying on (like an azure storage account for the storage of the dataflow results or something).&nbsp;

lbendlin · Answer

Your guess is as good as mine (wich is that this call requires the dataflow JSON to exist to be overwritable, but sometimes cannot find it) but my suggestion is to ask the Microsoft technical support team to explain it .

dbeavon3 · Answer

Here are two examples of powershell libraries that are buggy, as a result of the problems with Microsoft's underlying REST api (for dataflow imports):&nbsp;https://github.com/microsoft/powerbi-powershell/blob/master/examples/dataflows/https://github.com/marclelijveld/Power-BI-Automation&nbsp;I've been working with this Microsoft PG for over a month now and they are NOT willing to fix their bugs so far.&nbsp; They claim the underlying buggy behavior is "by design" and that using any permutation of "overwrite" for the conflict handler is "not supported".&nbsp; In other words, the only thing that Microsoft want you to do with this API is upload brand new dataflows.&nbsp; It seems totally pointless since the first time you upload a dataflow you must configure gateway, data source connections, and refresh schedules, etc.&nbsp; Not all of these operations appear like they can be automated.&nbsp; So it defeats the entire purpose of this REST API since the real payout isn't attained until multiple updates can be performed in an automated way by REST (eg. by way of CI/CD deployments for example).&nbsp;Given the fact that there are two github repros with examples of using "overwrite", it seems possible/likely that this worked in the past and then was subsequently broken.&nbsp;&nbsp;I haven't closed my support case yet.&nbsp; And I did get them to share the fact that other customers have reported the same bug.&nbsp; There is even a bug tracking ID.&nbsp; (below)The title of the bug is "REST API - imports with parameter CreateOrOverwrite throws PackagenotfoundError" and it was created on 5th Sept 2022.&nbsp;Here is the Bug ID: 875909Sadly Microsoft doesn't seem to write KB articles about their bugs nowadays (am I a graybeard?)&nbsp; So I don't think you will find this information in an authoritative format from Microsoft.&nbsp; Nor is it likely that they will give you a workaround for your occurrences of DuplicatePackageNotFoundError.&nbsp;&nbsp;

dbeavon3 · Answer

Another thing I should mention is that the error ("DuplicatePackageNotFoundError") is known by another name in the code itself.&nbsp; I think it is known as a "metadata mismatch exception".&nbsp; Meaning that if the Power BI service doesn't recognize the updated dataflow definition, when it compares to the original, then there is a mismatch that causes failure.&nbsp;I tried to ask the PG what characteristics of the dataflow are used for comparing dataflow-metadata during the REST API import operation.&nbsp; So far they have refused to share that information.&nbsp; From what I can tell, there are some pretty arbitrary &amp; silly reasons why a modified dataflow would "mismatch" the target.&nbsp; Here are some possible reasons:&nbsp;Number of tables changed, or schema of tablesDataflow has changed ownershipPossible timing issue - dataflow changes ownership since CSV files were generated.&nbsp; If dataflow changes ownership prior to refresh then perhaps there is not a metadata mismatch.The list of dataflow parameters was changedOther arbitrary types of mismatches.&nbsp;I also want to point out that the team creating the dataflow REST API for imports is very likely to be a different team than the one creating the dataset REST API for imports.&nbsp; I think there is potentially a high degree of inconsistency because they have different philosophies for building REST API's.&nbsp; It appears they must cohabitate the "imports" portion of the REST API, for better or worse.&nbsp; And it appears that the dataflow team doesn't integrate that portion of the API in a very useful or consistent way.

Forum Discussion

Power BI REST API Import Dataflow Overwrite error (DuplicatePackageNotFoundError issue in 2023)

7 Replies

Related content

Copy Job OVERWRITE

Overwrite not working in Copy Data?

Overwrite Dataflows

dataflow gen1

Fabric Dataflow Gen 2