Forum Discussion

b5lurker's avatar
b5lurker
Advocate II
7 years ago
Solved

Power BI Dataflow to Azure Data Lake Gen2 Setup

Excited to see all of the new Power BI Dataflow capabilities especially with using Azure Data Lake Gen2 as the storage location for those Dataflows.

 

But, I have been trying to get it setup using the Microsoft provided step-by-step guide on: https://docs.microsoft.com/en-us/power-bi/service-dataflows-connect-azure-data-lake-storage-gen2. The specific steps that I have been unable to complete are in the 'Grant Power BI permissions to the file system' section steps #7 and #8. I have only tried it using Azure Storage Explorer v1.6 since that is the method shown in the documentation.

 

When I try to Add the Object IDs for the Power BI Service and Power Query Online as shown in the screenshots I only get an error saying that it cannot find those User IDs/Groups. I have attached the full error message captured in ASE to see if that might help track it down. I did this with my elevated permissions Azure account and also had our admin grant me full Admin rights over our tenant and could not get it working.

 

Has anyone been able to complete the steps to grant the Power BI Service and Power Query Online applications access to the powerbi blob container in their Azure Data Lake Store Gen2? I even re-created the ADLSg2 yesterday since I had it created in the original private preview. Without these steps I have not been able to complete the process in the Power BI Admin screen to connect Power BI up to the Data Lake for Dataflows.

 

Thanks,

Steve

 

Below is the error message from ASE when trying to Add the Power BI Service Object ID in the Manage Access dialog:

FROM CHILD PROCESS 24608: {"id":"MessagePassingHostProxy151","messageType":"FunctionResponse","response":{"type":"error","error":"{\n  \"message\": \"{\\\"message\\\":\\\"HTTP ERROR 404: Not Found\\\",\\\"response\\\":{\\\"statusCode\\\":404,\\\"body\\\":\\\"{\\\\\\\"odata.error\\\\\\\":{\\\\\\\"code\\\\\\\":\\\\\\\"Request_ResourceNotFound\\\\\\\",\\\\\\\"message\\\\\\\":{\\\\\\\"lang\\\\\\\":\\\\\\\"en\\\\\\\",\\\\\\\"value\\\\\\\":\\\\\\\"Resource 'ee4bcd73-1e25-4154-befe-f8180e3f14d5' does not exist or one of its queried reference-property objects are not present.\\\\\\\"},\\\\\\\"requestId\\\\\\\":\\\\\\\"724657f5-7d0c-4896-9230-5226b96ff9b6\\\\\\\",\\\\\\\"date\\\\\\\":\\\\\\\"2018-12-11T16:34:51\\\\\\\"}}\\\",\\\"headers\\\":{\\\"cache-control\\\":\\\"no-cache\\\",\\\"pragma\\\":\\\"no-cache\\\",\\\"content-type\\\":\\\"application/json; odata=minimalmetadata; streaming=true; charset=utf-8\\\",\\\"expires\\\":\\\"-1\\\",\\\"server\\\":\\\"Microsoft-IIS/10.0\\\",\\\"ocp-aad-diagnostics-server-name\\\":\\\"QAcXmVYUoJaUSrw2ftJmfZfMjTyUjtSkz5dJvbTYsQ0=\\\",\\\"request-id\\\":\\\"724657f5-7d0c-4896-9230-5226b96ff9b6\\\",\\\"client-request-id\\\":\\\"587c72a6-407f-4024-95b6-fbc710401123\\\",\\\"x-ms-dirapi-data-contract-version\\\":\\\"1.6\\\",\\\"ocp-aad-session-key\\\":\\\"hqhNfIg59I6k-OSo-mWlGPFJUAeZDoSngtOC2dH4v4szDJot9OmJGoysNRoVpgZKj_T_ctHrILdmnh-x_E_1VTEIkhcExHCXJk-BPGZKyTzxWQqqAJYh7r3biDvlbfg1.tMDQP1ZfH1teuhHoVGVcULDVzDSD4PiGem3E2On_fFM\\\",\\\"dataserviceversion\\\":\\\"3.0;\\\",\\\"strict-transport-security\\\":\\\"max-age=31536000; includeSubDomains\\\",\\\"access-control-allow-origin\\\":\\\"*\\\",\\\"x-aspnet-version\\\":\\\"4.0.30319\\\",\\\"x-powered-by\\\":\\\"ASP.NET\\\",\\\"duration\\\":\\\"430184\\\",\\\"date\\\":\\\"Tue, 11 Dec 2018 16:34:51 GMT\\\",\\\"connection\\\":\\\"close\\\",\\\"content-length\\\":\\\"294\\\"},\\\"request\\\":{\\\"uri\\\":{\\\"protocol\\\":\\\"https:\\\",\\\"slashes\\\":true,\\\"auth\\\":null,\\\"host\\\":\\\"graph.windows.net\\\",\\\"port\\\":443,\\\"hostname\\\":\\\"graph.windows.net\\\",\\\"hash\\\":null,\\\"search\\\":\\\"?api-version=1.6\\\",\\\"query\\\":\\\"api-version=1.6\\\",\\\"pathname\\\":\\\"/cb2bab3d-7d90-44ea-9e31-531011b1213d/users/ee4bcd73-1e25-4154-befe-f8180e3f14d5\\\",\\\"path\\\":\\\"/cb2bab3d-7d90-44ea-9e31-531011b1213d/users/ee4bcd73-1e25-4154-befe-f8180e3f14d5?api-version=1.6\\\",\\\"href\\\":\\\"https://graph.windows.net/cb2bab3d-7d90-44ea-9e31-531011b1213d/users/ee4bcd73-1e25-4154-befe-f8180e3f14d5?api-version=1.6\\\"},\\\"method\\\":\\\"get\\\",\\\"headers\\\":{\\\"Content-Type\\\":\\\"application/json\\\",\\\"Authorization\\\":\\\"Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6IndVTG1ZZnNxZFF1V3RWXy1oeFZ0REpKWk00USIsImtpZCI6IndVTG1ZZnNxZFF1V3RWXy1oeFZ0REpKWk00USJ9.eyJhdWQiOiJodHRwczovL2dyYXBoLndpbmRvd3MubmV0LyIsImlzcyI6Imh0dHBzOi8vc3RzLndpbmRvd3MubmV0L2NiMmJhYjNkLTdkOTAtNDRlYS05ZTMxLTUzMTAxMWIxMjEzZC8iLCJpYXQiOjE1NDQ1NDU3OTEsIm5iZiI6MTU0NDU0NTc5MSwiZXhwIjoxNTQ0NTQ5NjkxLCJhY3IiOiIxIiwiYWlvIjoiQVZRQXEvOEpBQUFBTExXL2dQcVV2ZmhjODM1MGtrYXhKdVFsMGpUcU00N3dwTG14WGo3b05oZGJHRTRYTjZXdk01VFJYanhuWm9jdzh2Z2I3emZDWXR6ZWw5NFo4WHRsd0RKZ3l0LzFqSlFWbWZzNGlVZm44STQ9IiwiYW1yIjpbInB3ZCIsIm1mYSJdLCJhcHBpZCI6IjA0YjA3Nzk1LThkZGItNDYxYS1iYmVlLTAyZjllMWJmN2I0NiIsImFwcGlkYWNyIjoiMCIsImZhbWlseV9uYW1lIjoiV2FrZSIsImdpdmVuX25hbWUiOiIhU3RldmUiLCJpcGFkZHIiOiIyMDYuODAuMjA5LjE5NiIsIm5hbWUiOiIhU3RldmUgV2FrZSIsIm9pZCI6ImZjOTAxZTNkLWE0MGQtNDgzOC05MDZlLWE5Y2UwYTk0ZTk0MyIsIm9ucHJlbV9zaWQiOiJTLTEtNS0yMS0yMTQ3MjQyNzI2LTM2MzMxNTgwLTE4ODQ0MTQ0NC0xNzI1NTkiLCJwdWlkIjoiMTAwMzdGRkVBODFDNTIyNyIsInNjcCI6IjYyZTkwMzk0LTY5ZjUtNDIzNy05MTkwLTAxMjE3NzE0NWUxMCIsInN1YiI6Ilo0TjlINTQ4alBPNUlGSVE0dS1FU01OdllHRE02WndNVEs3dm80LVljSzQiLCJ0ZW5hbnRfcmVnaW9uX3Njb3BlIjoiTkEiLCJ0aWQiOiJjYjJiYWIzZC03ZDkwLTQ0ZWEtOWUzMS01MzEwMTFiMTIxM2QiLCJ1bmlxdWVfbmFtZSI6IiFzd2FrZUBicnduY2FsZC5jb20iLCJ1cG4iOiIhc3dha2VAYnJ3bmNhbGQuY29tIiwidXRpIjoiOGVCcThEa0o4VVNrT05RQ2dZa2tBQSIsInZlciI6IjEuMCJ9.DVQ1-dpeAfEPX9Lf62JuVtheaAiTNECZpTfoihlL93TKFzEJW-N8alEwDLAfkGTwvj92GRgi7MThS-HqWGfAla4C0ntKxuzq8NhTeGEuACNXVUDNO5mtfYi1W2jJDJirduMSOgdNkkBqCnFvVMqM49cUAYEaPOD-Wn-Fb-U_L3K-hOaZ2EL2_aFLXpm5w-JJs9MPQXcoOSR1rT1x5L9x7At0hIn-A97RzzabFTAMfoolAVDymjLrT2It5jf1qn1vZ1aGXBZPLH_uF7_Aq_v_gTPECuuj-B2rJV3FokQHHCYu9iNFl7lt95ZP_lv_Q9nH01QNp-xjxyHfTrub7iCYCA\\\",\\\"content-length\\\":0}}},\\\"body\\\":\\\"{\\\\\\\"odata.error\\\\\\\":{\\\\\\\"code\\\\\\\":\\\\\\\"Request_ResourceNotFound\\\\\\\",\\\\\\\"message\\\\\\\":{\\\\\\\"lang\\\\\\\":\\\\\\\"en\\\\\\\",\\\\\\\"value\\\\\\\":\\\\\\\"Resource 'ee4bcd73-1e25-4154-befe-f8180e3f14d5' does not exist or one of its queried reference-property objects are not present.\\\\\\\"},\\\\\\\"requestId\\\\\\\":\\\\\\\"724657f5-7d0c-4896-9230-5226b96ff9b6\\\\\\\",\\\\\\\"date\\\\\\\":\\\\\\\"2018-12-11T16:34:51\\\\\\\"}}\\\"}\"\n}"}}
  • Michael7's avatar
    Michael7
    7 years ago

    Hi,
     
    There is a temporary issue with settings ACLs through Microsoft Azure Storage Explorer, until this is resolved, as a temporary mitigation you can try using this powershel script:https://setacls.blob.core.windows.net/scriptfolder/setacls.ps1
     
    In order to run the script, your global tenant admin needs to have "Storage Blob Data Contributor (Preview)" role on your storage account.
     
    Instructions on how to run this script:
        
    1. Login as global tenant administrator into Azure portal and open PowerShell (Cloud Shell) in your browser
    2. Choose Azure CLI 2.0 (type ‘az’)
    3. Upload the script (there is a button for it)
    4. Run the script. For example:  /home/admin/setacls -storageaccountname your_alds_gen2_storage_account_name

     

    The script will ask you to sign in, please follow the instructions.
     
    Thanks,
    Michael

     

     

    Hi,

     

    There is no more need in this workaround, please use ASE 1.6.2 which is availably for update and download.

     

    Thanks,

    Michael

68 Replies

  • bensack's avatar
    bensack
    Microsoft Employee

    Hi Steve, 

     

     

    I am sorry for the trouble setting ACLs on the storage account per the documentation. The error you are seeing is a bug in Azure Storage Explorer that unfortunatly prevents setting ACLs for anything but people. It is being fixed, should be available soon, and we are working with the Storage Explorer team on an exact ETA.

     

    Until it is fixed, the only option for setting ACLs is via REST API call as documented here:

    https://docs.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/filesystem/setproperties

    but unfortunatly its not trivial, even with the above documentation.

     

     

     

     

    We are working to expedite the fix and also find an alternative path until such a fix is available, I hope to have an update soon.

    Thank you,

    Ben

     

     

     

    • bensack's avatar
      bensack
      Microsoft Employee

      I would like to update that a new version with the ACL issue fix should be available next week. Until then, another option is to opt in for insider builds of Azure Storage Explorer. In Azure Storage Explorer, click on the Help menu, then Opt in to insider builds. a build should be available via that route sooner.

       

       

      • bensack's avatar
        bensack
        Microsoft Employee

        Hi All, 

         

        Azure Storage Explorer version 1.6.1 is now available as an insider build, and I can confirm it has fixed the issue of setting ACLs.

         

        Here is what you can do to obtain it:

         

        1.  In Azure Storage Explorer, click on the Help menu, then Opt in to insider builds.

        2.  In Azure Storage Explorer, click on the Help menu, then Check for updates.

        After a few moments, a notification bar will appear notifying you a new version is available with an option to install it.

         

        Once installation is complete, you can follow the instructions in Power BI documentation to complete the setup:

        https://docs.microsoft.com/en-us/power-bi/service-dataflows-connect-azure-data-lake-storage-gen2

         

        Thank you,

        Ben

    • joshbrownwsna's avatar
      joshbrownwsna
      Frequent Visitor
      Hi Everyone,

      Just FYI I ran into the same issue today. However I’ve got one small wrinkle. While I was not able to update the ACLs, I was able to get past the screen in Power BI. Now the service is linked I to my storage account, but does not appear to be writing files to the data lake. FYI.

      Josh
      • bensack's avatar
        bensack
        Microsoft Employee

        Hi Josh, 

         

         

        The setup of the storage account must happen as documented - the ACLs are critical for the feature to work and the storage account should not be connected to Power BI without them. I recommend you do not try to use the feature until ACLs are set on the filesystem. 

         

        The fix from Azure Storage Explorer should be available next week, but as mentioned above you can opt in for insider builds and get it sooner. 

         

        Please let me know if you have questions or if you run into any trouble after setting the ACLs.

         

        Thanks,

        Ben

         

         

         

  • Some progress today!

     

    I was able to get the 1.6.1 release of ASE installed today by Opting In on Insider Builds and checking for updates. Once that was setup I was able to complete the ACL changes for the Power BI Service and Power Query Online applications!

     

    Now I'm stuck on the connection in the Power BI Admin panel where I have correctly filled in my Subscription ID, Resource Group Name and Data Lake Storage Account and it just shows a "Something went wrong" error:

     

    Something went wrong
    Error while connecting your Data Lake Storage account
    Please try again later or contact support. If you contact support, please provide these details.
    Activity ID: 3dfbcee0-9cdd-4137-b626-31d40d8aa15a
    Request ID: 4f8a2a8e-a28d-ee1d-39b2-f81c9e5048e0
    Correlation ID: d56c5d02-bd2a-1afd-4dc9-29f46a15d640
    Time: Fri Dec 14 2018 14:02:35 GMT-0700 (Mountain Standard Time)
    Version: 13.0.7683.181
    Cluster URI: https://wabi-west-us-redirect.analysis.windows.net

     

    Not sure what is causing this error as there no other details provided. I do know that there is some text in the documentation about having your ADLSg2 setup in the same region as your Power BI tenant. Our Power BI tenant is in West US as you can see from the link shown in the error above. I was unable to setup an ADLSg2 storage account in West US, so the one that I'm trying to point to is in West US 2. Not sure if that is part of the problem or not, but it is the only thing that stands out for our setup.

     

    At least we are a few steps closer now. :)

     

    Happy Friday!

    Steve

     

     

    • joshbrownwsna's avatar
      joshbrownwsna
      Frequent Visitor

      Hi Everyone,

       

      I was also able to add the ACLs using the internal build. In addition, since my account was already connected, I was able to work through all the steps in the configuration document. 

       

      I also created a new workspace and was able to configure the workspace to use data lake storage. 

       

      Please note, this option is only available in workspaces created to view the new experience.  If your workspace has been created via an O365 group, you will not be able to configure it to use data lake storage.

       

      Unfortunately, when I try to save a workflow I get the following error:

       

      Something went wrong
      Unable to save the dataflow.
      Please try again later or contact support. If you contact support, please provide these details.
      Activity ID: 6cf62757-8eed-4c8f-b89a-19e737873a62
      Request ID: 4ebf59ad-5980-ed22-5432-3f2915ef5446
      Correlation ID: 1f1bbc99-c76b-8c1b-54af-aaa270b10656
      Status code: 500
      Time: Fri Dec 14 2018 22:07:33 GMT-0800 (Pacific Standard Time)
      Version: 13.0.7683.181
      Cluster URI: https://wabi-west-us-redirect.analysis.windows.net

       

      So, unfortunately, no dice ... yet. But progress!

       

      Thanks to everyone for the help.  This has been a great thread!

       

      *** Update 12/16/18 ***

       

      Hi Everyone,

       

      Double checked my config tonight against the instructions laid out in https://docs.microsoft.com/en-us/power-bi/service-dataflows-connect-azure-data-lake-storage-gen2.  Noted my config appears correct, but still getting the 500 error.  Any help in review would be appreciated!

       

      Josh

      • xuanalytics's avatar
        xuanalytics
        Frequent Visitor

        Hi Josh,

         

        I am getting the exact same error as you got. I configured everything according to the Microsoft document. But I am not able to save any data flow in the new app workspace which Data Lake Storage enabled. I have then checked teh activity log of the storage account and I found everytime I am trying to save it 

         

    • Anonymous's avatar
      Anonymous
      Not applicable

      Hi all,

       

      I experience the same issue as described by Steve, got this working until step 2, but then got the following error, when I pressed 'continue'.

       

      Something went wrong

      Error while connecting your Data Lake Storage account

      Please try again later or contact support. If you contact support, please provide these details.

      Activity ID: f53bf51e-984e-4706-9667-49ab5ef8c6a5

      Request ID: 12daf846-e59f-7254-a62a-3ee0bfa3e305

      Correlation ID: 7bb1b282-ece4-856c-0c7e-8b8fdb9097f3

      Time: Thu Dec 20 2018 14:26:34 GMT+0100 (West-Europa (standaardtijd))

      Version: 13.0.7683.188

      Cluster URI: https://wabi-west-europe-b-primary-redirect.analysis.windows.net

       

      When I look at the 'request breakdown'-graph on the dashboard of my datalake, I don't see errors of 'ClientOtherError' at the time that I tried to connect to the lake via PowerBI. (as described by xuanalytics)

  • I was coming here to post about the exact same problem.

    I cannot get past that step either.

     

    It seems that those users (object ids?) seem to already need to be populated in the list because anything I put into the line as far as new user and hit "Add", it fails to find.

     

    Hopefully someone has gotten past this and can provide some insight.

    Seems like just a step missing in the documentation.

     

    I see MatthewRoche is providing feedback on some of the datflows posts.  Maybe he can help us.

    • MatthewRoche's avatar
      MatthewRoche
      Microsoft Employee

      Hi healthEteam - Unfortunately I'm still working throigh the same process. I'll help when I'm able, but it will likely be a day or three...

      • healthEteam's avatar
        healthEteam
        Resolver I
        Great - thanks for responding.
        Everything I know about dataflows thus far have been from your presentations.
        So figured if anyone would know it would likely be you.
        Glad to know you are still working through it as well.
  • Hi All,

     

    I am having similar issues connecting my Azure Data Lake account to Power BI Dataflows. I have called Microsoft Support to get this resolved. I wait till next week before the fix is in place.

     

    I find this thread really helpful.

     

     

    Thanks,

    Bhavesh

  • Anonymous's avatar
    Anonymous
    Not applicable
    I surely checked several times that I followed all steps to connect my ADLS gen 2 to Power BI, but I still get no success. 
    1. I made sure the ADLS is in the same location then my PBI tenant (in my case North Europe)
    2. I give the PBI Service access with the reader role
    3. I give PBI Svc and Power Query full access to the container folder "powerbi"
    4. I even tried full access to all 

    any suggestion why I still get no success?

     

    Aktivitäts-ID14edddab-cb86-48c7-9ce4-6ffc3e48c1ac
    Anforderungs-ID7eb3a30d-892c-c8fc-a5ba-018643494695
    Korrelations-IDac823118-22e0-3bc8-a0fb-60d41fde1b62
    ZeitFri Dec 21 2018 12:54:24 GMT+0100 (Mitteleuropäische Normalzeit)
    Version13.0.7683.188
     
    • julioleite's avatar
      julioleite
      Regular Visitor

      I am with the same issue. Checked the points several times and got the error.

       

       

       

      • sagivh's avatar
        sagivh
        Microsoft Employee

        Hi everyone, we acknowledge the reported setup issues. we have found a potential issue in Azure Storage Explorer and are working on a mitigation. 

        We will update this post when we have more information and ETA. 

         

        Thank you. 

    • Picci's avatar
      Picci
      Helper I

      Hi yehong, what about changing the ADLS Gen2 account to be associated with Power BI?

      What about if I need to move my storage account to another subscription?

       

      Thanks,

      Elisa

      • yehong's avatar
        yehong
        Power BI Team
        You can move the storage account between subscriptions and resource-groups as long as it's kept under the same tenant.