topic Re: Power BI Dataflow to Azure Data Lake Gen2 Setup in Service

Power BI Dataflow to Azure Data Lake Gen2 Setup

b5lurker — Tue, 11 Dec 2018 19:35:44 GMT

Excited to see all of the new Power BI Dataflow capabilities especially with using Azure Data Lake Gen2 as the storage location for those Dataflows.

But, I have been trying to get it setup using the Microsoft provided step-by-step guide on: https://docs.microsoft.com/en-us/power-bi/service-dataflows-connect-azure-data-lake-storage-gen2. The specific steps that I have been unable to complete are in the 'Grant Power BI permissions to the file system' section steps #7 and #8. I have only tried it using Azure Storage Explorer v1.6 since that is the method shown in the documentation.

When I try to Add the Object IDs for the Power BI Service and Power Query Online as shown in the screenshots I only get an error saying that it cannot find those User IDs/Groups. I have attached the full error message captured in ASE to see if that might help track it down. I did this with my elevated permissions Azure account and also had our admin grant me full Admin rights over our tenant and could not get it working.

Has anyone been able to complete the steps to grant the Power BI Service and Power Query Online applications access to the powerbi blob container in their Azure Data Lake Store Gen2? I even re-created the ADLSg2 yesterday since I had it created in the original private preview. Without these steps I have not been able to complete the process in the Power BI Admin screen to connect Power BI up to the Data Lake for Dataflows.

Thanks,

Steve

Below is the error message from ASE when trying to Add the Power BI Service Object ID in the Manage Access dialog:

FROM CHILD PROCESS 24608: {"id":"MessagePassingHostProxy151","messageType":"FunctionResponse","response":{"type":"error","error":"{\n  \"message\": \"{\\\"message\\\":\\\"HTTP ERROR 404: Not Found\\\",\\\"response\\\":{\\\"statusCode\\\":404,\\\"body\\\":\\\"{\\\\\\\"odata.error\\\\\\\":{\\\\\\\"code\\\\\\\":\\\\\\\"Request_ResourceNotFound\\\\\\\",\\\\\\\"message\\\\\\\":{\\\\\\\"lang\\\\\\\":\\\\\\\"en\\\\\\\",\\\\\\\"value\\\\\\\":\\\\\\\"Resource 'ee4bcd73-1e25-4154-befe-f8180e3f14d5' does not exist or one of its queried reference-property objects are not present.\\\\\\\"},\\\\\\\"requestId\\\\\\\":\\\\\\\"724657f5-7d0c-4896-9230-5226b96ff9b6\\\\\\\",\\\\\\\"date\\\\\\\":\\\\\\\"2018-12-11T16:34:51\\\\\\\"}}\\\",\\\"headers\\\":{\\\"cache-control\\\":\\\"no-cache\\\",\\\"pragma\\\":\\\"no-cache\\\",\\\"content-type\\\":\\\"application/json; odata=minimalmetadata; streaming=true; charset=utf-8\\\",\\\"expires\\\":\\\"-1\\\",\\\"server\\\":\\\"Microsoft-IIS/10.0\\\",\\\"ocp-aad-diagnostics-server-name\\\":\\\"QAcXmVYUoJaUSrw2ftJmfZfMjTyUjtSkz5dJvbTYsQ0=\\\",\\\"request-id\\\":\\\"724657f5-7d0c-4896-9230-5226b96ff9b6\\\",\\\"client-request-id\\\":\\\"587c72a6-407f-4024-95b6-fbc710401123\\\",\\\"x-ms-dirapi-data-contract-version\\\":\\\"1.6\\\",\\\"ocp-aad-session-key\\\":\\\"hqhNfIg59I6k-OSo-mWlGPFJUAeZDoSngtOC2dH4v4szDJot9OmJGoysNRoVpgZKj_T_ctHrILdmnh-x_E_1VTEIkhcExHCXJk-BPGZKyTzxWQqqAJYh7r3biDvlbfg1.tMDQP1ZfH1teuhHoVGVcULDVzDSD4PiGem3E2On_fFM\\\",\\\"dataserviceversion\\\":\\\"3.0;\\\",\\\"strict-transport-security\\\":\\\"max-age=31536000; includeSubDomains\\\",\\\"access-control-allow-origin\\\":\\\"*\\\",\\\"x-aspnet-version\\\":\\\"4.0.30319\\\",\\\"x-powered-by\\\":\\\"ASP.NET\\\",\\\"duration\\\":\\\"430184\\\",\\\"date\\\":\\\"Tue, 11 Dec 2018 16:34:51 GMT\\\",\\\"connection\\\":\\\"close\\\",\\\"content-length\\\":\\\"294\\\"},\\\"request\\\":{\\\"uri\\\":{\\\"protocol\\\":\\\"https:\\\",\\\"slashes\\\":true,\\\"auth\\\":null,\\\"host\\\":\\\"graph.windows.net\\\",\\\"port\\\":443,\\\"hostname\\\":\\\"graph.windows.net\\\",\\\"hash\\\":null,\\\"search\\\":\\\"?api-version=1.6\\\",\\\"query\\\":\\\"api-version=1.6\\\",\\\"pathname\\\":\\\"/cb2bab3d-7d90-44ea-9e31-531011b1213d/users/ee4bcd73-1e25-4154-befe-f8180e3f14d5\\\",\\\"path\\\":\\\"/cb2bab3d-7d90-44ea-9e31-531011b1213d/users/ee4bcd73-1e25-4154-befe-f8180e3f14d5?api-version=1.6\\\",\\\"href\\\":\\\"https://graph.windows.net/cb2bab3d-7d90-44ea-9e31-531011b1213d/users/ee4bcd73-1e25-4154-befe-f8180e3f14d5?api-version=1.6\\\"},\\\"method\\\":\\\"get\\\",\\\"headers\\\":{\\\"Content-Type\\\":\\\"application/json\\\",\\\"Authorization\\\":\\\"Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6IndVTG1ZZnNxZFF1V3RWXy1oeFZ0REpKWk00USIsImtpZCI6IndVTG1ZZnNxZFF1V3RWXy1oeFZ0REpKWk00USJ9.eyJhdWQiOiJodHRwczovL2dyYXBoLndpbmRvd3MubmV0LyIsImlzcyI6Imh0dHBzOi8vc3RzLndpbmRvd3MubmV0L2NiMmJhYjNkLTdkOTAtNDRlYS05ZTMxLTUzMTAxMWIxMjEzZC8iLCJpYXQiOjE1NDQ1NDU3OTEsIm5iZiI6MTU0NDU0NTc5MSwiZXhwIjoxNTQ0NTQ5NjkxLCJhY3IiOiIxIiwiYWlvIjoiQVZRQXEvOEpBQUFBTExXL2dQcVV2ZmhjODM1MGtrYXhKdVFsMGpUcU00N3dwTG14WGo3b05oZGJHRTRYTjZXdk01VFJYanhuWm9jdzh2Z2I3emZDWXR6ZWw5NFo4WHRsd0RKZ3l0LzFqSlFWbWZzNGlVZm44STQ9IiwiYW1yIjpbInB3ZCIsIm1mYSJdLCJhcHBpZCI6IjA0YjA3Nzk1LThkZGItNDYxYS1iYmVlLTAyZjllMWJmN2I0NiIsImFwcGlkYWNyIjoiMCIsImZhbWlseV9uYW1lIjoiV2FrZSIsImdpdmVuX25hbWUiOiIhU3RldmUiLCJpcGFkZHIiOiIyMDYuODAuMjA5LjE5NiIsIm5hbWUiOiIhU3RldmUgV2FrZSIsIm9pZCI6ImZjOTAxZTNkLWE0MGQtNDgzOC05MDZlLWE5Y2UwYTk0ZTk0MyIsIm9ucHJlbV9zaWQiOiJTLTEtNS0yMS0yMTQ3MjQyNzI2LTM2MzMxNTgwLTE4ODQ0MTQ0NC0xNzI1NTkiLCJwdWlkIjoiMTAwMzdGRkVBODFDNTIyNyIsInNjcCI6IjYyZTkwMzk0LTY5ZjUtNDIzNy05MTkwLTAxMjE3NzE0NWUxMCIsInN1YiI6Ilo0TjlINTQ4alBPNUlGSVE0dS1FU01OdllHRE02WndNVEs3dm80LVljSzQiLCJ0ZW5hbnRfcmVnaW9uX3Njb3BlIjoiTkEiLCJ0aWQiOiJjYjJiYWIzZC03ZDkwLTQ0ZWEtOWUzMS01MzEwMTFiMTIxM2QiLCJ1bmlxdWVfbmFtZSI6IiFzd2FrZUBicnduY2FsZC5jb20iLCJ1cG4iOiIhc3dha2VAYnJ3bmNhbGQuY29tIiwidXRpIjoiOGVCcThEa0o4VVNrT05RQ2dZa2tBQSIsInZlciI6IjEuMCJ9.DVQ1-dpeAfEPX9Lf62JuVtheaAiTNECZpTfoihlL93TKFzEJW-N8alEwDLAfkGTwvj92GRgi7MThS-HqWGfAla4C0ntKxuzq8NhTeGEuACNXVUDNO5mtfYi1W2jJDJirduMSOgdNkkBqCnFvVMqM49cUAYEaPOD-Wn-Fb-U_L3K-hOaZ2EL2_aFLXpm5w-JJs9MPQXcoOSR1rT1x5L9x7At0hIn-A97RzzabFTAMfoolAVDymjLrT2It5jf1qn1vZ1aGXBZPLH_uF7_Aq_v_gTPECuuj-B2rJV3FokQHHCYu9iNFl7lt95ZP_lv_Q9nH01QNp-xjxyHfTrub7iCYCA\\\",\\\"content-length\\\":0}}},\\\"body\\\":\\\"{\\\\\\\"odata.error\\\\\\\":{\\\\\\\"code\\\\\\\":\\\\\\\"Request_ResourceNotFound\\\\\\\",\\\\\\\"message\\\\\\\":{\\\\\\\"lang\\\\\\\":\\\\\\\"en\\\\\\\",\\\\\\\"value\\\\\\\":\\\\\\\"Resource 'ee4bcd73-1e25-4154-befe-f8180e3f14d5' does not exist or one of its queried reference-property objects are not present.\\\\\\\"},\\\\\\\"requestId\\\\\\\":\\\\\\\"724657f5-7d0c-4896-9230-5226b96ff9b6\\\\\\\",\\\\\\\"date\\\\\\\":\\\\\\\"2018-12-11T16:34:51\\\\\\\"}}\\\"}\"\n}"}}

Re: Power BI Dataflow to Azure Data Lake Gen2 Setup

healthEteam — Wed, 12 Dec 2018 02:36:57 GMT

I was coming here to post about the exact same problem.

I cannot get past that step either.

It seems that those users (object ids?) seem to already need to be populated in the list because anything I put into the line as far as new user and hit "Add", it fails to find.

Hopefully someone has gotten past this and can provide some insight.

Seems like just a step missing in the documentation.

I see MatthewRoche is providing feedback on some of the datflows posts. Maybe he can help us.

Re: Power BI Dataflow to Azure Data Lake Gen2 Setup

MatthewRoche — Wed, 12 Dec 2018 02:42:39 GMT

Hi healthEteam - Unfortunately I'm still working throigh the same process. I'll help when I'm able, but it will likely be a day or three...

Re: Power BI Dataflow to Azure Data Lake Gen2 Setup

healthEteam — Wed, 12 Dec 2018 03:14:05 GMT

Great - thanks for responding.
Everything I know about dataflows thus far have been from your presentations.
So figured if anyone would know it would likely be you.
Glad to know you are still working through it as well.

Re: Power BI Dataflow to Azure Data Lake Gen2 Setup

MatthewRoche — Wed, 12 Dec 2018 03:23:23 GMT

Thank you!

(Blushing furiously)

One of the benefits of working at Microsoft means that I get to play with all the cool toys...

...but to test this feature end to end I need to set up my own Power BI tenant, and that isn't fitting into my week so far. :-(

Re: Power BI Dataflow to Azure Data Lake Gen2 Setup

Anonymous — Wed, 12 Dec 2018 14:37:50 GMT

Same issue here when I try grant permissions to Apps. Following https://community.powerbi.com/t5/Service/Power-BI-Dataflow-to-Azure-Data-Lake-Gen2-Setup/m-p/583564/highlight/false#M58962.

It seems Azure Storage Explorer cant grant permissions to Apps ObjectId. The ids showed in the Storage Explorer screenshot are not the same as in the previous App ObjectId Screenshot.

Re: Power BI Dataflow to Azure Data Lake Gen2 Setup

bensack — Wed, 12 Dec 2018 17:24:39 GMT

Hi Steve,

I am sorry for the trouble setting ACLs on the storage account per the documentation. The error you are seeing is a bug in Azure Storage Explorer that unfortunatly prevents setting ACLs for anything but people. It is being fixed, should be available soon, and we are working with the Storage Explorer team on an exact ETA.

Until it is fixed, the only option for setting ACLs is via REST API call as documented here:

https://docs.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/filesystem/setproperties

but unfortunatly its not trivial, even with the above documentation.

We are working to expedite the fix and also find an alternative path until such a fix is available, I hope to have an update soon.

Thank you,

Ben

Re: Power BI Dataflow to Azure Data Lake Gen2 Setup

joshbrownwsna — Thu, 13 Dec 2018 08:19:33 GMT

Hi Everyone,

Just FYI I ran into the same issue today. However I’ve got one small wrinkle. While I was not able to update the ACLs, I was able to get past the screen in Power BI. Now the service is linked I to my storage account, but does not appear to be writing files to the data lake. FYI.

Josh

Re: Power BI Dataflow to Azure Data Lake Gen2 Setup

bensack — Thu, 13 Dec 2018 16:01:36 GMT

I would like to update that a new version with the ACL issue fix should be available next week. Until then, another option is to opt in for insider builds of Azure Storage Explorer. In Azure Storage Explorer, click on the Help menu, then Opt in to insider builds. a build should be available via that route sooner.

Re: Power BI Dataflow to Azure Data Lake Gen2 Setup

bensack — Thu, 13 Dec 2018 16:06:52 GMT

Hi Josh,

The setup of the storage account must happen as documented - the ACLs are critical for the feature to work and the storage account should not be connected to Power BI without them. I recommend you do not try to use the feature until ACLs are set on the filesystem.

The fix from Azure Storage Explorer should be available next week, but as mentioned above you can opt in for insider builds and get it sooner.

Please let me know if you have questions or if you run into any trouble after setting the ACLs.

Thanks,

Ben

Re: Power BI Dataflow to Azure Data Lake Gen2 Setup

healthEteam — Thu, 13 Dec 2018 16:16:09 GMT

Thanks Ben -

If you would, can you please update this thread when this update has been released so we will know when we can go retreive it.

Thanks again

Re: Power BI Dataflow to Azure Data Lake Gen2 Setup

joshbrownwsna — Thu, 13 Dec 2018 16:51:42 GMT

Hi Ben,

Thank you for your reply. Unfortunately, I linked my power bi account before finding this post. While we have linked the power bi account, we have not configured any of our app workspaces to use the linked storage account. Will using dataflows like this cause us issues or will our flows continue to populate the Power BI storage?

Is there any way in Power BI to unlink the account? There does not seem to be a setting to do so in the Power BI interface.

Josh

Re: Power BI Dataflow to Azure Data Lake Gen2 Setup

BhaveshPatel — Fri, 14 Dec 2018 01:37:44 GMT

Hi All,

I am having similar issues connecting my Azure Data Lake account to Power BI Dataflows. I have called Microsoft Support to get this resolved. I wait till next week before the fix is in place.

I find this thread really helpful.

Thanks,

Bhavesh

Re: Power BI Dataflow to Azure Data Lake Gen2 Setup

joshbrownwsna — Fri, 14 Dec 2018 02:13:15 GMT

Hi Everyone,

Posting in case anyone has linked their account before they have the ACLs in place as I have. I can answer the questions I posed above.

As I noted above I had linked our Powr BI account to our gen 2 storage account before I found this thread. However, since we had not enabled any of our workspaces to use the data lake storage account, our exiisting Dataflows have continued to write to the Power BI storage.

Said another way using Dataflows with Data Lake storage takes 5 steps:

1. Create a Gen2 Storage account
2. Configure the storage account so that the Powe BI service (power query, power bi service, power bi premium) can read and write to the storage account. This is done in Azure and the storage explorer (once it’s updated).
3. Use the admin panel in Power Bi web app to Enable/link your Power BI instance to the storage account.
4. Use the admin panel in Power BI to provide workspace admins the ability to use datalake storage for their Dataflows.
5. Use the Power BI web app to enable the use of Datalake storage for dataflows in for each workspace

Unless all of these steps are done the data flow will continue to use the internal power bi storage.

Hope that’s helpful to others. This is all laid out in the different pieces of documentation, however, it’s not as clear if you start out from the Powe BI admin panel. If I’m in correct please let me know!

Josh

Re: Power BI Dataflow to Azure Data Lake Gen2 Setup

b5lurker — Fri, 14 Dec 2018 21:08:13 GMT

Some progress today!

I was able to get the 1.6.1 release of ASE installed today by Opting In on Insider Builds and checking for updates. Once that was setup I was able to complete the ACL changes for the Power BI Service and Power Query Online applications!

Now I'm stuck on the connection in the Power BI Admin panel where I have correctly filled in my Subscription ID, Resource Group Name and Data Lake Storage Account and it just shows a "Something went wrong" error:

Something went wrong
Error while connecting your Data Lake Storage account
Please try again later or contact support. If you contact support, please provide these details.
Activity ID: 3dfbcee0-9cdd-4137-b626-31d40d8aa15a
Request ID: 4f8a2a8e-a28d-ee1d-39b2-f81c9e5048e0
Correlation ID: d56c5d02-bd2a-1afd-4dc9-29f46a15d640
Time: Fri Dec 14 2018 14:02:35 GMT-0700 (Mountain Standard Time)
Version: 13.0.7683.181
Cluster URI: https://wabi-west-us-redirect.analysis.windows.net

Not sure what is causing this error as there no other details provided. I do know that there is some text in the documentation about having your ADLSg2 setup in the same region as your Power BI tenant. Our Power BI tenant is in West US as you can see from the link shown in the error above. I was unable to setup an ADLSg2 storage account in West US, so the one that I'm trying to point to is in West US 2. Not sure if that is part of the problem or not, but it is the only thing that stands out for our setup.

At least we are a few steps closer now. :)

Happy Friday!

Steve

Re: Power BI Dataflow to Azure Data Lake Gen2 Setup

joshbrownwsna — Mon, 17 Dec 2018 04:58:43 GMT

Hi Everyone,

I was also able to add the ACLs using the internal build. In addition, since my account was already connected, I was able to work through all the steps in the configuration document.

I also created a new workspace and was able to configure the workspace to use data lake storage.

Please note, this option is only available in workspaces created to view the new experience. If your workspace has been created via an O365 group, you will not be able to configure it to use data lake storage.

Unfortunately, when I try to save a workflow I get the following error:

Something went wrong
Unable to save the dataflow.
Please try again later or contact support. If you contact support, please provide these details.
Activity ID: 6cf62757-8eed-4c8f-b89a-19e737873a62
Request ID: 4ebf59ad-5980-ed22-5432-3f2915ef5446
Correlation ID: 1f1bbc99-c76b-8c1b-54af-aaa270b10656
Status code: 500
Time: Fri Dec 14 2018 22:07:33 GMT-0800 (Pacific Standard Time)
Version: 13.0.7683.181
Cluster URI: https://wabi-west-us-redirect.analysis.windows.net

So, unfortunately, no dice ... yet. But progress!

Thanks to everyone for the help. This has been a great thread!

*** Update 12/16/18 ***

Hi Everyone,

Double checked my config tonight against the instructions laid out in https://docs.microsoft.com/en-us/power-bi/service-dataflows-connect-azure-data-lake-storage-gen2. Noted my config appears correct, but still getting the 500 error. Any help in review would be appreciated!

Josh

Re: Power BI Dataflow to Azure Data Lake Gen2 Setup

bensack — Sun, 16 Dec 2018 15:18:21 GMT

Hi All,

Azure Storage Explorer version 1.6.1 is now available as an insider build, and I can confirm it has fixed the issue of setting ACLs.

Here is what you can do to obtain it:

1. In Azure Storage Explorer, click on the Help menu, then Opt in to insider builds.

2. In Azure Storage Explorer, click on the Help menu, then Check for updates.

After a few moments, a notification bar will appear notifying you a new version is available with an option to install it.

Once installation is complete, you can follow the instructions in Power BI documentation to complete the setup:

https://docs.microsoft.com/en-us/power-bi/service-dataflows-connect-azure-data-lake-storage-gen2

Thank you,

Ben

Re: Power BI Dataflow to Azure Data Lake Gen2 Setup

xuanalytics — Wed, 19 Dec 2018 14:45:03 GMT

Hi Josh,

I am getting the exact same error as you got. I configured everything according to the Microsoft document. But I am not able to save any data flow in the new app workspace which Data Lake Storage enabled. I have then checked teh activity log of the storage account and I found everytime I am trying to save it

Re: Power BI Dataflow to Azure Data Lake Gen2 Setup

yehong — Thu, 20 Dec 2018 06:00:55 GMT

Hi xuanalytics - are you seeing the failures in PowerBI portal? can you please share the "Something went wrong" message contents?

Re: Power BI Dataflow to Azure Data Lake Gen2 Setup

xuanalytics — Thu, 20 Dec 2018 12:58:48 GMT

Hi yehong, yes, I got it from PowerBI portal and it is the same error as Josh got. Here is the information.

Something went wrong
Unable to save the dataflow.
Please try again later or contact support. If you contact support, please provide these details.
Activity ID: d33a532d-45eb-4d65-9b7a-f7041b3cd1a8
Request ID: 9135fad1-a6f5-c6f9-e204-165cfd0e48ba
Correlation ID: c7c3a7ba-233c-8e8f-1139-28bbb6935973
Status code: 500
Time: Thu Dec 20 2018 07:55:02 GMT-0500 (Eastern Standard Time)
Version: 13.0.7683.188
Cluster URI: https://wabi-us-west2-redirect.analysis.windows.net