Forum Discussion

tikka's avatar
tikka
Frequent Visitor
7 years ago

Reading dataset from Azure Databricks

Hi,   We have a Databricks (Premium) environment set up in Azure. Databricks is also set up under a custom Azure Vnet. We are reading prepared datasets from PowerBI using the Databricks cluster's J...
Need Help
  • tikka's avatar
    tikka
    Frequent Visitor
    7 years ago

    Hi,

     

    Thank you for your response.

     

    Unfortunately, we haven't found a solution to the original problem. The thing is that the connection doesn't fail, it just takes a very long time to get a response, like many hours. I'm thinking the problem might relate to the fact that we are running Databricks inside a Vnet. I did a comparison by creating another Databricks workspace, this time without the Vnet, and added a few sample tables. Now the connection is ok, although still quite slow, for example a few minutes to load a table with only 4 columns and 2 rows. Otherwise the network conditions between us and Azure seem ok; no problems with uploads/downloads to storage.

     

    We are still also facing the issue where we publish to PowerBI. It seems to force a gateway on PowerBI Premium, although for no apparent reason. This setting can't even be changed in the dataset settings, to remove the gateway usage. When we try to refresh the dataset, we see the following error:

     

    Refresh failed due to gateway configuration issues.

This dataset requires a properly configured gateway in order to refresh.

If you're using personal mode, please make sure your gateway is online.

If you're using enterprise mode, please make sure you've added the following data sources in the Gateway Management Portal

Extension { extensionDataSourceKind : "Spark" , extensionDataSourcePath : "https://westeurope.azuredatabricks.net:443/sql/protocolv1/o/***/***" }

Please try again later or contact support. If you contact support, please provide these details.

    To my understanding also, this kind of connections doesn't need any kind of gateway.

    • tikka's avatar
      tikka
      Frequent Visitor
      7 years ago

      Another thing we noticed is that the publishing to PowerBI Premium forces a scheduled refresh request into the future, exactly one hour from the time of publishing.

      • tikka's avatar
        tikka
        Frequent Visitor
        7 years ago

        Also, here is the data source from Advanced Editor (ids masked with "*"):

         

        let
    Source = ApacheSpark.Tables("https://westeurope.azuredatabricks.net:443/sql/protocolv1/o/***/***", 2, [BatchSize=null]),
    default_test_csv = Source{[Schema="default",Item="test_csv"]}[Data]
in
    default_test_csv