Forum Discussion

H26027's avatar
H26027
Regular Visitor
2 years ago
Solved

Extract Data from Databricks Unity Catalog

Hi,

I want to connect to Databricks UnityCatalog tables from Fabric Notebooks. Therefore I tried to utilize the databricks jdbc driver to read data with pyspark in this way:

 

personal_access_token = "XXX"
jdbc_url = "jdbc:databricks://adb-XXX.azuredatabricks.net:443/default"

connection_properties = {
"user": "token",
"password": personal_access_token,
"driver": "com.databricks.client.jdbc.Driver",
"ssl": "true",
"httpPath": "sql/protocolv1/o/XXX"
}

df = spark.read.jdbc(url=jdbc_url, table="XXX", properties=connection_properties)

 

I added the JDBC driver .jar file to my custom environment. But still, I am getting this error: 
java.lang.ClassNotFoundException: com.databricks.client.jdbc.Driver

 

What am I doing wrong? Is there an alternative solution? Shortcut is currently no option.

 

 

Thanks in advance.

data engineering
  • H26027's avatar
    H26027
    2 years ago

    I found a solution to my previous mentioned problem. The connection from Fabric Notebook to Databricks Unity Catalog is now working as expected - but not as described initially by Microsoft Documentation.

    What I did now - in case anybody meets the same issue:

    1. Download the Databricks JDBC Driver and store the .jar to Fabric Lakehouse.

    2. Create a custom environment in Fabric 

    3. Add a spark property for spark.jars that is pointing to the ABFSS location of the uploaded .jar file:


    4. In the Notebook itself, connect to this custom environment and use this call for reading the data:

    personal_access_token = "XXX"
jdbc_url = "jdbc:databricks://adb-XXX.azuredatabricks.net:443/default"
http_path = "sql/protocolv1/o/XXX/XXX" 
query = "[catalog].[schema].[table]" # or "(SELECT [selection] FROM [table]) as alias"

df = (spark.read.format("jdbc")
    .option("url", jdbc_url)
    .option("dbtable", query)
    .option("user", "token")
    .option("password", personal_access_token)
    .option("driver", "com.databricks.client.jdbc.Driver")
    .option("ssl", "1")
    .option("ThriftTransport", "2")
    .option("AuthMech", "3")
    .option("httpPath", http_path)
    .option("UseNativeQuery", "0")
    .load())

    The option "UseNativeQuery" is necessary too. Otherwise the jdbc dialect seems to be wrong, causing the before mentioned issue with data types (+ issues with quotes).

     

    Thanks v-cboorla-msft for your help.

11 Replies

  • v-cboorla-msft's avatar
    v-cboorla-msft
    Microsoft Employee
    2 years ago

    Hi H26027 

     

    Thanks for using Microsoft Fabric Community.

    As I understnad that you’re trying to connect to Databricks UnityCatalog tables from Fabric Notebooks using the Databricks JDBC driver. The error message you’re encountering, java.lang.ClassNotFoundException: com.databricks.client.jdbc.Driver, indicates that the JDBC driver class is not found.

    Could you please make sure that you’ve correctly added the Databricks JDBC driver .jar file to your custom environment. Double-check the path and make sure it’s accessible to your Spark session.

    Confirm that the driver version matches your Databricks environment.

    If the issue still persists please do let us know. Glad to help.

    Please refer to this documentation which might help you.

    Documentation link : Integrating Microsoft Fabric and Databricks Unity Catalog

     

    I hope this information helps.

     

    Thanks.

    • H26027's avatar
      H26027
      Regular Visitor
      2 years ago

      The exact steps I did were:
      Create a custom environment (preview) and upload a custom library to it:


      After the upload, I published these changes.

      Then I referenced this environment as the default environment for my workspace.
      The Notebook that I am using is connected to this environment. 
      Then I just launched the code I gave in my initial post. I did not call this library specifically in the notebook though. Is that necessary? If so, how?

      Thanks.

      • v-cboorla-msft's avatar
        v-cboorla-msft
        Microsoft Employee
        2 years ago

        Hi H26027 

         

        Apologize for the delay in response from my end.

        Following up to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others .

        If the issue still persists, kindly provide the following additional information for further assistance.

        To ensure compatibility and troubleshoot the Databricks JDBC driver error you're encountering, could you please confirm the version of your Databricks? Version compatibility might be a potential cause. We can troubleshoot by verifying the appropriate JDBC driver for your specific Databricks runtime version. This might help you resolve the issue.

        For additional information please refer : Unity Catalog limitations

                                                                       : Databricks JDBC Driver

         

        I hope this information helps. Please do let us know if you have any further queries.

         

        Thanks.