Forum Discussion

jjaeger94's avatar
jjaeger94
Icon for Helper II rankHelper II
2 years ago
Solved

Error Connecting to Delta Share

I get the following error when trying to connect to a Databricks Delta Share using a Notebook:   Failed to find the data source: deltaSharing. Please find packages at `https://spark.apache.org/thir...
data engineering
notebook
Spark Job Definition
  • jjaeger94's avatar
    jjaeger94
    1 year ago

    The solution was to set up an environment with the delta_sharing library and the following Spark configuration.

     

    runtime_version: '1.3'
    spark_conf:
    - spark.jars.packages: >-
    org.apache.hadoop:hadoop-azure:3.3.1,io.delta:delta-core_2.12:2.2.0,io.delta:delta-sharing-spark_2.12:3.1.0
    - spark.sql.extensions: io.delta.sql.DeltaSparkSessionExtension
    - spark.sql.catalog.spark_catalog: org.apache.spark.sql.delta.catalog.DeltaCatalog

jjaeger94's avatar
jjaeger94
Icon for Helper II rankHelper II
1 year ago

The solution was to set up an environment with the delta_sharing library and the following Spark configuration.

 

runtime_version: '1.3'
spark_conf:
- spark.jars.packages: >-
org.apache.hadoop:hadoop-azure:3.3.1,io.delta:delta-core_2.12:2.2.0,io.delta:delta-sharing-spark_2.12:3.1.0
- spark.sql.extensions: io.delta.sql.DeltaSparkSessionExtension
- spark.sql.catalog.spark_catalog: org.apache.spark.sql.delta.catalog.DeltaCatalog