Forum Discussion
Error Connecting to Delta Share
- 1 year ago
The solution was to set up an environment with the delta_sharing library and the following Spark configuration.
runtime_version: '1.3'
spark_conf:
- spark.jars.packages: >-
org.apache.hadoop:hadoop-azure:3.3.1,io.delta:delta-core_2.12:2.2.0,io.delta:delta-sharing-spark_2.12:3.1.0
- spark.sql.extensions: io.delta.sql.DeltaSparkSessionExtension
- spark.sql.catalog.spark_catalog: org.apache.spark.sql.delta.catalog.DeltaCatalog
No, I have been unable to resolve this and have tried numerous methods.
Hi jjaeger94 ,
I think I found the solution here. In your compute you need to go to the Advanced settings in the configuration tab.
Spark > Spark config
There you can enter some advanced settings. You need to add this:
spark.databricks.delta.formatCheck.enabled false
other than that have the python module installed in the libraries tab in the compute via PyPi:
delta-sharing
The config.share file should be added in your dbfs directory so that spark will be able to accesss it.
This is the python script I am using to get the data from a deltashare using the config.share file:
config_share_path = '/dbfs/FileStore/deltashare/config.share'
import delta_sharing
# Point to the profile file. It can be a file on the local file system or a file on a remote storage.
profile_path = config_share_path
# For spark to read it, the config.share path needs to be in a different format!!
profile_spark_path = config_share_path.replace("/dbfs","dbfs:")
# Create a SharingClient.
client = delta_sharing.SharingClient(profile_path)
# List all shared tables.
# client.list_all_tables()
table_name = "YourTableName"
table = [table for table in client.list_all_tables() if table.name == table_name][0]
# Create a url to access a shared table.
table_url = profile_spark_path + f"#{table.share}.{table.schema}.{table.name}"
# creating spark dataframe
sparkDataFrame = delta_sharing.load_as_spark(table_url)
# display spark dataframe
display(sparkDataFrame.head(100))
Let me know if this works for you! 🙂
- jjaeger941 year agoHelper II
I'm sorry, is this in Fabric? I am not seeing any of these options.