Forum Discussion
Error Connecting to Delta Share
- 1 year ago
The solution was to set up an environment with the delta_sharing library and the following Spark configuration.
runtime_version: '1.3'
spark_conf:
- spark.jars.packages: >-
org.apache.hadoop:hadoop-azure:3.3.1,io.delta:delta-core_2.12:2.2.0,io.delta:delta-sharing-spark_2.12:3.1.0
- spark.sql.extensions: io.delta.sql.DeltaSparkSessionExtension
- spark.sql.catalog.spark_catalog: org.apache.spark.sql.delta.catalog.DeltaCatalog
Hi jjaeger94 ,
Did you ever manage to solve this? I'm having the same issue. Loading as a pandas dataframe works indeed, but not loading as a spark dataframe. Seems like a Microsoft issue.
No, I have been unable to resolve this and have tried numerous methods.
- Anonymous2 years agoNot applicable
Hi jjaeger94 ,
I think I found the solution here. In your compute you need to go to the Advanced settings in the configuration tab.
Spark > Spark configThere you can enter some advanced settings. You need to add this:
spark.databricks.delta.formatCheck.enabled false
other than that have the python module installed in the libraries tab in the compute via PyPi:
delta-sharing
The config.share file should be added in your dbfs directory so that spark will be able to accesss it.
This is the python script I am using to get the data from a deltashare using the config.share file:
config_share_path = '/dbfs/FileStore/deltashare/config.share' import delta_sharing # Point to the profile file. It can be a file on the local file system or a file on a remote storage. profile_path = config_share_path # For spark to read it, the config.share path needs to be in a different format!! profile_spark_path = config_share_path.replace("/dbfs","dbfs:") # Create a SharingClient. client = delta_sharing.SharingClient(profile_path) # List all shared tables. # client.list_all_tables() table_name = "YourTableName" table = [table for table in client.list_all_tables() if table.name == table_name][0] # Create a url to access a shared table. table_url = profile_spark_path + f"#{table.share}.{table.schema}.{table.name}" # creating spark dataframe sparkDataFrame = delta_sharing.load_as_spark(table_url) # display spark dataframe display(sparkDataFrame.head(100))Let me know if this works for you! 🙂
- jjaeger942 years agoHelper II
I'm sorry, is this in Fabric? I am not seeing any of these options.