Forum Discussion
Error Connecting to Delta Share
I get the following error when trying to connect to a Databricks Delta Share using a Notebook:
Failed to find the data source: deltaSharing. Please find packages at `https://spark.apache.org/third-party-projects.html`.
Using this code:
shared_df = delta_sharing.load_as_spark(table_url)
I know the connection works because I can pull metadata with:
client = delta_sharing.SharingClient(share_file_path)
client.list_all_tables()
Furthermore, load_as_pandas works.
What is causing it to fail when loading as a Spark dataframe?
The solution was to set up an environment with the delta_sharing library and the following Spark configuration.
runtime_version: '1.3'
spark_conf:
- spark.jars.packages: >-
org.apache.hadoop:hadoop-azure:3.3.1,io.delta:delta-core_2.12:2.2.0,io.delta:delta-sharing-spark_2.12:3.1.0
- spark.sql.extensions: io.delta.sql.DeltaSparkSessionExtension
- spark.sql.catalog.spark_catalog: org.apache.spark.sql.delta.catalog.DeltaCatalog
8 Replies
- AnonymousNot applicable
Hi jjaeger94
Thanks for using Fabric Community.
Can you please send me the screenshot of the error along with the complete code?
Thanks- jjaeger94Helper II
Thank you for reaching out.
Here is the code:
Here is the error:
The following code loading as a pandas dataframe works, so I know the connection to the config file is not a problem and that the table exists:
- AnonymousNot applicable
Hi jjaeger94
The best course of action is to open a support ticket and have our support team take a closer look at it. Please reach out to our support team using this: https://support.fabric.microsoft.com/en-IN/support/ so they can conduct a more thorough investigation into why this is happening.
After creating a support ticket, please provide the ticket number to help us track and gather more information.
Hope this helps. Please let us know if you have any other queries.
- AnonymousNot applicable
Hi jjaeger94 ,
Did you ever manage to solve this? I'm having the same issue. Loading as a pandas dataframe works indeed, but not loading as a spark dataframe. Seems like a Microsoft issue.
- jjaeger94Helper II
No, I have been unable to resolve this and have tried numerous methods.
- AnonymousNot applicable
Hi jjaeger94 ,
I think I found the solution here. In your compute you need to go to the Advanced settings in the configuration tab.
Spark > Spark configThere you can enter some advanced settings. You need to add this:
spark.databricks.delta.formatCheck.enabled false
other than that have the python module installed in the libraries tab in the compute via PyPi:
delta-sharing
The config.share file should be added in your dbfs directory so that spark will be able to accesss it.
This is the python script I am using to get the data from a deltashare using the config.share file:
config_share_path = '/dbfs/FileStore/deltashare/config.share' import delta_sharing # Point to the profile file. It can be a file on the local file system or a file on a remote storage. profile_path = config_share_path # For spark to read it, the config.share path needs to be in a different format!! profile_spark_path = config_share_path.replace("/dbfs","dbfs:") # Create a SharingClient. client = delta_sharing.SharingClient(profile_path) # List all shared tables. # client.list_all_tables() table_name = "YourTableName" table = [table for table in client.list_all_tables() if table.name == table_name][0] # Create a url to access a shared table. table_url = profile_spark_path + f"#{table.share}.{table.schema}.{table.name}" # creating spark dataframe sparkDataFrame = delta_sharing.load_as_spark(table_url) # display spark dataframe display(sparkDataFrame.head(100))Let me know if this works for you! 🙂
- jjaeger94Helper II
The solution was to set up an environment with the delta_sharing library and the following Spark configuration.
runtime_version: '1.3'
spark_conf:
- spark.jars.packages: >-
org.apache.hadoop:hadoop-azure:3.3.1,io.delta:delta-core_2.12:2.2.0,io.delta:delta-sharing-spark_2.12:3.1.0
- spark.sql.extensions: io.delta.sql.DeltaSparkSessionExtension
- spark.sql.catalog.spark_catalog: org.apache.spark.sql.delta.catalog.DeltaCatalog