topic Re: Error Connecting to Delta Share in Data Engineering

Error Connecting to Delta Share

jjaeger94 — Thu, 20 Jun 2024 15:43:29 GMT

I get the following error when trying to connect to a Databricks Delta Share using a Notebook:

Failed to find the data source: deltaSharing. Please find packages at `https://spark.apache.org/third-party-projects.html`.

Using this code:

shared_df = delta_sharing.load_as_spark(table_url)

I know the connection works because I can pull metadata with:

client = delta_sharing.SharingClient(share_file_path) client.list_all_tables()

Furthermore, load_as_pandas works.

What is causing it to fail when loading as a Spark dataframe?

Re: Error Connecting to Delta Share

Anonymous — Fri, 21 Jun 2024 10:19:38 GMT

Hi jjaeger94
Thanks for using Fabric Community.
Can you please send me the screenshot of the error along with the complete code?

Thanks

Re: Error Connecting to Delta Share

jjaeger94 — Fri, 21 Jun 2024 13:48:11 GMT

Thank you for reaching out.

Here is the code:

Here is the error:

The following code loading as a pandas dataframe works, so I know the connection to the config file is not a problem and that the table exists:

Re: Error Connecting to Delta Share

Anonymous — Fri, 21 Jun 2024 16:54:47 GMT

Hi jjaeger94

The best course of action is to open a support ticket and have our support team take a closer look at it. Please reach out to our support team using this: https://support.fabric.microsoft.com/en-IN/support/ so they can conduct a more thorough investigation into why this is happening.

After creating a support ticket, please provide the ticket number to help us track and gather more information.

Hope this helps. Please let us know if you have any other queries.

Re: Error Connecting to Delta Share

Anonymous — Wed, 04 Sep 2024 21:53:22 GMT

Hi jjaeger94 ,

Did you ever manage to solve this? I'm having the same issue. Loading as a pandas dataframe works indeed, but not loading as a spark dataframe. Seems like a Microsoft issue.

Re: Error Connecting to Delta Share

jjaeger94 — Thu, 05 Sep 2024 17:47:07 GMT

No, I have been unable to resolve this and have tried numerous methods.

Re: Error Connecting to Delta Share

Anonymous — Thu, 05 Sep 2024 20:05:01 GMT

Hi jjaeger94 ,

I think I found the solution here. In your compute you need to go to the Advanced settings in the configuration tab.
Spark > Spark config

There you can enter some advanced settings. You need to add this:

spark.databricks.delta.formatCheck.enabled false

other than that have the python module installed in the libraries tab in the compute via PyPi:

delta-sharing

The config.share file should be added in your dbfs directory so that spark will be able to accesss it.

This is the python script I am using to get the data from a deltashare using the config.share file:

config_share_path = '/dbfs/FileStore/deltashare/config.share' import delta_sharing # Point to the profile file. It can be a file on the local file system or a file on a remote storage. profile_path = config_share_path # For spark to read it, the config.share path needs to be in a different format!! profile_spark_path = config_share_path.replace("/dbfs","dbfs:") # Create a SharingClient. client = delta_sharing.SharingClient(profile_path) # List all shared tables. # client.list_all_tables() table_name = "YourTableName" table = [table for table in client.list_all_tables() if table.name == table_name][0] # Create a url to access a shared table. table_url = profile_spark_path + f"#{table.share}.{table.schema}.{table.name}" # creating spark dataframe sparkDataFrame = delta_sharing.load_as_spark(table_url) # display spark dataframe display(sparkDataFrame.head(100))

Let me know if this works for you! 🙂

Re: Error Connecting to Delta Share

jjaeger94 — Thu, 05 Sep 2024 20:12:57 GMT

I'm sorry, is this in Fabric? I am not seeing any of these options.

Re: Error Connecting to Delta Share

jjaeger94 — Mon, 30 Sep 2024 18:24:52 GMT

The solution was to set up an environment with the delta_sharing library and the following Spark configuration.

runtime_version: '1.3'
spark_conf:
- spark.jars.packages: >-
org.apache.hadoop:hadoop-azure:3.3.1,io.delta:delta-core_2.12:2.2.0,io.delta:delta-sharing-spark_2.12:3.1.0
- spark.sql.extensions: io.delta.sql.DeltaSparkSessionExtension
- spark.sql.catalog.spark_catalog: org.apache.spark.sql.delta.catalog.DeltaCatalog