Forum Discussion

jjaeger94's avatar
jjaeger94
Helper II
2 years ago
Solved

Error Connecting to Delta Share

I get the following error when trying to connect to a Databricks Delta Share using a Notebook:

 

Failed to find the data source: deltaSharing. Please find packages at `https://spark.apache.org/third-party-projects.html`.

 

Using this code:

 

 

shared_df = delta_sharing.load_as_spark(table_url)

 

 

I know the connection works because I can pull metadata with:

 

client = delta_sharing.SharingClient(share_file_path)

client.list_all_tables()

 

 

Furthermore, load_as_pandas works.

 

What is causing it to fail when loading as a Spark dataframe?

  • The solution was to set up an environment with the delta_sharing library and the following Spark configuration.

     

    runtime_version: '1.3'
    spark_conf:
    - spark.jars.packages: >-
    org.apache.hadoop:hadoop-azure:3.3.1,io.delta:delta-core_2.12:2.2.0,io.delta:delta-sharing-spark_2.12:3.1.0
    - spark.sql.extensions: io.delta.sql.DeltaSparkSessionExtension
    - spark.sql.catalog.spark_catalog: org.apache.spark.sql.delta.catalog.DeltaCatalog

8 Replies

  • Anonymous's avatar
    Anonymous
    Not applicable

    Hi jjaeger94 
    Thanks for using Fabric Community.
    Can you please send me the screenshot of the error along with the complete code? 

    Thanks

    • jjaeger94's avatar
      jjaeger94
      Helper II

      Thank you for reaching out.

       

      Here is the code:

       

       

       

      Here is the error:

       

       

      The following code loading as a pandas dataframe works, so I know the connection to the config file is not a problem and that the table exists:

       

       

      • Anonymous's avatar
        Anonymous
        Not applicable

        Hi jjaeger94 

        The best course of action is to open a support ticket and have our support team take a closer look at it. Please reach out to our support team using this: https://support.fabric.microsoft.com/en-IN/support/ so they can conduct a more thorough investigation into why this is happening.

        After creating a support ticket, please provide the ticket number to help us track and gather more information.

        Hope this helps. Please let us know if you have any other queries.

  • Anonymous's avatar
    Anonymous
    Not applicable

    Hi jjaeger94 ,

     

    Did you ever manage to solve this? I'm having the same issue. Loading as a pandas dataframe works indeed, but not loading as a spark dataframe. Seems like a Microsoft issue.

    • jjaeger94's avatar
      jjaeger94
      Helper II

      No, I have been unable to resolve this and have tried numerous methods.

      • Anonymous's avatar
        Anonymous
        Not applicable

        Hi jjaeger94 ,

         

        I think I found the solution here. In your compute you need to go to the Advanced settings in the configuration tab.
        Spark > Spark config

        There you can enter some advanced settings. You need to add this:

        spark.databricks.delta.formatCheck.enabled false

         

        other than that have the python module installed in the libraries tab in the compute via PyPi:

        delta-sharing

         

        The config.share file should be added in your dbfs directory so that spark will be able to accesss it.

         

        This is the python script I am using to get the data from a deltashare using the config.share file:

         

        config_share_path = '/dbfs/FileStore/deltashare/config.share'
        import delta_sharing
        
        # Point to the profile file. It can be a file on the local file system or a file on a remote storage.
        profile_path = config_share_path
        
        # For spark to read it, the config.share path needs to be in a different format!!
        profile_spark_path = config_share_path.replace("/dbfs","dbfs:")
        
        # Create a SharingClient.
        client = delta_sharing.SharingClient(profile_path)
        
        # List all shared tables.
        # client.list_all_tables()
        
        table_name = "YourTableName"
        table = [table for table in client.list_all_tables() if table.name == table_name][0]
        
        # Create a url to access a shared table.
        table_url = profile_spark_path + f"#{table.share}.{table.schema}.{table.name}"
        
        # creating spark dataframe
        sparkDataFrame = delta_sharing.load_as_spark(table_url)
        
        # display spark dataframe
        display(sparkDataFrame.head(100))

         

        Let me know if this works for you! 🙂

  • The solution was to set up an environment with the delta_sharing library and the following Spark configuration.

     

    runtime_version: '1.3'
    spark_conf:
    - spark.jars.packages: >-
    org.apache.hadoop:hadoop-azure:3.3.1,io.delta:delta-core_2.12:2.2.0,io.delta:delta-sharing-spark_2.12:3.1.0
    - spark.sql.extensions: io.delta.sql.DeltaSparkSessionExtension
    - spark.sql.catalog.spark_catalog: org.apache.spark.sql.delta.catalog.DeltaCatalog