Forum Discussion

meiwah's avatar
meiwah
Resolver I
10 months ago
Solved

Spark Config for PySpark notebook using SAS

I would like to connect to a parquet in ADLS Gen 2 from a Fabric PySpark notebook. How to set the spark config? Below results in error # Set the authentication type to SAS spark.conf.set(     ...
data engineering
  • v-achippa's avatar
    v-achippa
    9 months ago

    Hi meiwah,

     

    Based on your requirement, you can use Lakehouse shortcuts in Microsoft Fabric to connect to ADLS Gen2 using a SAS token. Please follow below steps:

    • Navigate to the Lakehouse view in Fabric, then add a shortcut by providing the full DFS path of your ADLS Gen2 container.
    • When prompted for the authentication method, select SAS token and paste the token directly into the provided field.
    • Once the shortcut is created, it will mount the ADLS Gen2 container as a folder within your Lakehouse. You can then access this data from a PySpark notebook by adding the Lakehouse to your notebook.
    • After that, navigate to the desired file and use the auto-generated Spark code to read the data seamlessly.
      For example like "df = spark.read.format("parquet").load("Tables/<shortcut-folder>/<file-path>")"

    If possible, consider switching to Service Principal authentication for better security and manageability, especially in cross-tenant scenarios.

     

    Also, go through this similar post which might help you with your query.

    Solved: read Azure Data Lake from notebook fabric - Microsoft Fabric Community

     

    Thanks and regards,

    Anjan Kumar Chippa

     

v-achippa's avatar
v-achippa
Community Support
10 months ago

Hi meiwah,

 

Thank you for reaching out to Microsoft Fabric Community.

 

Thank you BhaveshPatel for the prompt response.

 

As we haven’t heard back from you, we wanted to kindly follow up to check if the solution provided by the user for the issue worked? or let us know if you need any further assistance.

 

Thanks and regards,

Anjan Kumar Chippa

meiwah's avatar
meiwah
Resolver I
9 months ago

Hi Anjan, 

 

It has not been resolved. I would like to use SAS token to access a parquet file in ADLS Gen 2 from a Fabric PySpark notebook but the class org.apache.hadoop.fs.azurebfs.sas.FixedSASTokenProvider to set the config is not available in the Fabric environment as shown in the below error message. Is there another way for SAS to access a ADLS Gen 2? Thanks!

"Unable to load SAS token provider class: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azurebfs.sas.FixedSASTokenProvider not foundjava.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azurebfs.sas.FixedSASTokenProvider not found"

  • v-achippa's avatar
    v-achippa
    Community Support
    9 months ago

    Hi meiwah,

     

    Based on your requirement, you can use Lakehouse shortcuts in Microsoft Fabric to connect to ADLS Gen2 using a SAS token. Please follow below steps:

    • Navigate to the Lakehouse view in Fabric, then add a shortcut by providing the full DFS path of your ADLS Gen2 container.
    • When prompted for the authentication method, select SAS token and paste the token directly into the provided field.
    • Once the shortcut is created, it will mount the ADLS Gen2 container as a folder within your Lakehouse. You can then access this data from a PySpark notebook by adding the Lakehouse to your notebook.
    • After that, navigate to the desired file and use the auto-generated Spark code to read the data seamlessly.
      For example like "df = spark.read.format("parquet").load("Tables/<shortcut-folder>/<file-path>")"

    If possible, consider switching to Service Principal authentication for better security and manageability, especially in cross-tenant scenarios.

     

    Also, go through this similar post which might help you with your query.

    Solved: read Azure Data Lake from notebook fabric - Microsoft Fabric Community

     

    Thanks and regards,

    Anjan Kumar Chippa