Forum Discussion

meiwah's avatar
meiwah
Resolver I
10 months ago
Solved

Spark Config for PySpark notebook using SAS

I would like to connect to a parquet in ADLS Gen 2 from a Fabric PySpark notebook. How to set the spark config? Below results in error # Set the authentication type to SAS spark.conf.set(     ...
data engineering
  • v-achippa's avatar
    v-achippa
    9 months ago

    Hi meiwah,

     

    Based on your requirement, you can use Lakehouse shortcuts in Microsoft Fabric to connect to ADLS Gen2 using a SAS token. Please follow below steps:

    • Navigate to the Lakehouse view in Fabric, then add a shortcut by providing the full DFS path of your ADLS Gen2 container.
    • When prompted for the authentication method, select SAS token and paste the token directly into the provided field.
    • Once the shortcut is created, it will mount the ADLS Gen2 container as a folder within your Lakehouse. You can then access this data from a PySpark notebook by adding the Lakehouse to your notebook.
    • After that, navigate to the desired file and use the auto-generated Spark code to read the data seamlessly.
      For example like "df = spark.read.format("parquet").load("Tables/<shortcut-folder>/<file-path>")"

    If possible, consider switching to Service Principal authentication for better security and manageability, especially in cross-tenant scenarios.

     

    Also, go through this similar post which might help you with your query.

    Solved: read Azure Data Lake from notebook fabric - Microsoft Fabric Community

     

    Thanks and regards,

    Anjan Kumar Chippa

     

meiwah's avatar
meiwah
Resolver I
10 months ago

Hi Anjan, 

 

It has not been resolved. I would like to use SAS token to access a parquet file in ADLS Gen 2 from a Fabric PySpark notebook but the class org.apache.hadoop.fs.azurebfs.sas.FixedSASTokenProvider to set the config is not available in the Fabric environment as shown in the below error message. Is there another way for SAS to access a ADLS Gen 2? Thanks!

"Unable to load SAS token provider class: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azurebfs.sas.FixedSASTokenProvider not foundjava.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azurebfs.sas.FixedSASTokenProvider not found"

v-achippa's avatar
v-achippa
Community Support
9 months ago

Hi meiwah,

 

Based on your requirement, you can use Lakehouse shortcuts in Microsoft Fabric to connect to ADLS Gen2 using a SAS token. Please follow below steps:

  • Navigate to the Lakehouse view in Fabric, then add a shortcut by providing the full DFS path of your ADLS Gen2 container.
  • When prompted for the authentication method, select SAS token and paste the token directly into the provided field.
  • Once the shortcut is created, it will mount the ADLS Gen2 container as a folder within your Lakehouse. You can then access this data from a PySpark notebook by adding the Lakehouse to your notebook.
  • After that, navigate to the desired file and use the auto-generated Spark code to read the data seamlessly.
    For example like "df = spark.read.format("parquet").load("Tables/<shortcut-folder>/<file-path>")"

If possible, consider switching to Service Principal authentication for better security and manageability, especially in cross-tenant scenarios.

 

Also, go through this similar post which might help you with your query.

Solved: read Azure Data Lake from notebook fabric - Microsoft Fabric Community

 

Thanks and regards,

Anjan Kumar Chippa