Forum Discussion

meiwah's avatar
meiwah
Icon for Resolver I rankResolver I
10 months ago
Solved

Spark Config for PySpark notebook using SAS

I would like to connect to a parquet in ADLS Gen 2 from a Fabric PySpark notebook. How to set the spark config? Below results in error # Set the authentication type to SAS spark.conf.set(     ...
data engineering
  • v-achippa's avatar
    v-achippa
    10 months ago

    Hi meiwah,

     

    Based on your requirement, you can use Lakehouse shortcuts in Microsoft Fabric to connect to ADLS Gen2 using a SAS token. Please follow below steps:

    • Navigate to the Lakehouse view in Fabric, then add a shortcut by providing the full DFS path of your ADLS Gen2 container.
    • When prompted for the authentication method, select SAS token and paste the token directly into the provided field.
    • Once the shortcut is created, it will mount the ADLS Gen2 container as a folder within your Lakehouse. You can then access this data from a PySpark notebook by adding the Lakehouse to your notebook.
    • After that, navigate to the desired file and use the auto-generated Spark code to read the data seamlessly.
      For example like "df = spark.read.format("parquet").load("Tables/<shortcut-folder>/<file-path>")"

    If possible, consider switching to Service Principal authentication for better security and manageability, especially in cross-tenant scenarios.

     

    Also, go through this similar post which might help you with your query.

    Solved: read Azure Data Lake from notebook fabric - Microsoft Fabric Community

     

    Thanks and regards,

    Anjan Kumar Chippa