Forum Discussion

meiwah's avatar
meiwah
Resolver I
10 months ago
Solved

Spark Config for PySpark notebook using SAS

I would like to connect to a parquet in ADLS Gen 2 from a Fabric PySpark notebook. How to set the spark config? Below results in error # Set the authentication type to SAS spark.conf.set(     ...
data engineering
  • v-achippa's avatar
    v-achippa
    9 months ago

    Hi meiwah,

     

    Based on your requirement, you can use Lakehouse shortcuts in Microsoft Fabric to connect to ADLS Gen2 using a SAS token. Please follow below steps:

    • Navigate to the Lakehouse view in Fabric, then add a shortcut by providing the full DFS path of your ADLS Gen2 container.
    • When prompted for the authentication method, select SAS token and paste the token directly into the provided field.
    • Once the shortcut is created, it will mount the ADLS Gen2 container as a folder within your Lakehouse. You can then access this data from a PySpark notebook by adding the Lakehouse to your notebook.
    • After that, navigate to the desired file and use the auto-generated Spark code to read the data seamlessly.
      For example like "df = spark.read.format("parquet").load("Tables/<shortcut-folder>/<file-path>")"

    If possible, consider switching to Service Principal authentication for better security and manageability, especially in cross-tenant scenarios.

     

    Also, go through this similar post which might help you with your query.

    Solved: read Azure Data Lake from notebook fabric - Microsoft Fabric Community

     

    Thanks and regards,

    Anjan Kumar Chippa

     

BhaveshPatel's avatar
BhaveshPatel
Super User
10 months ago

Hi meiwah 

 

Is it Azure Spark ( Pyspark ) or Delta Lake ( both are Parquet files except Delta lake has transaction log with parquet).

 

Also, Please include why do you need to connect to ADLS Gen 2 ( Storage ).   You first need to connect to Azure Storage Explorer.

 

 

 

Also You need a GroupID and WorkspaceID to connect to Microsoft Fabric to Azure Storage Explorer.

 

https://app.fabric.microsoft.com/groups/08946547d-e0b7-6578-b5ff-ff4a96567753056/synapsenotebooks/5a4564459c-7721-4817-be56-88758a9085b2d?redirectedFromSignup=1&experience=fabric-developer

  • meiwah's avatar
    meiwah
    Resolver I
    10 months ago

    Hi Bhavesh, I'm running the code in PySpark notebook in Fabric environment, not in Azure. It is just a parquet and not delta. The reason for connecting to the ADLS is to receive the data there. And I'm using a SAS token becos it is cross tenant, meaning the fabric and adls are of different tenant