Forum Discussion

antoniofarias's avatar
antoniofarias
Frequent Visitor
2 years ago
Solved

Authentication to read data from Lakehouse using ABFS path and python/spark code

I'm trying to read data from lakehouse using a python code, is there any documentation of authentication to use and help me, because I did't found any one.

 

I already created an app registration conected into fabric and have client id, secret value...

 

I know how to write and read with a abfs path but I don't know how to authenticate into.

 

I already found something using the Azure Databricks, but at this time I need to do this without use azure. 

Need Help
Tutorial Requests
  • antoniofarias's avatar
    antoniofarias
    2 years ago

    Hi Anonymous, I'm using vscode + spark(docker) to run the code. I found the problem, it was my firewall blocking, so here is the code to read a file outside a Microsoft service:

     

    from pyspark.sql import SparkSession

spark = (SparkSession.builder.config("spark.jars.packages", "org.apache.hadoop:hadoop-azure:3.3.1")
         .appName("Fabric").getOrCreate())

spark.conf.set("fs.azure.account.auth.type", "OAuth")
spark.conf.set("fs.azure.account.oauth.provider.type", "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider")
spark.conf.set("fs.azure.account.oauth2.client.id", service_principal_id)
spark.conf.set("fs.azure.account.oauth2.client.secret", service_principal_password)
spark.conf.set("fs.azure.account.oauth2.client.endpoint", f"https://login.microsoftonline.com/{tenant_id}/oauth2/token")
spark.conf.set("spark.sql.execution.arrow.pyspark.enabled", "true")
spark.conf.set("spark.sql.files.ignoreCorruptFiles", "true")

table_name = ''

df = spark.read.format("parquet").load(f"abfss://{workspace_name}@onelake.dfs.fabric.microsoft.com/{lakehouse_name}.Lakehouse/Tables/{table_name}/*.parquet")

df.show()

4 Replies