Forum Discussion

jcamilo1985's avatar
jcamilo1985
Helper III
2 years ago
Solved

read Azure Data Lake from notebook fabric

Good morning The question is simple, I fear that the solution is not so simple, someone has been able to connect from a factory notebook to an Azure Data Lake Storage Gen2 without having to perform ...
data engineering
  • govindarajan_d's avatar
    govindarajan_d
    2 years ago

    Hi jcamilo1985,

     

    You need to go to Lakehouse view and a new shortcut: 

    1. Specify the DFS path of the storage account (you can get it from the endpoint properties of the storage account)

    2. Authentication method - You can either use your own org account or use a SAS token or service principal account. I would suggest service principal account as the best practice.

    3. In the next screen, add a specific container as the path and then click on create. 

    4. Once added, it will be shown under unidentified folder as a shorcut with the storage account name. You will be able to see all the files underneath that. 

     

    5. To use it in Spark notebook, add the corresponding lakehouse in the notebook and then navigate to the file you want to load and click on the '...' and then you will be able to see this option for loading the data using Spark or Pandas. You can use either of them and the code for loading them in a dataframe will automatically be generated. 

     

    This is the Pyspark code that was generated for the 1.csv file I had shown

    spark.read.format("csv").option("header","true").load("Tables/satrainingdc/week3/files/1.csv")