Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Get certified in Microsoft Fabric—for free! For a limited time, the Microsoft Fabric Community team will be offering free DP-600 exam vouchers. Prepare now

Reply
tolgakurt
Frequent Visitor

Write delta file Error : Authentication Failed with Bearer token is not present in the request

Hi,

I'm not sure I'm posting this in the right place. Based on here as I couldn't find a forum about PySpark.

In summary I want to write spark dataframe to directory as delta output with pyspark. But I get the error "Authentication Failed with Bearer token is not available in request" and not much is gained from there.

Is there anyone to help?

 

Thanks,

Tolga

 

Screenshot 2023-07-18 173721.png

4 REPLIES 4
LineshGajeraq
Microsoft Employee
Microsoft Employee

I was also able to resolve it as well. Basically, you need to add lakehouse as source in the notebook and it will work

 

LineshGajeraq_0-1724941090916.png

 

Anonymous
Not applicable

I think that's because you haven't specified the area (Files/Tables) to which you want to write the data.

 

Assuming you want to create a delta table in a managed area of the lakehouse (Tables), try this:

df_bonus.write.format("delta").save("Tables/WriteTest")

Make sure you have your lakehouse pinned on in the Lakehouse explorer on the left.

I am having the same issue. I am simply reading a file from Lakehouse, and made sure file exist and path is correct 

 

df = spark.read.format("csv").option("header","true").load("Files/orders/2019.csv")
display(df)
 
Getting following error
Py4JJavaError: An error occurred while calling o4726.load. : Operation failed: "Bad Request", 400, HEAD, "http://onelake.dfs.fabric.microsoft.com/XXXXXXXXXXXXXXXXX/user/trusted-service-user/Files/orders/201...
 

Hi @LineshGajeraq,

 

I solved this issue many month ago 🙂 Try this pyspark code please.

 

# Write to SQL Table
import time
start = time.time()

data.write \
  .format("jdbc") \
  .mode("overwrite") \
  .option("url", "jdbc:sqlserver://[Your DB IP];databaseName=[Your Database Name];") \
  .option("dbtable", "[TableName]") \
  .option("user", "") \
  .option("password", "").save()
end = time.time()
print(f"Execution time: {end-start:.2f} second")
 
If you get an error again, can you let me know?

Helpful resources

Announcements
Oct Fabric Update Carousel

Fabric Monthly Update - October 2024

Check out the October 2024 Fabric update to learn about new features.

September Hackathon Carousel

Microsoft Fabric & AI Learning Hackathon

Learn from experts, get hands-on experience, and win awesome prizes.

October NL Carousel

Fabric Community Update - October 2024

Find out what's new and trending in the Fabric Community.