Write delta file Error : Authentication Failed with Bearer token is not present in the request

[tolgakurt]

Hi,

I'm not sure I'm posting this in the right place. Based on here as I couldn't find a forum about PySpark.

In summary I want to write spark dataframe to directory as delta output with pyspark. But I get the error "Authentication Failed with Bearer token is not available in request" and not much is gained from there.

Is there anyone to help?

Thanks,

Tolga

[LineshGajeraq]

I was also able to resolve it as well. Basically, you need to add lakehouse as source in the notebook and it will work

[Anonymous]

I think that's because you haven't specified the area (Files/Tables) to which you want to write the data.

Assuming you want to create a delta table in a managed area of the lakehouse (Tables), try this:

df_bonus.write.format("delta").save("Tables/WriteTest")

Make sure you have your lakehouse pinned on in the Lakehouse explorer on the left.

[LineshGajeraq]

I am having the same issue. I am simply reading a file from Lakehouse, and made sure file exist and path is correct 

df = spark.read.format("csv").option("header","true").load("Files/orders/2019.csv")

display(df)

Getting following error

Py4JJavaError: An error occurred while calling o4726.load. : Operation failed: "Bad Request", 400, HEAD, "http://onelake.dfs.fabric.microsoft.com/XXXXXXXXXXXXXXXXX/user/trusted-service-user/Files/orders/201...

[tolgakurt]

Hi @LineshGajeraq,

I solved this issue many month ago 🙂 Try this pyspark code please.

# Write to SQL Table

import time

start = time.time()

data.write \

  .format("jdbc") \

  .mode("overwrite") \

  .option("url", "jdbc:sqlserver://[Your DB IP];databaseName=[Your Database Name];") \

  .option("dbtable", "[TableName]") \

  .option("user", "") \

  .option("password", "").save()

end = time.time()

print(f"Execution time: {end-start:.2f} second")

If you get an error again, can you let me know?