topic Re: Geeting error while overwrite the data in delta table in Developer

Geeting error while overwrite the data in delta table

Udaysutar28apr — Mon, 18 Nov 2024 18:47:37 GMT

In fabric notebook I'm overwriting data in delta table but now getting error :

Please help me to fix this error

org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 255.0 failed 4 times, most recent failure: Lost task 0.3 in stage 255.0 (TID 3314) (vm-e7066017 executor 1): org.apache.spark.SparkFileNotFoundException: Operation failed: "Not Found", 404, HEAD,

Re: Geeting error while overwrite the data in delta table

Anonymous — Tue, 19 Nov 2024 06:50:39 GMT

Hi, @Udaysutar28apr
Regarding the issue you raised, my solution is as follows:

1.Firstly, based on your error message, the most straightforward explanation may be that the file does not exist. Therefore, please ensure that the file path specified in your code is correct and that the file is indeed present at that location.

2.Next, check whether you have sufficient permissions to access and modify the Delta table. Ensure that your Spark job has the appropriate access rights.

3.Finally, if you are using Microsoft Fabric, make sure that the DirectLake feature is enabled. This can be configured after creating a report in Power BI and may resolve any connection issues with the Delta table.

Here are some links related to similar issues you may find helpful:

Solved: Issue with Delta Table Overwrite in Lakehouse Once... - Microsoft Fabric Community
Of course, if you have any new discoveries or questions, please feel free to get in touch with us.

Best Regards,

Leroy Lu

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Re: Geeting error while overwrite the data in delta table

Udaysutar28apr — Tue, 19 Nov 2024 07:23:23 GMT

I have all permisions. I'm overwiting data frequently in delta table. sometime working fine but some time failure due to 404 error . Please suggest how to proceed with this .

Re: Geeting error while overwrite the data in delta table

Anonymous — Tue, 19 Nov 2024 07:37:30 GMT

Hi, @Udaysutar28apr

Thank you for your prompt response.

If this is a temporary issue, we cannot rule out the possibility that it may be caused by network latency, as all changes made on the service require network transmission for updates.

Of course, you might also consider employing a controlled variable approach to document the reasons behind the occasional successes.

Best Regards,

Leroy Lu

Re: Geeting error while overwrite the data in delta table

Udaysutar28apr — Tue, 19 Nov 2024 07:43:19 GMT

No i dont think continous network issue is there .I'm executing pipeline every 10 minutes and every hour's we are getting 3-4 failure out of 6 runs. lets take look on this , Please suggest how overcome this.