GCS Export Issue with PySpark Notebook

Hello Harsha_k_111,
Thank you for reaching out to the Microsoft Fabric Community Forum.

we understand that the script is failing because it tries to delete temporary staging files in the GCS path, but your service account lacks delete permissions due to company restrictions.

To resolve this, you can configure the script to write directly to the GCS path without creating temporary files. This can be achieved by adjusting the Spark configurations to use a direct write mode, which avoids the need for delete permissions. Specifically, you can set the output stream type to bypass the default behavior of creating staging files. This should allow the job to complete successfully while adhering to your permission constraints.

Best regards,
Ganesh Singamshetty