Forum Discussion
Large Parquet files 2GB
Hi Anonymous , Thank you for reaching out to the Microsoft Community Forum.
This may not because of a size limit, but due to how Spark handles execution and authentication. Microsoft Fabric uses time-limited tokens for accessing shortcut-linked storage (like S3) and Spark delays file access until an action is triggered. If this delay exceeds the token's validity, the job fails with an authentication error, which is exactly what you're seeing.
In notebooks, that token is scoped to the session and doesn’t auto-refresh once expired. You should force Spark to access the file immediately after reading, while the token is still valid. You can do this by caching and counting the DataFrame before any further transformation or by writing a python code snippet to ensure the file is accessed while the token is still valid, avoiding deferred failures.
If this helped solve the issue, please consider marking it “Accept as Solution” and giving a ‘Kudos’ so others with similar queries may find it more easily. If not, please share the details, always happy to help.
Thank you.
Community SupportHi Anonymous , Thank you for reaching out to the Microsoft Community Forum.
This may not because of a size limit, but due to how Spark handles execution and authentication. Microsoft Fabric uses time-limited tokens for accessing shortcut-linked storage (like S3) and Spark delays file access until an action is triggered. If this delay exceeds the token's validity, the job fails with an authentication error, which is exactly what you're seeing.
In notebooks, that token is scoped to the session and doesn’t auto-refresh once expired. You should force Spark to access the file immediately after reading, while the token is still valid. You can do this by caching and counting the DataFrame before any further transformation or by writing a python code snippet to ensure the file is accessed while the token is still valid, avoiding deferred failures.
If this helped solve the issue, please consider marking it “Accept as Solution” and giving a ‘Kudos’ so others with similar queries may find it more easily. If not, please share the details, always happy to help.
Thank you.