Forum Discussion
increase file size
Hello KrishnaMoola,
This is expected behavior. In Microsoft Fabric, dbutils.fs.put() has a hard limit of 100 KB per write and is intended only for small files such as configuration or metadata. When the file size exceeds this limit, no further content is written. The limit cannot be increased.
Microsoft documentation
dbutils.fs.put() is designed for small files only:
https://learn.microsoft.com/azure/databricks/dev-tools/databricks-utils#dbutilsfsFabric Lakehouse file writing best practices:
https://learn.microsoft.com/fabric/data-engineering/lakehouse-overview
Recommended approach
Use Spark DataFrame or RDD writes (or log to a Lakehouse table) to handle larger or growing log files, as these scale without file-size limitations
Hello KrishnaMoola,
This is expected behavior. In Microsoft Fabric, dbutils.fs.put() has a hard limit of 100 KB per write and is intended only for small files such as configuration or metadata. When the file size exceeds this limit, no further content is written. The limit cannot be increased.
Microsoft documentation
dbutils.fs.put() is designed for small files only:
https://learn.microsoft.com/azure/databricks/dev-tools/databricks-utils#dbutilsfsFabric Lakehouse file writing best practices:
https://learn.microsoft.com/fabric/data-engineering/lakehouse-overview
Recommended approach
Use Spark DataFrame or RDD writes (or log to a Lakehouse table) to handle larger or growing log files, as these scale without file-size limitations