The error you're seeing (StreamConstraintsException: (String length exceeds maximum) happens because PySpark is trying to work with a string that is longer than Fabric's 20, 000, 000 character limit in a single column. This usually comes from big text columns, JSON, or poorly optimized queries.
Possible Solutions:
Truncate or Substring Large Text Fields
If you have large text fields in your data, try to limit the size of text fields before querying:
python
from pyspark.sql.functions import expr
df = df.withColumn("truncatedcolumn", expr("substring(longtextcolumn, 1, 5000)"))
Then, drop the original long column before running queries:
python
df = df.drop("longtextcolumn")
Enhance String Length Limit in PySpark (Partial Success).
Try increasing the max result size or adjusting spark.sql.broadcastTimeout:
python
spark. conf.set("spark.driver.maxResultSize", "4g")
spark. conf.set("spark.sql.broadcastTimeout", "600")
But note: Fabric might still enforce a hard limit.
Convert Data to a More Efficient Format
When working with JSON or large text files, think of:.
Converting the fields to Parquet/Delta format that store the fields in the native format of strings, rather than directly storing the raw strings.
Splitting text into multiple smaller chunks.
Work on chunks rather than one big query.
If your query retrieves a very large amount of data, split it up using filtering or pagination:
python
df = spark.read.format("delta").load("yourpath").filter("id BETWEEN 1 AND 1000")
Use UDFs to Handle Large Text Processing
Please use a UDF (user defined function) to split or condense the text before feeding it to Fabric.
Answers to Your Questions:
1 Is it possible to configure Fabric so strings can be much longer?
Not directly. There is a 20, 000, 000 character limit at the system level. It is better to shorten strings prior to querying.
2. Is this related to PySpark serialization limits?
Yes probably because of PySpark JSON serialization limits when transferring data between nodes. Optimizing your query and reducing large strings should help.
3 Has anyone else faced this issue?
Yes! A lot of Fabric users have seen this problem and solved it by shortening text, processing data in chunks, or rewriting queries.
