Forum Discussion
Fabric CU Leak
Im running 4 background jobs
An event stream
2 pipelines running a notebook using pyspark streaming.
A pipeline running SQL to copy for Lakehouse to ware house.
I have been careful to add timeouts in the pipelines and notebooks
Amount of data is like 20 messages an hour.
What im seeing is during the day the amount of CU these background jobs use increase.
However when i stop and start Fabric instance it goes back down again.
How do i go about solving this ?
4 Replies
Hi bklooste ,
Based on the description, try staggering job start times to distribute the load more evenly throughout the day.
Besides, try to reduce the polling frequency of the event stream and pipelines.
Best Regards,
Wisdom Wu
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.
During day restart same. I cant believe that this is coincidence of more jobs running at the same peak load. There is only 4 jobs and it was doing the same when there were 2 jobs. ( Both pipelines running every 11 minutes running a notebook with a 2 minute trigger).
I know i can prob work around it by running the notebook once per day and leave it running but i want to know why this is happening and how to get detail ?2 of the notebooks are like this.
def write2table(df2, epoch_id😞df2.write.format("delta").mode("append").partitionBy("partition").save(table_delta_file_location)df = spark \.readStream \.format("eventhubs") \.options(**ehConf) \.option("failOnDataLoss", "false") \.load()dfa.writeStream \.outputMode("append") \.trigger(processingTime='120 seconds') \.option("mergeSchema", "false") \.option("checkpointLocation",checkpointLocation) \.foreachBatch(write2table) \.start() \.awaitTermination(590)staggering did not help
