Forum Discussion
Fabric CU Leak
Im running 4 background jobs An event stream 2 pipelines running a notebook using pyspark streaming. A pipeline running SQL to copy for Lakehouse to ware house. I have been careful to add tim...
2 of the notebooks are like this.
def write2table(df2, epoch_id😞
df2.write.format("delta").mode("append").partitionBy("partition").save(table_delta_file_location)
df = spark \
.readStream \
.format("eventhubs") \
.options(**ehConf) \
.option("failOnDataLoss", "false") \
.load()
dfa.writeStream \
.outputMode("append") \
.trigger(processingTime='120 seconds') \
.option("mergeSchema", "false") \
.option("checkpointLocation",checkpointLocation) \
.foreachBatch(write2table) \
.start() \
.awaitTermination(590)