Forum Discussion
Spark in Notebook taking more time to process the data.
Hi all, I'm working on a diagnostic log ingestion engine built with PySpark and Delta Lake on Microsoft Fabric. My setup parses incoming ZIP logs from a server, transforms signal data per file in...
Hi NagaRK
Always use single delta table ( opmistic concurrency control ) when writing to a delta table. Also, Remove append and use overwrite table in delta.
spark.createDataFrame(rows, schema).write.format("delta").mode("overwrite").saveAsTable(tableName)We are not using EventHub ( streaming data) so use overwrite ( batch data ). This way its near real time data....minimal latency