Initial load of spark streams

[innocence84]

Hi all,

Operational system is sending transaction data to Kafka topic, then it is ingested into bronze layer in the lakehouse by using Spark Streaming. Example table name is SALES_delta

Operational system gets historical data from SQL Server. Then we are ingesting this into bronze layer by using Spark batch processing. Example table name is SALES.

In this architecture we have two sales table, SALES Table is loaded once and SALES_delta is loaded whenever new sales transactions occured.

When and how will we combine these two tables so we may have one table for sales transactions?

[v-shamiliv]

Hi @innocence84 
I hope this information is helpful. Please let me know if you have any further questions or if you'd like to discuss this further. If this answers your question, please Accept it as a solution and give it a 'Kudos' so others can find it easily.
Thank you.

[v-shamiliv]

Hi @innocence84 
I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions. If my response has addressed your query, please accept it as a solution and give a 'Kudos' so other members can easily find it.
Thank you.

[v-shamiliv]

Hi @innocence84 

May I ask if you have resolved this issue? If so, please mark the helpful reply and accept it as the solution. This will be helpful for other community members who have similar problems to solve it faster.

Thank you.

[v-shamiliv · Accepted solution]

Hi  @innocence84 
Thank you for reaching out microsoft fabric community forum.

You have two tables:

• SALES: one-time historical load from SQL Server
• SALES_delta: ongoing new transactions via streaming from Kafka

To get a complete view of sales, you should combine them into a single table (e.g., SALES_silver) after the historical load is done and streaming is stable.

How to combine:

• If SALES_delta only has new rows, use INSERT to append to SALES_silver.
• If it includes updates or late data, use a Delta MERGE to handle both inserts and updates.

This unified table (SALES_silver) becomes your single source of truth, and you can stop querying SALES and SALES_delta separately.
If this solution helps, please consider giving us Kudos and accepting it as the solution so that it may assist other members in the community
Thank you.