Forum Discussion

innocence84's avatar
innocence84
Icon for Helper I rankHelper I
1 year ago
Solved

Initial load of spark streams

Hi all, Operational system is sending transaction data to Kafka topic, then it is ingested into bronze layer in the lakehouse by using Spark Streaming. Example table name is SALES_delta Operational...
  • Anonymous's avatar
    Anonymous
    1 year ago

    Hi  innocence84 
    Thank you for reaching out microsoft fabric community forum.

    You have two tables:

    • SALES: one-time historical load from SQL Server
    • SALES_delta: ongoing new transactions via streaming from Kafka

    To get a complete view of sales, you should combine them into a single table (e.g., SALES_silver) after the historical load is done and streaming is stable.

    How to combine:

    • If SALES_delta only has new rows, use INSERT to append to SALES_silver.
    • If it includes updates or late data, use a Delta MERGE to handle both inserts and updates.

    This unified table (SALES_silver) becomes your single source of truth, and you can stop querying SALES and SALES_delta separately.
    If this solution helps, please consider giving us Kudos and accepting it as the solution so that it may assist other members in the community
    Thank you.