Forum Discussion

gstafflannel's avatar
gstafflannel
New Member
10 months ago
Solved

update policy and streaming policy conflict

My issue stems from the fact that an update policy cannot contain a join or lookup on a table with a streaming policy. I saw a post talking about a intermediary table . So I have tried to setup this ...
eventhouse
  • rajendraongole1's avatar
    rajendraongole1
    9 months ago

    @Hi gstafflannel - Faced similar issue . 

    Can you separate streaming ingestion and enrichment which is recommended.

    Update policy validation is based on the source lineage, not just the target table.

    Joins are allowed only when the source table of the update policy is not streaming-derived. By separating ingestion and enrichment, the join happens on a non-streaming table.

    Eg.

    load_stocks
    Streaming ingestion enabled
    Raw data only (no joins / lookups)
    b_stocks_enriched
    Newly created table
    No streaming ingestion ever enabled
    Update policy sourced only from load_stocks (single-table query)
    s_stocks
    Update policy joins b_stocks_enriched with stock_sectors

    Hope this helps

rajendraongole1's avatar
rajendraongole1
Icon for Super User rankSuper User
9 months ago

@Hi gstafflannel - Faced similar issue . 

Can you separate streaming ingestion and enrichment which is recommended.

Update policy validation is based on the source lineage, not just the target table.

Joins are allowed only when the source table of the update policy is not streaming-derived. By separating ingestion and enrichment, the join happens on a non-streaming table.

Eg.

load_stocks
Streaming ingestion enabled
Raw data only (no joins / lookups)
b_stocks_enriched
Newly created table
No streaming ingestion ever enabled
Update policy sourced only from load_stocks (single-table query)
s_stocks
Update policy joins b_stocks_enriched with stock_sectors

Hope this helps