Forum Discussion
DataFlow Gen 2 with KQL Engine Query URI as the destination
Hello AndrewPage ,
what you experience with the connection is default behavior. The connection to an Eventhouse is static.
If you want to distribute difference messages over multiple tables, there are two strategies:
1. Create multiple connections and use staging and branching
I have no experience with this solution and had to google a lot to get this explained.
This could be the solution if you need to split the data towards multiple Eventhouses.
2. use one destionation to a 'source' table in Eventhouse and use table update policies to split the data towards target tables
This is a common scenario in Eventhouse where a medaillion architcture is used to split raw data over multiple silver tables.
The raw/source/bronse table then gets a retention time so raw table rows are automatically deleted over time (otherwise you have duplicate data in both bronse and silver tables).
This works well within one eventhouse.
--
If this answer helps you, a thumbs-up or marking it as the accepted answer is appreciated. All community members with similar questions will benefit by doing so. Your contribution is highly appreciated.
I have followed the instructions for setting up the 'destination' of the Dataflow: Dataflow Gen2 data destinations and managed settings - Microsoft Fabric | Microsoft Learn
I am selecting the folowing as the type of destination. I cannot find any others that could be relevant (please advise if there is a better choice that I have not seen):
If I manually enter the cluster URI on the next wizard page, the connection is automatically populated:
This combination works ok (but is static).
However, when I try to parametrise the connection I encounter problems.
I select a 'workspace variable'.
The problem then is that it prompts for a connection but does not allow me to populate one:
Super UserHello AndrewPage ,
what you experience with the connection is default behavior. The connection to an Eventhouse is static.
If you want to distribute difference messages over multiple tables, there are two strategies:
1. Create multiple connections and use staging and branching
I have no experience with this solution and had to google a lot to get this explained.
This could be the solution if you need to split the data towards multiple Eventhouses.
2. use one destionation to a 'source' table in Eventhouse and use table update policies to split the data towards target tables
This is a common scenario in Eventhouse where a medaillion architcture is used to split raw data over multiple silver tables.
The raw/source/bronse table then gets a retention time so raw table rows are automatically deleted over time (otherwise you have duplicate data in both bronse and silver tables).
This works well within one eventhouse.
--
If this answer helps you, a thumbs-up or marking it as the accepted answer is appreciated. All community members with similar questions will benefit by doing so. Your contribution is highly appreciated.