Forum Discussion
DataFlow Gen 2 with KQL Engine Query URI as the destination
Hello AndrewPage ,
what you experience with the connection is default behavior. The connection to an Eventhouse is static.
If you want to distribute difference messages over multiple tables, there are two strategies:
1. Create multiple connections and use staging and branching
I have no experience with this solution and had to google a lot to get this explained.
This could be the solution if you need to split the data towards multiple Eventhouses.
2. use one destionation to a 'source' table in Eventhouse and use table update policies to split the data towards target tables
This is a common scenario in Eventhouse where a medaillion architcture is used to split raw data over multiple silver tables.
The raw/source/bronse table then gets a retention time so raw table rows are automatically deleted over time (otherwise you have duplicate data in both bronse and silver tables).
This works well within one eventhouse.
--
If this answer helps you, a thumbs-up or marking it as the accepted answer is appreciated. All community members with similar questions will benefit by doing so. Your contribution is highly appreciated.
I have followed the instructions for setting up the 'destination' of the Dataflow: Dataflow Gen2 data destinations and managed settings - Microsoft Fabric | Microsoft Learn
I am selecting the folowing as the type of destination. I cannot find any others that could be relevant (please advise if there is a better choice that I have not seen):
If I manually enter the cluster URI on the next wizard page, the connection is automatically populated:
This combination works ok (but is static).
However, when I try to parametrise the connection I encounter problems.
I select a 'workspace variable'.
The problem then is that it prompts for a connection but does not allow me to populate one:
- svelde
Super User
Hello AndrewPage ,
what you experience with the connection is default behavior. The connection to an Eventhouse is static.
If you want to distribute difference messages over multiple tables, there are two strategies:
1. Create multiple connections and use staging and branching
I have no experience with this solution and had to google a lot to get this explained.
This could be the solution if you need to split the data towards multiple Eventhouses.
2. use one destionation to a 'source' table in Eventhouse and use table update policies to split the data towards target tables
This is a common scenario in Eventhouse where a medaillion architcture is used to split raw data over multiple silver tables.
The raw/source/bronse table then gets a retention time so raw table rows are automatically deleted over time (otherwise you have duplicate data in both bronse and silver tables).
This works well within one eventhouse.
--
If this answer helps you, a thumbs-up or marking it as the accepted answer is appreciated. All community members with similar questions will benefit by doing so. Your contribution is highly appreciated.
Hi AndrewPage,Thank you for the additional details and screenshots.From your description, when the KQL Cluster URI is entered manually, Dataflow Gen2 associates and populates the connection as part of the standard destination configuration flow. However, when the URI is parameterized, the connection is not automatically populated.According to Microsoft documentation, Dataflow Gen2 supports configuring destinations through the standard setup experience, but not all connection or parameterization scenarios are supported for dynamic resolution.If this capability is important for your use case, it would be a good idea to raise a Microsoft support ticket or submit a request through Fabric Ideas for further clarification.
Thank you.
Hi AndrewPage ,
I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions.