DataFlow Gen 2 with KQL Engine Query URI as the destination

[AndrewPage]

I'm trying to implement the Content Delivery Lifecycle using workspace Git integration- see https://learn.microsoft.com/en-us/fabric/cicd/git-integration/intro-to-git-integration

I need the destination of my Data Flow to automatically pick up the right KQL Engine instance + connection, here:

I can make it pick up the cluster URI from a library variable as show above. However, the problem that I have is that the connection name does not populate (also shown above).

When I run the Data Flow (in this state) I get the error below:

[AndrewPage]

I have followed the instructions for setting up the 'destination' of the Dataflow: Dataflow Gen2 data destinations and managed settings - Microsoft Fabric | Microsoft Learn

I am selecting the folowing as the type of destination. I cannot find any others that could be relevant (please advise if there is a better choice that I have not seen):

If I manually enter the cluster URI on the next wizard page, the connection is automatically populated: 

This combination works ok (but is static).

However, when I try to parametrise the connection I encounter problems.

I select a 'workspace variable'.

The problem then is that it prompts for a connection but does not allow me to populate one:

[v-sshirivolu]

Hi @AndrewPage,

Thank you for the additional details and screenshots.

From your description, when the KQL Cluster URI is entered manually, Dataflow Gen2 associates and populates the connection as part of the standard destination configuration flow. However, when the URI is parameterized, the connection is not automatically populated.

According to Microsoft documentation, Dataflow Gen2 supports configuring destinations through the standard setup experience, but not all connection or parameterization scenarios are supported for dynamic resolution.

If this capability is important for your use case, it would be a good idea to raise a Microsoft support ticket or submit a request through Fabric Ideas for further clarification. 

Thank you.

[v-sshirivolu]

Hi @AndrewPage ,

I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions. 

[svelde · Accepted solution]

Hello @AndrewPage ,

what you experience with the connection is default behavior. The connection to an Eventhouse is static.

If you want to distribute difference messages over multiple tables, there are two strategies:

1. Create multiple connections and use staging and branching

I have no experience with this solution and had to google a lot to get this explained.

This could be the solution if you need to split the data towards multiple Eventhouses.

2. use one destionation to a 'source' table in Eventhouse and use table update policies to split the data towards target tables

This is a common scenario in Eventhouse where a medaillion architcture is used to split raw data over multiple silver tables.

The raw/source/bronse table then gets a retention time so raw table rows are automatically deleted over time (otherwise you have duplicate data in both bronse and silver tables). 

This works well within one eventhouse.

--

If this answer helps you, a thumbs-up or marking it as the accepted answer is appreciated. All community members with similar questions will benefit by doing so. Your contribution is highly appreciated.

[svelde]

Hello @AndrewPage 

welcom to this community forum.

The Dataflow Gen2 can forward messages to Eventhouse tables.

There, you can use table update policies to pick up new rows and transform/distribute/enzich them internally to other tables. 

Or you can add Eventhouse CDC to pick up new rows in a Fabric Eventstream.

Finally, check out the Activator integration on the KQL queryset if you want to act on the incoming data.

You can change the retention time of the first table (eg. 30 days) so rows do not stay there for ever if not needed.

--

If this answer helps you, a thumbs-up or marking it as the accepted answer is appreciated. All community members with similar questions will benefit by doing so. Your contribution is highly appreciated.

[v-sshirivolu]

HI @AndrewPage ,
Thanks for reaching out to Community Forum.

According to Microsoft documentation, Dataflow Gen2 supports writing data to Fabric KQL Database destinations through the supported destination configuration experience. The documentation explains configuring KQL Database connections and supported destination settings, but it does not mention using a KQL Engine Query URI directly as a destination endpoint for Dataflow Gen2.

Based on the current documented capabilities, it would be recommended to use the officially supported KQL Database destination configuration method described in Microsoft Learn. If a specific Query URI scenario is required, raising a Microsoft support ticket or Fabric Ideas request may help confirm whether this scenario is currently supported or planned.

Thank you 

[v-sshirivolu]

Hi @AndrewPage ,

I hope the information provided above assists you in resolving the issue. If you have any additional questions or concerns, please do not hesitate to contact us. We are here to support you and will be happy to help with any further assistance you may need.