Eventstreams with on-premesis SQL data.

[michaelgambling]

Hi, i am exploring MS eventstreams and came accross this: https://learn.microsoft.com/en-us/fabric/real-time-intelligence/event-streams/add-source-sql-server-...

Its a SQL Server on VM DB (CDC) connector, designed to seemingly utilise SQL CDC services to stream data changes to Fabric.

There is one major floor in this connector, that is you need to make your SQL data source publically exposed. Whitch obviously is a huge no no.

Are there any alternative soloutions to this? I need to try and keep costs down (i.e paying for the Fabric capacitys only)

It may also be worth noting we may end up streaming up to 100 databases that are spread accross 4 on-premis SQL servers...

[v-tsaipranay]

Hi @michaelgambling  ,

Just wanted to check if you had the opportunity to review the suggestion provided?
If the response has addressed your query, please Accept it as a solution so other members can easily find it.

Thank You.

[v-tsaipranay]

Hi @michaelgambling ,

May I ask if you have resolved this issue? If so, please mark the helpful reply and accept it as the solution. This will be helpful for other community members who have similar problems to solve it faster.

Thank you.

[lbendlin]

Eventstreams and eventhouse have an extraordinary high CU cost. That is much more impactful than the security concerns. It's just way too expensive.

[v-tsaipranay · Accepted solution]

Hi @michaelgambling ,

Thank you for reaching out to the Microsoft Fabric Community forum and for looking into the SQL Server (CDC) connector in Microsoft Fabric Eventstreams.

As you mentioned, the connector currently needs the SQL instance to be publicly accessible, which can be a security issue for on-premises setups.

Depending on your real-time requirements, there are some secure and cost-effective alternatives:

For near real-time data (such as updates every few minutes), you can use Dataflows Gen2 with Self-hosted Integration Runtime (SHIR). This allows secure data transfer from your on-prem SQL Server without making it public, since SHIR runs inside your network and uses outbound connections.

If you need true real-time data, you can capture CDC data using Azure Data Factory or Synapse pipelines with SHIR, then send changes to Fabric Eventstreams via the REST API. This keeps your SQL Server private but may require some custom setup, like Azure Functions or Logic Apps.

If your organization uses Azure Arc, it can securely connect on-prem SQL Servers to Azure Event Hubs, which can then integrate with Fabric Eventstreams. This keeps your data secure, but may add some extra infrastructure management if Arc isn’t already in use.

Since you have up to 100 databases across 4 on-prem SQL Servers, SHIR can scale to handle this, but you may need to deploy multiple SHIR nodes or use load balancing for efficient processing.

Hope this helps. Please reach out for further assistance.

If this post helps, then please consider to Accept as the solution to help the other members find it more quickly and a kudos would be appreciated.

Thank you.

[v-tsaipranay]

Hi @michaelgambling ,

I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions. If my response has addressed your query, please accept it as a solution and give a 'Kudos' so other members can easily find it.

Thank you.