Migrating to Fabric RTI from ADX

[jenbeiser]

Benefits of Migrating to Fabric RTI

Migrating to Microsoft Fabric Real-Time Intelligence offers several benefits for organizations looking to enhance their data analytics capabilities:

• A comprehensive SaaS offering: An all-encompassing solution that facilitates the discovery of insights from your time-sensitive data, allowing you to ingest, process, query, visualize, and act upon it in real-time.
• A centralized hub for your dynamic data: A unified data estate for all your event data in motion, simplifying the ingestion, storage, and curation of fine-grained data from across your organization through the Real-Time Hub.
• Rapid solution development: Empower team members of varying expertise to extract more value from data and quickly build solutions on top it for further business growth.
• Insights powered by real-time AI: Scale manual monitoring and effortlessly initiate actions with ready-to-use, automated features that uncover hidden patterns, and fully use the Microsoft ecosystem to drive your business forward.

Azure PaaS vs. Fabric RTI solution architecture

Overall Feature Comparison

Capability	Azure PaaS-based solution	Real-time Intelligence solution
Integration of services	Depends on integration compatibility between the services in scope of the architecture.	One-click integration at each step of data ingestion, process, analyze, visualize, and act.
Pro and citizen dev experience	More suitable for pro developers.	Pro developers, citizen developers, and business users can coexist.
Low-code/No-code	Only available for transformation in Azure Stream Analytics and for creating alerts using Logics Apps or Power Automate. Pro development required for end-to-end implementation.	End-to-end implementation from ingestion to analyze to transform to visualize and act can be realized.
Consumption Model	Service dependent estimation, consumption, and billing model.	Uniform Fabric Capacity Unit consumption and billing model.
Multicloud connectors	Azure Stream Analytics connects to Confluent Kafka. No connectors to read data from Amazon Kinesis or Google Pub/Sub.	Native integration for Confluent Kafka, Amazon Kinesis, Google Pub/Sub.
Support for CDC streams	Requires deployment of other services such as Debezium.	Native integration for Azure Cosmos DB, PostgreSQL, MySQL DB, and Azure SQL.
Support for protocols	Azure Event Hubs, AMQP, Kafka, and MQTT.	Azure Event Hubs, AMQP, Kafka.
Data profiling	Not available	Data profiling view of your real-time tables provides out-of-the-box histograms and min-max ranges for each column.
Visual data exploration	Not available	Drag-and-drop features to visually analyze your real-time data.
Copilot experience	Azure Data Explorer cluster can be added as a source in Fabric KQL Queryset to use Copilot capabilities.	Natively available
Built-in ML models	Anomaly detection and forecasting models available. Pro development required to deploy anomaly detection and forecasting models.	Anomaly detection and forecasting models available. Business users can also apply anomaly detection models on incoming streaming data.
Visualization (Microsoft)	Power BI, Azure Data Explorer dashboards	Native one-click integration with Power BI and Real-Time Dashboard.
Visualization (Third party)	Grafana, Kibana, Matlab.	Grafana, Kibana, Matlab can also be integrated with Eventhouse.
Driving business actions from insights	Requires Azure Logic Apps or Power Automate or Azure Functions, Azure Monitor alerts.	Natively available in Fabric using Fabric Activator items with out-of-the-box integration with Power BI semantic models, Eventstream, and KQL queries.
Reactive system events	Not available	Built-in events published through Real-Time hub; Use Activator items to automate data processes, such as pipelines and notebooks.
Real-time Semantic Models	Not available or code-first solution using Logic Apps or Azure Functions	Not available
Built-in AI	Not available	Not available
Notification destinations	Depends on the connector portfolio of the service.	Microsoft Teams, Microsoft Outlook, and Power Automate connectors.
Driving business actions from insights	Requires Azure Logic Apps or Power Automate or Azure Functions, Azure Monitor alerts.	Natively available in Fabric using Fabric Activator items with out-of-the-box integration with Power BI semantic models, Eventstream, and KQL queries.
Unified catalog of data streams	Not available	Real-time hub
Discovery of Microsoft data streams	Not available	Real-time Intelligence hub discovers data streams in your Azure tenant.
Capture and act on events from Azure Storage	Requires deploying Azure Event Grid to act on events occurring in Azure Storage.	Azure Blob Storage event-based triggers available.
Capture and act on events from Fabric	Not applicable	Natively available in Fabric

Migration Script + Fabric Pipeline

Repo: iamjenetzler/ADXtoRTI: Script to migrate schema from ADX database to Fabric RTI KQL database

• Purpose: PowerShell script to migrate schema from an ADX database to a Fabric RTI KQL database.
• Prerequisites:
  • Azure Service Principal with specific API permissions.
  • Reader access on ADX Clusters.
  • Security Group setup in Fabric admin portal.
  • Kusto .NET client libraries installed.
• Steps:
  • Retrieve Fabric access token.
  • Connect to source ADX cluster and database.
  • Add service principal as admin to source database.
  • Create or verify Eventhouse in Fabric Workspace.
• Data Movement:
  • Use a copy data pipeline in Fabric Data Factory to move data from ADX to Fabric RTI.