Forum Discussion
Data replication from AKS
- 5 months ago
Hello Asam3833
The best approach for moving your data to Fabric depends on the type of data you have.
If your data includes things like system activity, logs, or real-time information from services or platforms such as Kafka, Event Hubs, or IoT Hub, then choose Events. This is ideal when you want to analyse data as it arrives, receive alerts, or quickly send information to tools like KQL, Lakehouse, Notebooks, or Power BI. You can send data from your applications using your current systems, set up simple data flows without coding, and make your data available in OneLake by enabling the right settings.
If your data is already organised in batches, such as Parquet, CSV, or JSON files, choose Files. This works well for jobs that process data in groups, providing a cost-effective and efficient way to move large amounts of information. You can upload files directly to OneLake using familiar methods, and use standard authentication. This is commonly used in automated data pipelines.
If you need to manage your data storage or trigger updates, use the Fabric APIs. This is suitable for tasks like creating or updating Lakehouses, or running table updates, but not for adding individual rows. If your systems are already saving files in cloud storage, you can make them available in Fabric immediately using OneLake Shortcuts, without extra steps. For private storage, use Trusted Workspace Access.
Write Parquet or Delta files to ADLS Gen2, then use a OneLake Shortcut to expose the data in Fabric.
Send events to Event Hubs/Kafka → use Fabric Eventstream → land in Lakehouse.
Don’t use Fabric APIs for data ingestion.
If you tell me whether your K8s workload is streaming or batch, I can give you a very concrete architecture.