Forum Discussion
Relational database as a destination in Fabric
Hi Team,
We are looking for a relational database, usually a Azure SQL database in Fabric to manage the ETL metadata, configuration with small size storage to store those kind of reference or meta data configurations. At the moment non of the destinations support to store the configurations as those are specialized in different purposes.
- Data lakehouse - big data analytics with Spark
- Data warehouse - designed for data wahouse schema
- KQL database - real time analytics.
But there is no relational database engine to manage small scale processes. Really need a such database with in the Fabric context becuase if we use a ASQL db, it is out of the Fabric.
Does anybody know when it will be available if it is in the read map ?
Thanks,
Priyan.
Why not just use a Warehouse table, Lakehouse table or even a csv or json in Lakehouse files?
Why can't you use a Delta Tables on the Lakehouse? They are ACID-compliant and relational. Just because they are capable of handling large datasets does not mean they are incapable of handling small datasets.
If you are just reading meta and configurations, you don't even need a relational database. You can just write that data to a file (XML, JSON, CSV, etc.) in a Lakehouse and read that file whenever you need it.
If it's smaller than 5mb, you could even store it in a variable in a data pipeline and retrieve it in real time when you need to refresh it.
5 Replies
Why not just use a Warehouse table, Lakehouse table or even a csv or json in Lakehouse files?
Why can't you use a Delta Tables on the Lakehouse? They are ACID-compliant and relational. Just because they are capable of handling large datasets does not mean they are incapable of handling small datasets.
If you are just reading meta and configurations, you don't even need a relational database. You can just write that data to a file (XML, JSON, CSV, etc.) in a Lakehouse and read that file whenever you need it.
If it's smaller than 5mb, you could even store it in a variable in a data pipeline and retrieve it in real time when you need to refresh it.Hi priyanfabriccom ,
Currently, none of Fabric is specifically designed for small relational database processes, and I have an alternative:
You can mirror an Azure SQL database in Microsoft Fabric to easily replicate the data that resides in the Azure SQL database directly into OneLake in the fabric in near real-time, without the need for a complex ETL process.
For more information on how to set up and use Azure SQL Database mirroring in the fabric, please see:
Best Regards,
Yang
Community Support TeamIf there is any post helps, then please consider Accept it as the solution to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!Hi Yang,
Thank you for the response. But my concern is not the data ingestion from Azure SQL database or SQL database. Just to have small scale relational database to store simple configurations which are required for ETL automation. I can understand the use case of ASQL replication into OneLake.
Thanks
Priyan.
Hi priyanfabriccom ,
Thanks for the reply from frithjof_v / IntegrateGuru .
Is my follow-up just to ask if the problem has been solved?
If so, can you accept the correct answer as a solution or share your solution to help other members find it faster?
Thank you very much for your cooperation!
Best Regards,
Yang
Community Support TeamIf there is any post helps, then please consider Accept it as the solution to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!
