I have several data coming from Dynamics 365 Business Central.
Most of those data I can process/visualize directly using Power BI and dax, but a unfortunately remains a part that requires a complex ETL/preprocessing. I have done the ETL/preprocessing with python and custom libraries (is not a simple script).
I know that I cannot call that code from Power BI, and setup a personal gateway is not what I want because all the pipiline must run and refresh regurarly indipendently from my pc, so the question is, what are my options?
- Initially I thought that using dataflow (and so having data inside CDM folders) would help me, but I could't find a proper way to read the CDM folders from python. I found some code on the Microsoft github repository, but is marked as obsolete:
Azure-Samples/cdm-azure-data-services-integration
I only have found a connector for Apache Spark (it's overkill for my purpose).
- Another possibility is to create a fake machine learning model in **Azure Machine Learning** that just do my stuff, and use it with the integration of PowerBI/Azure Machine Learning. Can it works?
- Another way could be using Azure Data Factory to copy the CDM folder to a Azure Blob Storage, do my stuff and than import the resulting csv into the dataflow. But using this way I see problems in synchronizing all the various steps when I schedule a refresh in Power BI.
Have you experience in some complex scenario like this?
Have you some advice?
Thanks
