Forum Discussion
Post ProdigyPerformance about a notebook to read Dynamics 365 Business Central data by using API
Hi pmscorca,
In terms of performance, the Data Pipeline approach generally provides better results. It is optimized for bulk data movement, supports built-in parallelism, and is tuned for efficient handling of OData sources like Dynamics 365 Business Central. This makes it more suitable for high-volume or full-load scenarios.
When considering cost, Data Pipelines again come out ahead. They have lower orchestration overhead and are designed to be lightweight and efficient for recurring ETL tasks. Since they don’t require custom compute or extended execution times like notebooks, they’re usually more cost-effective.
In terms of reliability, especially when dealing with rate-limiting, API throttling, or complex retry strategies, Notebooks are more reliable. They allow full control over retry logic, timeouts, and error handling mechanisms, which is essential for robust data extraction from business APIs.
Finally, when it comes to simplicity, Data Pipelines are the preferred choice. They provide a user-friendly interface, come with built-in OData connectors, and handle authentication and API interactions automatically. This makes them well-suited for quick setup and straightforward data integration tasks without the need for complex configurations.
Thanks,
Prashanth Are
MS fabric community support
Post ProdigyOk, but in terms of performance and costs is it better implementing a notebook that uses the BC API or creating a data pipeline (copy job) that uses the Fabric OData connector? Thanks
Community SupportHi pmscorca,
In terms of performance, the Data Pipeline approach generally provides better results. It is optimized for bulk data movement, supports built-in parallelism, and is tuned for efficient handling of OData sources like Dynamics 365 Business Central. This makes it more suitable for high-volume or full-load scenarios.
When considering cost, Data Pipelines again come out ahead. They have lower orchestration overhead and are designed to be lightweight and efficient for recurring ETL tasks. Since they don’t require custom compute or extended execution times like notebooks, they’re usually more cost-effective.
In terms of reliability, especially when dealing with rate-limiting, API throttling, or complex retry strategies, Notebooks are more reliable. They allow full control over retry logic, timeouts, and error handling mechanisms, which is essential for robust data extraction from business APIs.
Finally, when it comes to simplicity, Data Pipelines are the preferred choice. They provide a user-friendly interface, come with built-in OData connectors, and handle authentication and API interactions automatically. This makes them well-suited for quick setup and straightforward data integration tasks without the need for complex configurations.
Thanks,
Prashanth Are
MS fabric community support
- pmscorca
Ok, therefore to get structured data as those BC ones reading some large table, using Data pipeline with OData connector seems as better solution as possible in terms of performance and costs, isn't it?
- pmscorca
Hi, thinking to use Data Factory it seems that the OData connector is supported by data pipelines and data flows gen2 but not by copy jobs, while the Dynamics 365 Business Central connector is only supported by data flows gen2.
Is correct the official documentation about these connectors?
It could be very useful to use a copy job with these 2 connectors to a BC source.
Thanks