Forum Discussion
Post ProdigyPerformance about a notebook to read Dynamics 365 Business Central data by using API
Hi pmscorca,
In terms of performance, the Data Pipeline approach generally provides better results. It is optimized for bulk data movement, supports built-in parallelism, and is tuned for efficient handling of OData sources like Dynamics 365 Business Central. This makes it more suitable for high-volume or full-load scenarios.
When considering cost, Data Pipelines again come out ahead. They have lower orchestration overhead and are designed to be lightweight and efficient for recurring ETL tasks. Since they don’t require custom compute or extended execution times like notebooks, they’re usually more cost-effective.
In terms of reliability, especially when dealing with rate-limiting, API throttling, or complex retry strategies, Notebooks are more reliable. They allow full control over retry logic, timeouts, and error handling mechanisms, which is essential for robust data extraction from business APIs.
Finally, when it comes to simplicity, Data Pipelines are the preferred choice. They provide a user-friendly interface, come with built-in OData connectors, and handle authentication and API interactions automatically. This makes them well-suited for quick setup and straightforward data integration tasks without the need for complex configurations.
Thanks,
Prashanth Are
MS fabric community support
Post ProdigyHi, in terms of performance and costs is it better implementing a notebook that uses the BC API or creating a data pipeline (copy job) that uses the Fabric OData connector?
BC API (vers. 2.0) and Fabric OData connector are both based on the OData v4 protocol.
Thanks
Community SupportHi pmscorca,
Key considerations include:
-
Pagination Handling: OData responses are paged; proper looping with @odata.nextLink is necessary.
-
Filtering and $select: Use query options to limit data volume and improve performance.
-
Initial vs. Incremental Load: Design a full load for historical data and a separate incremental load based on modified timestamps.
-
Error and Retry Logic: Implement robust error handling for timeouts or throttling.
-
Data Volume Management: For large volumes, consider batching the load or using staging layers.
Thanks,
Prashanth Are
MS fabric community support
- pmscorca
Ok, but in terms of performance and costs is it better implementing a notebook that uses the BC API or creating a data pipeline (copy job) that uses the Fabric OData connector? Thanks
- v-prasare
Hi pmscorca,
In terms of performance, the Data Pipeline approach generally provides better results. It is optimized for bulk data movement, supports built-in parallelism, and is tuned for efficient handling of OData sources like Dynamics 365 Business Central. This makes it more suitable for high-volume or full-load scenarios.
When considering cost, Data Pipelines again come out ahead. They have lower orchestration overhead and are designed to be lightweight and efficient for recurring ETL tasks. Since they don’t require custom compute or extended execution times like notebooks, they’re usually more cost-effective.
In terms of reliability, especially when dealing with rate-limiting, API throttling, or complex retry strategies, Notebooks are more reliable. They allow full control over retry logic, timeouts, and error handling mechanisms, which is essential for robust data extraction from business APIs.
Finally, when it comes to simplicity, Data Pipelines are the preferred choice. They provide a user-friendly interface, come with built-in OData connectors, and handle authentication and API interactions automatically. This makes them well-suited for quick setup and straightforward data integration tasks without the need for complex configurations.
Thanks,
Prashanth Are
MS fabric community support
- pmscorca
Ok, therefore to get structured data as those BC ones reading some large table, using Data pipeline with OData connector seems as better solution as possible in terms of performance and costs, isn't it?