tutorial requests

3 Topics

SAP ECC into a Microsoft Fabric Lakehouse using Fabric Data Factory.
We are currently extracting data from SAP ECC into a Microsoft Fabric Lakehouse using Fabric Data Factory. *SAP ECC side:* We are using the SAP .NET Connector (NCo), which communicates with SAP through RFC calls. *Fabric side:* We are using the SAP Table – Application Server connector, which leverages the Open Data Gateway (ODG) to retrieve data from SAP ECC. *Small tables:* Data extraction and loading work successfully with relatively small SAP tables. *Large tables:* When extracting larger tables, the process fails with a "Not enough space" error. *Partitioning / Chunking:* We have tried the partitioning and RFC table options available in the Fabric connector to split large-table extraction into smaller chunks/batches based on indexed fields. *Performance issue:* Although chunking allows us to process the data, the resulting throughput is very low, and extracting large tables takes an excessive amount of time. *Key Challenge* We need to identify a scalable and performant approach for extracting large SAP ECC tables into Microsoft Fabric Lakehouse, while avoiding the "Not enough space" error and significantly improving extraction throughput. In particular, we need to determine whether the bottleneck is related to SAP RFC/NCo processing, Open Data Gateway, Fabric's SAP Table connector, partitioning strategy, batch size, or the underlying data transfer mechanism, and what configuration or architecture would provide better performance for large-volume SAP extractions.
Solved
Murtaza_Ghafoor
15 days ago Place Apache Airflow Job
Views
0likes
Comments
New in Airflow Fabric
Hello, dear all. I'm starting with Airflow in Fabric. So, I want to know how I can migrate my project that is working in my local machine if Fabric only allows add .py files. Usually, in a Airflow project we have to work with folders containing other scripts, for example. My project get data from a API and store json files 5 or 6 times per day. So, how can I connect the json results to a fabric workspace lakehouse? Finally, once stored I have to transform them using notebooks in pipelines to be shown in dashboards. I would appreciate your help. Thanks in advance!
Solved
ArielSubia
1 year ago Place Apache Airflow Job
Views
0likes
Comments
stuck with MS Learn Tutorial for Apache Airflow Jobs for dbt-fabric Orchestration
Hello, does anyone managed to implement the sample/tutorial Transform data using dbt - Microsoft Fabric | Microsoft Learn ? For us, with a Starter Pool running on Apache Airflow version 2.10.5, only the version of astronomer-cosmos==1.5.1 and dbt-fabric==1.5.0 are valid. ....but when running the DAG 'dags/my_cosmos_dag.py we systematically get a Failed to start error. Thanks
Solved
rabbyn
1 year ago Place Apache Airflow Job
Views
1like
Comments