Forum Discussion
Need tip on which tools to use for metadata driven data import (design included) (gateway)
Hi m_vandermeij,
Thank you for the update.
Based on your description, the pipeline and dataflow setup are functioning as expected; but, the issue arises when Dataflow attempts to write to the Lakehouse. Since you can load tables directly from the gateway to the Lakehouse using a pipeline, we can confirm that firewall restrictions are not the cause, as the Lakehouse is a cloud service and firewalls do not apply.
To address this, I recommend first testing the Dataflow output by writing to Lakehouse files (Parquet). If successful, this suggests the issue lies with table schema mapping.
Also, you can use Dataflows for raw ingestion only and perform transformations in the Lakehouse using Notebooks. For large-scale data loads, especially with over 1000 tables, it is generally more effective to use Pipelines with the gateway for raw data ingestion, and then manage transformations within Fabric using Notebooks or SQL.
Thank you.We have learned that firewall was the issue.
Some firewalls apparently need IP ranges authorized instead of just DNS names, because they are only resolving a small number of IP adresses.
We needed to inlucde large ranges from this: Download Azure IP Ranges and Service Tags – Public Cloud from Official Microsoft Download Center
After that, importing started working.
Note: Also, the large use of capacity is apparently a bug in notebooks as hosted in Fabric at the moment. A job keeps running after failure and grabs capacity, where you only want it to take minimal capacity and stop on failure. This apparently is known.
We have succesfully made a dataflow and pipeline.
The pipeline now feeds the tablelist & database list to the dataflow.
We're currently having problems with storing the data to the lakehouse though.
It is generating errors which I do no expect.
I've been searching for answers. Strangely the responses start asking questions about firewalls.
But I have been able to load a set of tables directly via pipelines from source gateway to lakehouse files and tables.
I have no clue why this would fail, as I thought browsing your lakehouse is something purely on the cloud side...
Community SupportHi m_vandermeij,
Thank you for the update.
Based on your description, the pipeline and dataflow setup are functioning as expected; but, the issue arises when Dataflow attempts to write to the Lakehouse. Since you can load tables directly from the gateway to the Lakehouse using a pipeline, we can confirm that firewall restrictions are not the cause, as the Lakehouse is a cloud service and firewalls do not apply.
To address this, I recommend first testing the Dataflow output by writing to Lakehouse files (Parquet). If successful, this suggests the issue lies with table schema mapping.
Also, you can use Dataflows for raw ingestion only and perform transformations in the Lakehouse using Notebooks. For large-scale data loads, especially with over 1000 tables, it is generally more effective to use Pipelines with the gateway for raw data ingestion, and then manage transformations within Fabric using Notebooks or SQL.
Thank you.
We have learned that firewall was the issue.
Some firewalls apparently need IP ranges authorized instead of just DNS names, because they are only resolving a small number of IP adresses.
We needed to inlucde large ranges from this: Download Azure IP Ranges and Service Tags – Public Cloud from Official Microsoft Download Center
After that, importing started working.
Note: Also, the large use of capacity is apparently a bug in notebooks as hosted in Fabric at the moment. A job keeps running after failure and grabs capacity, where you only want it to take minimal capacity and stop on failure. This apparently is known.