Error when trying to create a dataflow gen 2.

[YairCzitrom]

Hi Guys.

I have a data lake and date warehouse.

Tried to create on both of the the same thing and i got the same error.

Crteated a dataflow gen 2 that gets table from my on prem SQL server via a working gateway.

When i create the dataflow, all is well and i can see the date in the table below:

When i try to add destination (no mather if it is lake or warehouse) i get the following error when i try to see the existing tables or trying to add a new table:

Something went wrong while retrieving the list of tables. Please try again later.

I have tried to enable and disable the staging in the query panel on the left.

Really fustrating!

Anyone got an idea on what's wrong and how to fix?

Tnx guys.

[nik1031]

Facing exactly the same issue. Why does the gateway even need to exist for the destination table? If the dataflow sees the data one would assume it is already in Fabric, why would there be any further work needed on the gateway to be done if both Lakehouse and the Dataflow is in the cloud & same workspace??? Literally having to turn from a data analyst into a network engineer

[data_power]

Hi Folk,
I got the same error while trying to use my oracle database as source. But didn't work like the others. I made another try to see if the firewall issue or not with giving a try with an csv file which is located on my client. So i don't expect to get an error because of firewall. But still getting the same error. 

Any idea what could be the issue? I have the fabric admin user rights and while checking the tables via Lakehouse component I be able to see the tables files and get any query if I want. 

Thanks in advance for the advices.

[YairCzitrom]

We ended up with opening our fire wall to specific IP's that we could see that they are being used and that fixed the problem.

[Element115]

Yeah finally, we got it fixed to.  The sysadmin set a rule like this: 

outgoing IP: 0.0.0.0 (essentially any IP address)
protocol: TCP

port: 1433

[YairCzitrom]

id= 8ee7c58b-bb97-4709-8fd7-e91d97018510

i can't send you the diagnostic files as this chat will not take zip, saz og json

How do you want me to send them?

[FB33]

I'm having the exact same issue, DFG2 pulling data from an on prem SQL Server, when selecting the target table in a Lakehouse, I get the same error message as described.

[pqian_MSFT]

Can you also follow the guidance below to reproduce the error with gateway additional logging on. In the additional logs we can see what end point it's trying to talk to and failing.

[YairCzitrom]

Guys...tnx for the answers.

We added the rule to the firewall... same problem.

I don't see anything in our firewall that blocks.

Moreover, i have removed the staging part from the dataflow... the same problem.

Let me say it again - bthe issue is when i am addding target for the data to be stored (datalake or data warehouse)...it is trying to create a table there... writing "validating data destination target" for like 2-3 minutes and there "Something went wrong when retriveing the lsit of table. try again later"

For me it looks like more a Fabric problem and not something on my side.

[pqian_MSFT]

@YairCzitrom , let's debug this - can you share the session ID where you are running into this issue.

1. In your dataflow editor, click Options -> diagnositics -> copy the session ID

2. If possible, run a HTTP capture via fiddler or browser dev tools

3. Repro the issue where in the output destination selector, it produces "Something went wrong"

4. If you are running the network capture, save it locally

(Export HAR...)

Keep the file for future references and post the session ID here.

[pqian_MSFT]

Also, since this is routing to a gateway, make sure you turn on gateway logging (additional logging, see: https://learn.microsoft.com/en-us/data-integration/gateway/service-gateway-performance#slow-performi...) before the repro.

Once you reproduced the error, flush the logs and zip them up for future reference

[YairCzitrom]

Guys.... it is fixed

My bad... after all.

It IS all about the fire wall.

We open it on the on-prem server BUT it is not enough!

AS we are hosting it on some data center, THEY have to open the firewall and it will work.

The biggest problem now is that they are not happy about it as we open another port in thier firewall, and they cannot do the *.domainname roles apperently so they open for all!!!

Microsoft support - is there any way to give them a specific address/s to open?

[pqian_MSFT]

Glad to hear that it's firewall after all. For specific IP addresses, you can check the PowerBI service tag - Power BI\Fabric uses specific preallocated IP addresses for inbound and outbound connectivity, the entire range is listed here:

Download Azure IP Ranges and Service Tags – Public Cloud from Official Microsoft Download Center

You can probably narrow it down to your specific cluster region instead of the entire range. Find out which FQDN you had to allowlist for this to work, and then nslookup on it. Then you can find the PowerBI's IP addresses in that region in the document above.

[YairCzitrom]

What is i put in the firewall role the urls of the end point of the lakehouse, warehouse and the staging DB's?

Will that be enough?

[YairCzitrom]

 pqian_MSFT have you seen my question?

[pqian_MSFT]

That should be enough but it won't be scalable, since these are per workspace. If you or anyone in your org in the future creates more workspaces or dbs, they will also need allowlisting.

This is also assume you can even do FQDN at the firewall level. Usually L2 firewalls do not allow FQDNs.

[YairCzitrom]

Ok... status:

1. Our hosting company apperently can not do the *.doaminname with their firewall
2. They have tried to put the full endpoint address... no joy:Any good ideas of how to overcome this?

[pqian_MSFT]

They need to fix what they consider as "FQDN" 🙂

What about just IPv4 address ranges via CIDR block? 

[Element115]

You're welcome.  I'd tend to agree as I keep bumping into bugs every other day.  Last night, I couldn't even create a new Power BI report with the Power BI persona--Fabric couldn't retrieve the model that was in the lakehouse after I ran all my DFg2s.  The product is definitely not GA ready.  Let's cross fingers it will not take 6 years to fix, like the broken CTRL-Z / Undo in Power App studio that people have been complaining about, I kid you not, for 6 years!  

[Element115]

@pqian_MSFT @YairCzitrom 

Navigate to this page: On-premises data gateway considerations for data destinations in Dataflow Gen2 - Microsoft Fabric | ...

and read section 'Solution: Set new firewall rules on server running the gateway'. Perhaps it is a firewall issue?

[pqian_MSFT]

Does your gateway have a firewall that blocks line of sight to the lakehouse\warehouse?