topic Re: Error when trying to create a dataflow gen 2. in Dataflow

Error when trying to create a dataflow gen 2.

YairCzitrom — Wed, 14 Feb 2024 15:01:38 GMT

Hi Guys.

I have a data lake and date warehouse.

Tried to create on both of the the same thing and i got the same error.

Crteated a dataflow gen 2 that gets table from my on prem SQL server via a working gateway.

When i create the dataflow, all is well and i can see the date in the table below:

When i try to add destination (no mather if it is lake or warehouse) i get the following error when i try to see the existing tables or trying to add a new table:

Something went wrong while retrieving the list of tables. Please try again later.

I have tried to enable and disable the staging in the query panel on the left.

Really fustrating!

Anyone got an idea on what's wrong and how to fix?

Tnx guys.

Re: Error when trying to create a dataflow gen 2.

pqian_MSFT — Wed, 14 Feb 2024 18:44:51 GMT

Does your gateway have a firewall that blocks line of sight to the lakehouse\warehouse?

Re: Error when trying to create a dataflow gen 2.

Element115 — Wed, 14 Feb 2024 19:42:32 GMT

pqian_MSFT YairCzitrom

Navigate to this page: On-premises data gateway considerations for data destinations in Dataflow Gen2 - Microsoft Fabric | Microsoft Learn

and read section 'Solution: Set new firewall rules on server running the gateway'. Perhaps it is a firewall issue?

Re: Error when trying to create a dataflow gen 2.

YairCzitrom — Thu, 15 Feb 2024 08:51:02 GMT

Guys...tnx for the answers.

We added the rule to the firewall... same problem.

I don't see anything in our firewall that blocks.

Moreover, i have removed the staging part from the dataflow... the same problem.

Let me say it again - bthe issue is when i am addding target for the data to be stored (datalake or data warehouse)...it is trying to create a table there... writing "validating data destination target" for like 2-3 minutes and there "Something went wrong when retriveing the lsit of table. try again later"

For me it looks like more a Fabric problem and not something on my side.

Re: Error when trying to create a dataflow gen 2.

Element115 — Thu, 15 Feb 2024 17:49:11 GMT

You're welcome. I'd tend to agree as I keep bumping into bugs every other day. Last night, I couldn't even create a new Power BI report with the Power BI persona--Fabric couldn't retrieve the model that was in the lakehouse after I ran all my DFg2s. The product is definitely not GA ready. Let's cross fingers it will not take 6 years to fix, like the broken CTRL-Z / Undo in Power App studio that people have been complaining about, I kid you not, for 6 years!

Re: Error when trying to create a dataflow gen 2.

pqian_MSFT — Thu, 15 Feb 2024 18:14:33 GMT

YairCzitrom , let's debug this - can you share the session ID where you are running into this issue.

1. In your dataflow editor, click Options -> diagnositics -> copy the session ID

2. If possible, run a HTTP capture via fiddler or browser dev tools

3. Repro the issue where in the output destination selector, it produces "Something went wrong"

4. If you are running the network capture, save it locally

(Export HAR...)

Keep the file for future references and post the session ID here.

Re: Error when trying to create a dataflow gen 2.

pqian_MSFT — Thu, 15 Feb 2024 21:34:20 GMT

Also, since this is routing to a gateway, make sure you turn on gateway logging (additional logging, see: https://learn.microsoft.com/en-us/data-integration/gateway/service-gateway-performance#slow-performing-queries) before the repro.

Once you reproduced the error, flush the logs and zip them up for future reference

Re: Error when trying to create a dataflow gen 2.

Anonymous — Thu, 15 Feb 2024 21:42:25 GMT

I'm having the exact same issue, DFG2 pulling data from an on prem SQL Server, when selecting the target table in a Lakehouse, I get the same error message as described.

Re: Error when trying to create a dataflow gen 2.

pqian_MSFT — Thu, 15 Feb 2024 23:12:12 GMT

Can you also follow the guidance below to reproduce the error with gateway additional logging on. In the additional logs we can see what end point it's trying to talk to and failing.

Re: Error when trying to create a dataflow gen 2.

YairCzitrom — Fri, 16 Feb 2024 08:07:31 GMT

id= 8ee7c58b-bb97-4709-8fd7-e91d97018510

i can't send you the diagnostic files as this chat will not take zip, saz og json

How do you want me to send them?

Re: Error when trying to create a dataflow gen 2.

YairCzitrom — Fri, 16 Feb 2024 10:26:58 GMT

Guys.... it is fixed

My bad... after all.

It IS all about the fire wall.

We open it on the on-prem server BUT it is not enough!

AS we are hosting it on some data center, THEY have to open the firewall and it will work.

The biggest problem now is that they are not happy about it as we open another port in thier firewall, and they cannot do the *.domainname roles apperently so they open for all!!!

Microsoft support - is there any way to give them a specific address/s to open?

Re: Error when trying to create a dataflow gen 2.

pqian_MSFT — Fri, 16 Feb 2024 18:17:22 GMT

Glad to hear that it's firewall after all. For specific IP addresses, you can check the PowerBI service tag - Power BI\Fabric uses specific preallocated IP addresses for inbound and outbound connectivity, the entire range is listed here:

Download Azure IP Ranges and Service Tags – Public Cloud from Official Microsoft Download Center

You can probably narrow it down to your specific cluster region instead of the entire range. Find out which FQDN you had to allowlist for this to work, and then nslookup on it. Then you can find the PowerBI's IP addresses in that region in the document above.

Re: Error when trying to create a dataflow gen 2.

YairCzitrom — Mon, 19 Feb 2024 07:25:37 GMT

What is i put in the firewall role the urls of the end point of the lakehouse, warehouse and the staging DB's?

Will that be enough?

Re: Error when trying to create a dataflow gen 2.

YairCzitrom — Tue, 20 Feb 2024 14:49:15 GMT

pqian_MSFT have you seen my question?

Re: Error when trying to create a dataflow gen 2.

pqian_MSFT — Wed, 21 Feb 2024 06:06:08 GMT

That should be enough but it won't be scalable, since these are per workspace. If you or anyone in your org in the future creates more workspaces or dbs, they will also need allowlisting.

This is also assume you can even do FQDN at the firewall level. Usually L2 firewalls do not allow FQDNs.

Re: Error when trying to create a dataflow gen 2.

YairCzitrom — Wed, 21 Feb 2024 09:33:10 GMT

Ok... status:

Our hosting company apperently can not do the *.doaminname with their firewall
They have tried to put the full endpoint address... no joy:

Any good ideas of how to overcome this?

Re: Error when trying to create a dataflow gen 2.

pqian_MSFT — Wed, 21 Feb 2024 18:29:03 GMT

They need to fix what they consider as "FQDN" 🙂

What about just IPv4 address ranges via CIDR block?

Re: Error when trying to create a dataflow gen 2.

data_power — Thu, 14 Mar 2024 08:04:16 GMT

Hi Folk,
I got the same error while trying to use my oracle database as source. But didn't work like the others. I made another try to see if the firewall issue or not with giving a try with an csv file which is located on my client. So i don't expect to get an error because of firewall. But still getting the same error.

Any idea what could be the issue? I have the fabric admin user rights and while checking the tables via Lakehouse component I be able to see the tables files and get any query if I want.

Thanks in advance for the advices.

Re: Error when trying to create a dataflow gen 2.

YairCzitrom — Thu, 14 Mar 2024 08:16:52 GMT

We ended up with opening our fire wall to specific IP's that we could see that they are being used and that fixed the problem.

Re: Error when trying to create a dataflow gen 2.

Element115 — Fri, 15 Mar 2024 05:47:09 GMT

Yeah finally, we got it fixed to. The sysadmin set a rule like this:

outgoing IP: 0.0.0.0 (essentially any IP address)
protocol: TCP

port: 1433