Fabric Datflow Gen 2 Creating multiple staging datawarehouse and Tables on every run

[noorudeen100]

Hi Team,

I am using dataflow gen 2 to load on premise data to fabric datawarehouse upon using it for every new dataflow gen 2 it creating new staging datawarehouse and it inside it creating multiple tables and storing the records this is dont want why it is happening , why i am using dataflow gen 2 for data loading as fabric data factory not allowing to load data from onprimise to datawarehouse for that it expecting the inbetween staging area like azure blob storage like that currently i have only fabric liscence i didnt know this bug before i purchase the fabric, so in order to handle this i used dataflow gen 2 for my work but this also creating a dublicate staging datwarehouse and tables.

my concern will increase in future means how i can handle it and it also consumes more storage so is there any way i can stop or cler this activity.

[noorudeen100]

HI Prasanth,

i dont find why it is been automatcially creating the multiple staging datawarehouse with timestamp also i did manually deleting it in onelakecatalog and i gace user login token issue so that i am unable to login into the ssms and power bi it casued an issue for accesing the micrsosft fabric i checkd in onlune as there is no official guidlines to manage this autocreated staging items.

so now i decided to keep all the staging items as it is so i will not face any token access issue, but this is really worry that microsoft has doesnt have any guidelines to cleanup this staging artifacts as it grows and it will casue additional storage and cost 

[v-prasare · Accepted solution]

Hi @noorudeen100,

Meanwhile you can submit your detailed feedback and ideas via Microsoft's official feedback channels, such as the Microsoft Fabric Ideas.  Feedback submitted here is often reviewed by the product teams and can lead to meaningful improvement, your inputs will truely help grow our community.

Thanks,

Prashanth Are

MS Fabric community support

[v-prasare]

Hi @noorudeen100,

As we haven’t heard back from you, we wanted to kindly follow up to check if the solution provided for your issue worked? or let us know if you need any further assistance here?

Thanks,

Prashanth Are

MS Fabric community support

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly and give Kudos if helped you resolve your query

[v-prasare]

Hi @noorudeen100,

As we haven’t heard back from you, we wanted to kindly follow up to check if the solution provided for your issue worked? or let us know if you need any further assistance here?

Thanks,

Prashanth Are

MS Fabric community support

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly and give Kudos if helped you resolve your query

[burakkaragoz]

Hi @noorudeen100 ,

Use a Custom Connector Configuration

Sometimes the default PostgreSQL connector used by the On-Premises Data Gateway can be finicky. You can try the following:

• Manually configure the connection string in the gateway using advanced settings. For example:

Host=your_host;Port=5432;Database=your_db;Username=your_user;Password=your_password;SSL Mode=Prefer;

• Make sure the SSL mode matches what your PostgreSQL server expects (Require, Prefer, or Disable).

Check for 32-bit vs 64-bit Driver Mismatch

Ensure that the bit version of the PostgreSQL ODBC driver matches the bit version of the gateway. A mismatch can cause silent failures.

Try Using a Different Connector (ODBC vs Native)

If you're using the native PostgreSQL connector, try switching to the ODBC connector in the gateway setup and configure it using a DSN that you’ve already tested successfully.

Let us know if this helps—or if you can share the exact connector type and version you're using, we can dig deeper.

[Witmanm]

As mentioned in this thread: Solved: Re: Dataflow gen2 - multiple staging artifacts - Microsoft Fabric Community

It is certainly possible that this behavior could change as time goes on. For now, the only way to control these staging warehouses is to disable staging in your dataflow.