Missing Columns in Dataflow Gen2

[Mohammed-]

Hi

1- I created warehouse and then dataflow to get data from sql server and the destination target was created automatically in the bottom right of the page as in the attached photo below

2- found that columns with decimal type were missing in warehouse

3- I then created a new table in the destination target and it displayed all columns successfully, However. this table is not displaying inside dataflow Gen2

my questions are :

1- why this newly created table isn't showing in dataflow?

2- why the columns were missing in the first time I published in the dataflow and showing in warehouse?

Thank you

[Mohammed-]

Very sorry for delay, Thanks for your replay  it is very helpfull to me

I have onther queries there is any way to work in pipelie through sql queries  as i did in datafllow gen 2

i mean there is such options like Advanced option in the connection

thre reason i need that option i am testing fabric i with  big sql databae it is JD Edwards database aboun 4000 tables and tons off columns i need to get data form some table and sum column  i dont knoow my query ,some thing like the photo

I allrady create  a simple data set from  my sql database, create connection to sql server then data follow gen 2 and  warehouse , then connected the warhouse to power bi desktop to create some time intelligence report  ans i i ok   i will try to make it as document and send it you just to hear from you yor comments if you this allow to me

any way sorry for delay again and thank you very very much

[v-cboorla-msft]

Hi @Mohammed- 

Thank you for using Microsoft Fabric Community.

Glad to know that your issue got resolved.

Thanks for asking your query . The ask here is different from the original ask. Can you please start a new post with your ask as it would be helpful for others?

Thanks for understanding.

[v-cboorla-msft]

Hi @Mohammed- 

Welcome to Microsoft Fabric platform and thanks for posting your question here.

As I understand that you want to create a dataflow and publish the data from SQL server DB to warehouse as a destination.

While I tried to do the repro from my side, I am able to see all the columns which are successfully displayed in the destination target. For reference, please find the screenshots attached.

Source Data - all columns will present.

While working with steps -

Whatever data you are seeing here is not Source Data, it is just a data preview (data present in that selected table)

Attached screenshots for reference.

THIS IS ACTUALLY MISGUIDING YOU IN YOUR CASE.

After successfully completing the configuration step please wait for dataflow gen 2 to run successfully. Please find the attached screenshot for reference.

Attached screenshot of final destination - warehouse.

Please follow the troubleshooting steps below to overcome the issues:

• The newly created table is not showing in Dataflow Gen2 because it has not yet been ingested by the dataflow. You can manually refresh a dataflow to force it to ingest data immediately.
• The columns were missing in the first time you published in the dataflow and showing in the warehouse because When you publish a dataflow, the dataflow tries to infer the schema of the source data source automatically. If the dataflow is not able to infer the schema of the source data source correctly, it may not be able to ingest all of the data from the source data source. This can result in columns being missing in the dataflow, even though they are present in the source data source.

I hope this information is helpful. Please do let us know if you have any questions.

[Raul_Riv]

Then which is the solution to avoid missing columns, not beign recognized to infer the schema?

[v-cboorla-msft · Accepted solution]

Hi @Mohammed- 

Following up to see if the above answer was helpful. And, if you have any further query do let us know.

[v-cboorla-msft]

Hi @Mohammed- 

We haven’t heard from you on the last response and was just checking back to see if your issue got resolved. Please let us know if you have any further queries.