topic Re: Dataflow published failed: At least one of the queries in this dataflow does not contain a schema. in Dataflow

Dataflow published failed: At least one of the queries in this dataflow does not contain a schema.

mwsmoley — Fri, 28 Jul 2023 18:48:57 GMT

Hello,

I have been working on a creating a dataflow gen2 in fabric that transforms a JSON files stored in a Lakehouse. I have created a Sample file using 1 of the jsons, a parameter that references the sample file, a transform query referencing the parameter, a function from that. I then invoke the function in my main query and everything looks and works correct when I am in the dataflow. When I click publish I receive an error "Dataflow publish failed At least one of the queries in this dataflow does not contain a schema, Please update the query to include a scheam or delete it and try again". I have searched across various resources and not found any further details on what this means or how to solve it. I would prefer not to delete this as it took a fair amount of time to build and I do not see how redoing this would change anything. Does anyone have ny idea on how to resolve this or figure out where the problem is?

Re: Dataflow published failed: At least one of the queries in this dataflow does not contain a schema.

LuitwielerMSFT — Mon, 31 Jul 2023 12:08:26 GMT

Hi!

That is unfortunate that you are not able to publish your dataflow with the json schema. I have a couple of questions:

Do you get a similar error in the orignal Dataflows? (Power BI Dataflows)
Can you share a screenshot of the query diagram to show how the dataflow query dependencies are constructed?
Do you have requestid's that are returned with the error messages?

Re: Dataflow published failed: At least one of the queries in this dataflow does not contain a schema.

mwsmoley — Mon, 31 Jul 2023 12:23:25 GMT

Hello,

Thank you for responding. I have continued to to try to solve the issue on my own and have isolated it to the "Sample file" query I created that references a single JSON file for the function. Below are respones to your questions and images as requested. Thank you.

Do you get a similar error in the orignal Dataflows? (Power BI Dataflows)... Our company did not use Dataflows and this was originally done in a SSIS script but we have to decided to move to Fabric and are working to move all of our ETLs.
Can you share a screenshot of the query diagram to show how the dataflow query dependencies are constructed? Let me know if the image below provides the details you are looking for.
Do you have requestid's that are returned with the error messages? Below is a copy of the exact error.

Re: Dataflow published failed: At least one of the queries in this dataflow does not contain a schema.

miguel — Mon, 31 Jul 2023 15:28:54 GMT

Hi!

Based on the screenshot shared, it appears that you only have one query that outputs a binary (perhaps a JSON file).

Please confirm if this is the case.

Re: Dataflow published failed: At least one of the queries in this dataflow does not contain a schema.

mwsmoley — Mon, 31 Jul 2023 15:52:36 GMT

Hello Miguel,

Yes the screen shots only show a single query however his was me trying to figure out what query was causing the problem. Below are some screen shots of the full flow that I was working to create. The below also has the same error as the single query. Let me know if you need any further details.

Re: Dataflow published failed: At least one of the queries in this dataflow does not contain a schema.

miguel — Mon, 31 Jul 2023 15:59:20 GMT

Are you using the Combine Files experience or manually creating the sample file and function?

Re: Dataflow published failed: At least one of the queries in this dataflow does not contain a schema.

mwsmoley — Mon, 31 Jul 2023 16:03:07 GMT

I am mannualy creating the sample file and function. I tried using the combine file expereince but it runs until an error pops up saying that the interface has not responded in a required amount of time and then it fails. It is about 700 json files and they have a lot of complexity that I want to break up into multiple queries with different functions to publish to seperate tables.

Re: Dataflow published failed: At least one of the queries in this dataflow does not contain a schema.

mwsmoley — Sun, 06 Aug 2023 14:42:41 GMT

To resolve this issue I have created a pipeline that copys each file one at a time to a seperate folder to be processed one at a time. This is much slower and more complicated but with not being able to use the sample file as originally intended I found no other way to solve this.

Re: Dataflow published failed: At least one of the queries in this dataflow does not contain a schema.

miguel — Mon, 07 Aug 2023 20:59:43 GMT

could you please raise a support ticket so an engineer can take a closer look at your original dataflow?

Below is the link to raise the support ticket:

https://go.microsoft.com/fwlink/?linkid=2230908

Re: Dataflow published failed: At least one of the queries in this dataflow does not contain a schema.

mwsmoley — Mon, 07 Aug 2023 21:10:59 GMT

Ticket has been submitted.

Re: Dataflow published failed: At least one of the queries in this dataflow does not contain a schema.

mrojze — Thu, 31 Aug 2023 15:16:27 GMT

I am having the same issue. Any update from Engineering?

Re: Dataflow published failed: At least one of the queries in this dataflow does not contain a schema.

mrojze — Thu, 31 Aug 2023 15:47:13 GMT

Ha! I got it!

You are getting this error because you have some queries set to go to staging. Basically, if you select them to "Enable staging" it'd require you to enter a Target (aka schema).

In other words, make sure only the queries that you are landing are checked and they do have a defined data destination.

Re: Dataflow published failed: At least one of the queries in this dataflow does not contain a schema.

mwsmoley — Wed, 06 Sep 2023 15:44:25 GMT

I continued to run into issues with other sample file ones including those created by the combine file experience so decided that instead of using the combine file am doing each file individually with a loop. It is not efficent and takes much longer but it has been sucessful everytime where the combine file became a problem.

I ended up telling them to close the ticket as I had a alternative solution. I do believe that Microsoft has a long way to go on Fabric as alot of things seem to be half working or only in the specific situations Microsoft allows which limits the ability of users to have a full solution.

Re: Dataflow published failed: At least one of the queries in this dataflow does not contain a schema.

mrojze — Wed, 06 Sep 2023 15:49:31 GMT

Hi there,

have you seen my latest answer?

I fixed my issue with that setting change.

Let me know,

Martin

Re: Dataflow published failed: At least one of the queries in this dataflow does not contain a schema.

mwsmoley — Wed, 06 Sep 2023 15:58:19 GMT

I only had staging enabled on the final query that was set with a data destination. From what I can see the sample file canot be set to enable staging as it is greyed out and disabling staging on the transform file query had no effect. I also have had previous combine files which worked just fine for a while that had staging working on the transform file.

This seems to be a inconsistency in the use of the combine files using a transform file. Hopefully when fabric comes out of preview the issues will be resolved and properly documented.