Forum Discussion
Memorable MemberTables missing when connecting to dataflow from PBI desktop (2023)
Community ChampionIt is a little buggy. Try this:
- Set all 5 tables to load. Can you see them in Power Query?
- If not, go to the tables you cannot see and add a column. Doesn't matter, just have the column be 1 or something.
- Save and let the dataflow process.
- Can you see them now? You should.
- Go back and delete those fake columns, and let it refresh again.
That is what I've done a few times this past week with dataflows that look ok in the service but cannot be seen by Power Query.
dbeavon3edhans Thanks for the reply.
Can you share the underlying principles or theories behind your suggestion? I'd rather not flail around without first having some sort of framework for understanding why the dataflows aren't working. (I already waste tremendous amounts of time with Power BI tools, even when things are working as expected.) Also can you tell me, if you ever open your own support cases with CSS? It doesn't seem right to me that some people do the work of reporting bugs to CSS and others don't do their share of that work. Even if bugs aren't fixed, Microsoft should have an extensive KB list that describes the bugs that are being reported by their customers. </rant>
There is a clue that something is wrong in the service as well. The dataflow's summary screen (before you go into the authoring experience) shows no columns for the tables that are missing. If I hit the little drop-down expansion button, I'm not seeing the columns that should be there.
Here is another thing I should mention. The problem goes away if I decrease the size of data in one of my tables. One of the tables is extremely large, but it consists of only one row. It has a base64-text-encoded binary value.
What I really wished, when I first started using dataflows, is that Microsoft wouldn't have put such a heavy reliance on CSV for encoding data. It is no surprise that this stuff is unreliable. I think there are lots of better options these days, and I don't think anyone with a basic degree in computer science would encode their mission-critical information in CSV tables. (Please don't suggest I try using datamarts. I REALLY don't want to go down that rabbit hole as well. I just want them to give me the option of several file types, like json, xml, avro, parquet, sqlite, or whatever other file makes sense for a given dataset).
Also I wish Microsoft would expose the unexpected/internal errors that their software is producing. My guess is that there are lots of things going wrong internally. However they don't seem to want to display error messages to users, and they don't have any logging mechanisms either.
Thanks, again for your pointers. I'll probably start the support case today. But usually the process takes 1 or 2 months before I reach the engineers who actually have access to read their code.
- edhans
I've seen issues where it doesn't look like the schema of the table is being published for consumption correctly. "Schema" might not be the right word. By changing the tables, saving and refreshing, then changing back, it forces the schema to be updated and republished.
- dbeavon3
edhans You may be on to something. It seems that I was able to use your trick and correct the schema.
It just bothers me that the service doesn't alert us about the bad schema right away. Moreover, when running refresh operations on these tables, they happily run a bunch of PQ and then throw away the results - this makes no sense!
... I think I assumed that the PQ code which defines the tables would be always be fully respected. This doesn't seem to be the case; it appears that there is some additional schema being kept somewhere else in these dataflows, and the additional schema actually "trumps" the PQ code. However the additional schema isn't always maintained very carefully as you point out. (I'm not sure why I spend all that time waiting on the "validating" of my dataflow, if not for the purpose of storing the schema at the end of it all.)
FYI, I went back and looked at the json of the dataflow, as it existed before using your trick. As you can see below, the offending tables had no columns. The good tables have columns listed in the json.Thanks for the help. You should write a blog about this or something. Otherwise I'll try to get Microsoft to add it in their "troubleshooting" documentation (or whatever). I miss the good old days when Microsoft actually maintained a KB containing the known software bugs and potential workarounds. Nowadays you are lucky if they acknowledge a bug at all, or add it to their so-called "troubleshooting" docs.
Thanks for the information! I had the same problem in 3 dataflows. Yesterday, I added columns and the problems was solved.
