Forum Discussion
Dataflow Gen 2 - SharePoint List - Target
based on the screenshots, it appears that what you're looking at is what we call a Staging Lakehouse which is not meant to be accessed by the end-user. This is a system-level artifact that is used by Dataflows to help you with the loading of the data to the Lakehouse in your workspace.
We do not recommend interacting with staging artifacts and we will be hiding these artifacts in the future as well as preventing changes from the user. These are only meant to be used by the system.
You should be able to see a different Lakehouse which should be the one that you initially created, but it cannot be either of the Staging ones. it should also say that is owned by you. Can you confirm that you can see such Lakehouse?
Also, please let us know if you can share the screenshots that I mentioned in my previous reply from the Dataflows experience around the schema for your query (just a screenshot of what the columns look like) and the "mapping columns" dialog in the output destinations dialog that also showcases the name of what should be the new table.
After the initial execution.. this is in the lakehouse.
After 1 minute or so... I have the file and the table.
Waited 15 minutes... same thing.
Checked the refresh time in the mode file and got "2023-06-02T17:07:17.5296454+00:00"
Triggered the refresh of the dataflow.
Succesfull
Right after, have 3 tables (files)
THe model file has no 2 JSON in it with the second one having the following refresh.
Is this the expected behavior and does that mean that my table/file has been updated?
Does the system keep the other files as a backup or are those removed after a while?
Thanks
based on the screenshots, it appears that what you're looking at is what we call a Staging Lakehouse which is not meant to be accessed by the end-user. This is a system-level artifact that is used by Dataflows to help you with the loading of the data to the Lakehouse in your workspace.
We do not recommend interacting with staging artifacts and we will be hiding these artifacts in the future as well as preventing changes from the user. These are only meant to be used by the system.
You should be able to see a different Lakehouse which should be the one that you initially created, but it cannot be either of the Staging ones. it should also say that is owned by you. Can you confirm that you can see such Lakehouse?
Also, please let us know if you can share the screenshots that I mentioned in my previous reply from the Dataflows experience around the schema for your query (just a screenshot of what the columns look like) and the "mapping columns" dialog in the output destinations dialog that also showcases the name of what should be the new table.
Yes, miguel is correct. The tables named with GUIDs are the results of dataflow staging in the Staging Lakehouse, not the output destination you defined.
Hi,
Thanks for the feedback. FYI. I deleted everything and did the following.
- Created a new data lakehouse
- Created a new flow v2
As for the schema, really simple. I am reading a SP list and loading 2 columns for now: ID & Title. Use the detect type option, set the ID as the key.
Target is now my lakehouse.
And now I see the table in my Lakehouse
Now if I refresh it. It is succesful.
One suggestion for your system. Would it be possible to add a last refresh date & time in the Properties menu option under the Explorer?
Your answers were really useful.
Pascal
Glad it was helpful.
I like the idea of the Last Refresh time. miguel - do you know which Ideas forum (Home (microsoft.com)) is used for Lakehouse?
-John
