PBI Dataflows Folder data source & custom columns

[mark_carlisle]

Appreciate that Dataflows is currently in Preview so not expecting everything immediately. Could I just ask the following;

1. Are custom columns going to be available as an option within the in future. Currently I only seem to be able to create a Conditional Column or a Index Column via the UI and a custom column needs to be created via the Advanced Editor.
2. After adding the custom columns the data took longer to refresh, increasing from 39 seconds to 104 seconds is this performance hit to be expected when adding custom columns?
3. Will Folder be getting added as a datasource?

This is a possible bug - After setting up Incremental refresh on the dataflow it automatically creates the necessary RangeStart and RangeEnd parameters and adds a step to titled #"AgentByTeamBySkillByDay-54686544617465-autogenerated_for_incremental_refresh". Renaming this step for readability and placing it earlier in the list of steps causes a new auto generated incremental refresh step to be added.

[mmace1]

Twice my searches have led me to this thread - similar problems seemingly. 

Another vote for adding 'Folder' as a Datasource option.  Is it meaningful to start suggested-votes for these, or no since parity is the goal anyway...? 

[mark_carlisle]

@mmace1 Just thought I'd mention that I've done some testing with a folder data source and PBI dataflows and it is possible to get it working.

You need to connect to the Folder data source in PBI Desktop and then copy each of the querries, along with the parameter and function from the Advanced Editior in PBI desktop in to a Blank query within dataflows. The only thing I had to do was to remove the Filtered Hidden Files step within the final query

Be great if MS add it as a source so the process is automated but this is an acceptable work around. 

[mmace1]

Blast from the past here - 

For loading all the files in a folder: I tried copying all of the advanced editor M code in a Dataflow blank query, but got the following error.  Pretty sure that's because the Sample Queries that are also produced when merging files, obviously aren't present. 

How did you get around that? 

[mark_carlisle]

That's correct you'll need to copy in all the individual queries generated by Power BI when combining files from a Folder data source.

[otravers]

Yes. this is very tedious. I posted on the Ideas forum so that PQW automatically imports query dependencies, like happens when you copy/paste between desktop apps:

https://ideas.powerbi.com/forums/265200-power-bi-ideas/suggestions/36184816-copy-paste-m-queries-wit...

[Anonymous]

@mark_carlisle  

As I described in the post below, I have an issue where I cannot save a dataflow containing queries/parameters/function to get csv data from a folder - do you know if there is a way to fix it?

https://community.powerbi.com/t5/Service/Unable-to-save-a-dataflow-status-400/m-p/643658#M64081

Thanks in advance.

[mark_carlisle]

I've not tested CSV from OneDrive but I've found you can't save Data flows that have duration column types.

[Anonymous]

Thank you for the response, I do not have any duration type columns (tried converting all columns to text but didn't resolve the issue)...

[mark_carlisle]

How are you connecting to the OneDrive folder? I was not able to connect to my OneDrive folder and combine files like I can when I use a (local) folder data source. This article (https://docs.microsoft.com/en-us/power-bi/desktop-use-onedrive-business-links) also suggests that you have to connectto each individual file and then merge/append as required.

Could you share your queries so I can understand how you are connecting to OneDrive?

[mmace1]

For the Custom Column - you can also create it in Power BI desktop using the interface, then just copy the Advanced editor code out, and into Dataflows. 

To me that seems easier than messing with non-intellisense M language.  

[maya]

Hi Mark, the autogenerated step for incremental refresh (in this case #"AgentByTeamBySkillByDay-54686544617465-autogenerated_for_incremental_refresh") should remain as is - without renaming it or placing it earlier in the list of steps.

[Ian-H]

I would also love to see 'Folder' as a data source for Dataflows!

[otravers]

Microsoft said they intended to bring Dataflows' Power Query to parity with Power BI Desktop.