Forum Discussion
Power Query adding sequential columns
Hello,
I am trying to use Power Query within Excel to automatically load a single column of data from a workbook into the column directly to the right of the column that was previously uploaded. The trigger would be whenever a new workbook is uploaded to a SharePoint folder and the query is refreshed. I am able to create a query to overwrite the column I am wanting to upload to but i can't figure out how to make the query upload the new data to the column directly to the right of the previously uploaded column while still keeping the previously uploaded data in the column to the left of the new data. Basically, I want an automated solution for copy pasting one column of data into one spreadsheet as soon as the workbooks are uploaded to a sharepoint folder.
Any help is appreciated!
Thank you.
Almost there, I think. Now that I have a better sense of the strucutre of your workbooks and the exact desired output, I think this should do it:
let //connect to your site files Source = SharePoint.Contents( "<your site>", [ApiVersion = 15] ), //navigate to folder with xlsx #"Shared Documents" = Source{[Name="Shared Documents"]}[Content], #"Test File" = #"Shared Documents"{[Name="Test File"]}[Content], //perform the transformation on the xlsx, //output will be a list of Column6's from all xlsx ParseAndDrill = List.Transform( #"Test File"[Content], each let //parse excel ParseExcel = Excel.Workbook( _ ), //navigate to the sheet you want OpenStudy1Sheet = ParseExcel{[Item="Study1",Kind="Sheet"]}[Data], //get single column you want as a list //(want this format to construct table later) GetCol6 = OpenStudy1Sheet[Column6], //remove nulls from the col RemoveNulls = List.Select( GetCol6, each _ <> null ) in RemoveNulls ), //with list of Column6's from all xlsx, put into table ToTable = Table.FromColumns( ParseAndDrill ) in ToTableYes, I think this should do it. We can use Table.TransformRows to get access to all the fields rather than just [Content]; then, all we have to do is tag [Date created] to the top of our Column6's:
let //connect to your site files Source = SharePoint.Contents( "<your site>", [ApiVersion = 15] ), //navigate to folder with xlsx #"Shared Documents" = Source{[Name="Shared Documents"]}[Content], #"Test File" = #"Shared Documents"{[Name="Test File"]}[Content], //perform the transformation on the xlsx, //output will be a list of Column6's from all xlsx ParseAndDrill = Table.TransformRows( #"Test File", each let //parse excel ParseExcel = Excel.Workbook( [Content] ), //navigate to the sheet you want OpenStudy1Sheet = ParseExcel{[Item="Study1",Kind="Sheet"]}[Data], //get single column you want as a list //(want this format to construct table later) GetCol6 = OpenStudy1Sheet[Column6], //remove nulls from the col + add Date created RemoveNulls = {[Date created]} & List.Select( GetCol6, each _ <> null ) in RemoveNulls ), //with list of Column6's from all xlsx, put into table ToTable = Table.FromColumns( ParseAndDrill ) in ToTable
19 Replies
- Omid_MotamediseSuper User
Hi,
Can you provide an example for your data?- jbotFrequent Visitor
Please see my reply to v-nmadadi-msft for more content on what I am trying to achieve.
- SamanthaPuaXYHelper II
Hi jbot , I don't think that's possible by creating multiple tables/power query.
You could Read all the files (assuming they contain same data), and do a pivot to make the file name the header and load them into excel. This way all new files will appear as a new column to the right of existing columns.Better if you have a sample/screenshot of what you are trying to achieve
- jbotFrequent Visitor
Please see my reply to v-nmadadi-msft for more content on what I am trying to achieve.
- v-nmadadi-msftCommunity Support
Hi jbot,
May we know how exactly you are creating the query to overwrite the column, based on that information we will look for possible suggestions to reorder the column in correct way.
Also in this forum you will find people who are good at Fabric and Power BI if you believe your query can be resolved by people who are good at Excel and who have experience using powerquery within Excel, you can utilize Excel community: Welcome to the Excel Community | Microsoft Community Hub
Thanks and Regards- jbotFrequent Visitor
Hello,
Thank you for the response.
I have uploaded a few pictures to show what I am trying to accomplish. The first picture shows shows the close and load to operation and the destination of the first imported column. The second picture shows the first file in the sharepoint folder being imported. There is only one file in the sharepoint folder at this point. This result is good. The problem comes when I add another excel file to the sharepoint folder. I manually added file "M1-8" to the folder to replicate what our system will be doing automatically. I then manually refreshed the query and the result showed in third picture that the query loaded the new file on top of my old file loaction while moving the previous column of data down to the bottom as shown in fourth picture. I desire the new data column to be uploaded to the column to the right of the previous column once a new file is added to the sharepoint folder.
I also uploaded a picture of the query editor page (picture 5).
Thank you for the help.
Josh Query Load ToQuery Load to ResultM1-8 file AddedM1-8 file added 2query
- MarkLafSuper User
You'll want to connect to the folder with all the workbooks, transform them all at once in a list so that you've got the list of single columns from each workbook, then convert that into a table. As new workbooks are added into the folder, new columns will get added.
Here is the M with some examples. I'm assuming that the data are of different shapes in the workbooks.
sample1.xlsx
Column A Column B Column C 1 abc2 0 2 jwc731 10 3 ova249 20 4 ath260 30 5 plm111 40 sample2.xlsx
Column A Column C 5 100 4 1000 2 10000 1 100000 0 1000000 sample3.xlsx
Column A Column C 1 100000 o 1000000 FYI these are all just entered simply in Sheet1 of each workbook:
Now, I start with just samples 1-2 in a local folder:
Here is M that converts this into two columns:
let Source = Folder.Files("<local folder location with xlsx>"), //main initial action is to parse workbooks into a list ParseExcel = List.Transform( Source[Content], Excel.Workbook ), //now we can do whatever series of transformations we need that will //convert each list item / workbook into a single list (column) of data WhateverTransformsToOneColEach = //I'm doing a bunch of transforms at once here, //but you can take whatever approach you want. //The main thing is that the output needs to be //a list of column data per workbook List.Transform( ParseExcel, each let openSheet1 = _{[Name="Sheet1"]}[Data], promoteHeaders = Table.PromoteHeaders( openSheet1 ), tableToCols = Table.ToColumns( promoteHeaders ), combineColsIntoOne = List.Combine( tableToCols ) in combineColsIntoOne ), ToTable = Table.FromColumns( WhateverTransformsToOneColEach ) in ToTableWhat I mean by "output needs to be a list of column data per workbook" - this is the output of my WhateverTransformsToOneColEach step showing preview of each workbook output / list item:
Output on load (including some random data in columns to left to mimic your setup):
Now, I'll drop in sample3.xlsx
And hit refresh in my workbook - the parsing in our query, the <WhateverTransformsToOneColEach> part, needs to be able to handle whatever shape sample3.xlsx or any new workbooks will be in:
- jbotFrequent Visitor
Hello,
Thank you for looking into my issue. I am not very familar with M, the ParseExcel function you used gives me an error when I try it in my query. Is it a custom function I have to create beforehand?
Thank you.
- MarkLafSuper User
You may have to tweak the code depending on how you are connecting to your data. The key thing, though, is that you have to connect to the folder/directory that contains your files rather than connect individually to each file.
So, the key two steps to focus on to get working in your scenario:
Source = Folder.Files("<local folder location with xlsx>"), ParseExcel = List.Transform( Source[Content], Excel.Workbook ),The first step, Source, is connecting to a folder on my local machine. This should be switched out to the appropriate connection (e.g. SharePoint Folder, etc.)
In my example, the output of my Source step looks like:
This snip of the output structure above is key to understanding what happens in the next step, PraseExcel. For the first argument of List.Transform, we reference the Content column containing the Excel binaries with Source[Content]. This is where you may have to tweak depending on your particular connection. The second argument of List.Transform is just calling the Excel.Workbook function, which is the main built-in function for parsing an Excel binary.
If you share your anonymized M, then people can provide more specific guidance.
- v-nmadadi-msftCommunity Support