Forum Discussion

jbot's avatar
jbot
Frequent Visitor
1 year ago
Solved

Power Query adding sequential columns

Hello,

I am trying to use Power Query within Excel to automatically load a single column of data from a workbook into the column directly to the right of the column that was previously uploaded. The trigger would be whenever a new workbook is uploaded to a SharePoint folder and the query is refreshed. I am able to create a query to overwrite the column I am wanting to upload to but i can't figure out how to make the query upload the new data to the column directly to the right of the previously uploaded column while still keeping the previously uploaded data in the column to the left of the new data. Basically, I want an automated solution for copy pasting one column of data into one spreadsheet as soon as the workbooks are uploaded to a sharepoint folder. 

Any help is appreciated!

Thank you.

  • Almost there, I think. Now that I have a better sense of the strucutre of your workbooks and the exact desired output, I think this should do it:

     

     

    let
        //connect to your site files
        Source = SharePoint.Contents( "<your site>", [ApiVersion = 15] ),
    
        //navigate to folder with xlsx
        #"Shared Documents" = Source{[Name="Shared Documents"]}[Content],
        #"Test File" = #"Shared Documents"{[Name="Test File"]}[Content],
    
        //perform the transformation on the xlsx, 
        //output will be a list of Column6's from all xlsx
        ParseAndDrill = 
        List.Transform( 
            #"Test File"[Content],
            each let
                //parse excel
                ParseExcel = Excel.Workbook( _ ),
                //navigate to the sheet you want
                OpenStudy1Sheet = ParseExcel{[Item="Study1",Kind="Sheet"]}[Data],
                //get single column you want as a list 
                //(want this format to construct table later)
                GetCol6 = OpenStudy1Sheet[Column6], 
                //remove nulls from the col
                RemoveNulls = List.Select( GetCol6, each _ <> null )
            in
                RemoveNulls
        ),
    
        //with list of Column6's from all xlsx, put into table
        ToTable = Table.FromColumns( ParseAndDrill )
    in
        ToTable

     

  • Yes, I think this should do it. We can use Table.TransformRows to get access to all the fields rather than just [Content]; then, all we have to do is tag [Date created] to the top of our Column6's:

     

    let
        //connect to your site files
        Source = SharePoint.Contents( "<your site>", [ApiVersion = 15] ),
    
        //navigate to folder with xlsx
        #"Shared Documents" = Source{[Name="Shared Documents"]}[Content],
        #"Test File" = #"Shared Documents"{[Name="Test File"]}[Content],
    
        //perform the transformation on the xlsx, 
        //output will be a list of Column6's from all xlsx
        ParseAndDrill = 
        Table.TransformRows( 
            #"Test File",
            each let
                //parse excel
                ParseExcel = Excel.Workbook( [Content] ),
                //navigate to the sheet you want
                OpenStudy1Sheet = ParseExcel{[Item="Study1",Kind="Sheet"]}[Data],
                //get single column you want as a list 
                //(want this format to construct table later)
                GetCol6 = OpenStudy1Sheet[Column6], 
                //remove nulls from the col + add Date created
                RemoveNulls = {[Date created]} & List.Select( GetCol6, each _ <> null )
            in
                RemoveNulls
        ),
    
        //with list of Column6's from all xlsx, put into table
        ToTable = Table.FromColumns( ParseAndDrill )
    in
        ToTable

19 Replies

    • jbot's avatar
      jbot
      Frequent Visitor

      Please see my reply to v-nmadadi-msft for more content on what I am trying to achieve.

  • Hi jbot , I don't think that's possible by creating multiple tables/power query.
    You could Read all the files (assuming they contain same data), and do a pivot to make the file name the header and load them into excel. This way all new files will appear as a new column to the right of existing columns.

     

    Better if you have a sample/screenshot of what you are trying to achieve

    • jbot's avatar
      jbot
      Frequent Visitor

      Please see my reply to v-nmadadi-msft for more content on what I am trying to achieve.

  • v-nmadadi-msft's avatar
    v-nmadadi-msft
    Community Support

    Hi jbot,

    May we know how exactly you are creating the query to overwrite the column, based on that information we will look for possible suggestions to reorder the column in correct way.

    Also in this forum you will find people who are good at Fabric and Power BI if you believe your query can be resolved by people who are good at Excel and who have experience using powerquery within Excel, you can utilize Excel community: Welcome to the Excel Community | Microsoft Community Hub 


    Thanks and Regards

    • jbot's avatar
      jbot
      Frequent Visitor

      Hello, 

      Thank you for the response. 

      I have uploaded a few pictures to show what I am trying to accomplish. The first picture shows shows the close and load to operation and the destination of the first imported column. The second picture shows the first file in the sharepoint folder being imported. There is only one file in the sharepoint folder at this point. This result is good. The problem comes when I add another excel file to the sharepoint folder. I manually added file "M1-8" to the folder to replicate what our system will be doing automatically. I then manually refreshed the query and the result showed in third picture that the query loaded the new file on top of my old file loaction while moving the previous column of data down to the bottom as shown in fourth picture. I desire the new data column to be uploaded to the column to the right of the previous column once a new file is added to the sharepoint folder. 

      I also uploaded a picture of the query editor page (picture 5). 

       

      Thank you for the help.

      Josh Query Load ToQuery Load to ResultM1-8 file AddedM1-8 file added 2query

  • You'll want to connect to the folder with all the workbooks, transform them all at once in a list so that you've got the list of single columns from each workbook, then convert that into a table. As new workbooks are added into the folder, new columns will get added.

     

    Here is the M with some examples. I'm assuming that the data are of different shapes in the workbooks.

     

    sample1.xlsx

    Column AColumn BColumn C
    1abc20
    2jwc73110
    3ova24920
    4ath26030
    5plm11140

     

    sample2.xlsx

    Column AColumn C
    5100
    41000
    210000
    1100000
    01000000

     

    sample3.xlsx

    Column AColumn C
    1100000
    o1000000

     

    FYI these are all just entered simply in Sheet1 of each workbook:

     

    Now, I start with just samples 1-2 in a local folder:

     

    Here is M that converts this into two columns:

     

     

    let
        Source = Folder.Files("<local folder location with xlsx>"),
    
        //main initial action is to parse workbooks into a list
        ParseExcel = List.Transform( Source[Content], Excel.Workbook ), 
    
        //now we can do whatever series of transformations we need that will
        //convert each list item / workbook into a single list (column) of data
        WhateverTransformsToOneColEach = 
        //I'm doing a bunch of transforms at once here,
        //but you can take whatever approach you want.
        //The main thing is that the output needs to be 
        //a list of column data per workbook
        List.Transform( 
            ParseExcel, 
            each let 
                openSheet1 = _{[Name="Sheet1"]}[Data],
                promoteHeaders = Table.PromoteHeaders( openSheet1 ),
                tableToCols = Table.ToColumns( promoteHeaders ),
                combineColsIntoOne = List.Combine( tableToCols )
            in
                combineColsIntoOne
        ),
    
        ToTable = Table.FromColumns( WhateverTransformsToOneColEach )
    
    in
        ToTable

     

     

    What I mean by "output needs to be a list of column data per workbook" - this is the output of my WhateverTransformsToOneColEach step showing preview of each workbook output / list item:

     

    Output on load (including some random data in columns to left to mimic your setup):

     

    Now, I'll drop in sample3.xlsx

     

    And hit refresh in my workbook - the parsing in our query, the <WhateverTransformsToOneColEach> part, needs to be able to handle whatever shape sample3.xlsx or any new workbooks will be in:

     

     

    • jbot's avatar
      jbot
      Frequent Visitor

      Hello,

      Thank you for looking into my issue. I am not very familar with M, the ParseExcel function you used gives me an error when I try it in my query. Is it a custom function I have to create beforehand?

      Thank you. 

      • MarkLaf's avatar
        MarkLaf
        Super User

        You may have to tweak the code depending on how you are connecting to your data. The key thing, though, is that you have to connect to the folder/directory that contains your files rather than connect individually to each file.

         

        So, the key two steps to focus on to get working in your scenario:

         

            Source = Folder.Files("<local folder location with xlsx>"),
        
            ParseExcel = List.Transform( Source[Content], Excel.Workbook ), 

         

        The first step, Source, is connecting to a folder on my local machine. This should be switched out to the appropriate connection (e.g. SharePoint Folder, etc.)

         

        In my example, the output of my Source step looks like:

         

        This snip of the output structure above is key to understanding what happens in the next step, PraseExcel. For the first argument of List.Transform, we reference the Content column containing the Excel binaries with Source[Content]. This is where you may have to tweak depending on your particular connection. The second argument of List.Transform is just calling the Excel.Workbook function, which is the main built-in function for parsing an Excel binary.

         

        If you share your anonymized M, then people can provide more specific guidance.

  • v-nmadadi-msft's avatar
    v-nmadadi-msft
    Community Support

    Hi jbot 

    May I ask if you have resolved this issue by the reply provided by MarkLaf ? If so, please mark the helpful reply and accept it as the solution. This will be helpful for other community members who have similar problems to solve it faster.

    Thank you.