Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Get certified in Microsoft Fabric—for free! For a limited time, get a free DP-600 exam voucher to use by the end of 2024. Register now

Reply
jcru1999
Frequent Visitor

Automate Duplicate data sets in Power Query

Hi Community, 

I hope you are doing well, I wanted to ask for some guidance. I work in HR and we are required to keep headcounts as of last day of the month. We can connect to the data source (Workday) using the web service and importing the information in a JSON format. 

To keep track of the monthly view, we must duplicate the previous month data set and edit the link dates to reflect the closing month (example of link: wd3-services1.myworkday.com/ccx/customreport2/XXX/IDCODE/All_Employee_-_Web_Services_Copy?Effective_Date=2023-01-31-08%3A00&Worker_Active=1&format=json // here we would edit the date to the closing month and in other reports to saturday of each week).

 

I was wondering if there is an easier way to make this automatized, as each month/week we must go back to power query and do this process manually, we have the same problem with other reports. Is there an automatic way to do this that we are missing?

 

Any suggestions are appreciated! 

Thank you, 

1 ACCEPTED SOLUTION
v-yueyunzh-msft
Community Support
Community Support

Hi, @jcru1999 

According to your description, your data comes from a URL, and you need to modify the fixed parameters of the URL every time you get the data.
If you want to automate, I think you can define the way you get the data in Power Query as a custom function.

The parameter passed in is your date, and then you create a date column at the end of the month.

The end of month's date column you can use this M code(put this in the "Advanced Editor"):

let
    Source = List.Distinct(List.Transform(List.Dates(#date(2023,1,1), Duration.Days(Duration.From(#date(2023,12,31)-#date(2023,1,1))) +1, #duration(1, 0, 0, 0))
,(x)=> Date.EndOfMonth(x))),
    #"Converted to Table" = Table.FromList(Source, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
    #"Changed Type" = Table.TransformColumnTypes(#"Converted to Table",{{"Column1", type date}})
in
    #"Changed Type"

vyueyunzhmsft_0-1675647800494.png

Then you can use "Add custom column."
And pass in this [Column1] column to call the Custom Column you wrote to return your data row by row.

 

For more information, you can refer to this:
How to use Power Query Custom Functions (exceloffthegrid.com)

 

Thank you for your time and sharing, and thank you for your support and understanding of PowerBI! 

 

Best Regards,

Aniya Zhang

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly

View solution in original post

4 REPLIES 4
v-yueyunzh-msft
Community Support
Community Support

Hi, @jcru1999 

According to your description, your data comes from a URL, and you need to modify the fixed parameters of the URL every time you get the data.
If you want to automate, I think you can define the way you get the data in Power Query as a custom function.

The parameter passed in is your date, and then you create a date column at the end of the month.

The end of month's date column you can use this M code(put this in the "Advanced Editor"):

let
    Source = List.Distinct(List.Transform(List.Dates(#date(2023,1,1), Duration.Days(Duration.From(#date(2023,12,31)-#date(2023,1,1))) +1, #duration(1, 0, 0, 0))
,(x)=> Date.EndOfMonth(x))),
    #"Converted to Table" = Table.FromList(Source, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
    #"Changed Type" = Table.TransformColumnTypes(#"Converted to Table",{{"Column1", type date}})
in
    #"Changed Type"

vyueyunzhmsft_0-1675647800494.png

Then you can use "Add custom column."
And pass in this [Column1] column to call the Custom Column you wrote to return your data row by row.

 

For more information, you can refer to this:
How to use Power Query Custom Functions (exceloffthegrid.com)

 

Thank you for your time and sharing, and thank you for your support and understanding of PowerBI! 

 

Best Regards,

Aniya Zhang

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly

Thank you  v-yueyunzh-msft! I'll give it a go but so far it seems like a good solution. 

amitchandak
Super User
Super User

@jcru1999 , You can create date with help from

 

DateTime.Date(DateTime.LocalNow())

 

or

 

Date.StartOfYear(DateTime.Date(DateTime.LocalNow()))

 

or

Date.EndOfYear(DateTime.Date(DateTime.LocalNow()))

 

 

You can use Date.ToText to get desired format

 

Date.AddMonth to go to last month

 

refer: https://www.youtube.com/playlist?list=PLPaNVDMhUXGZdm-2DIMrS54Z2J0TAxj0X

Join us as experts from around the world come together to shape the future of data and AI!
At the Microsoft Analytics Community Conference, global leaders and influential voices are stepping up to share their knowledge and help you master the latest in Microsoft Fabric, Copilot, and Purview.
️ November 12th-14th, 2024
 Online Event
Register Here

Thank you amitchandak! The resources in your youtube channel are very helful, thank you, I'll also give it a try! 

Helpful resources

Announcements
November Carousel

Fabric Community Update - November 2024

Find out what's new and trending in the Fabric Community.

Live Sessions with Fabric DB

Be one of the first to start using Fabric Databases

Starting December 3, join live sessions with database experts and the Fabric product team to learn just how easy it is to get started.

Las Vegas 2025

Join us at the Microsoft Fabric Community Conference

March 31 - April 2, 2025, in Las Vegas, Nevada. Use code MSCUST for a $150 discount! Early Bird pricing ends December 9th.

Nov PBI Update Carousel

Power BI Monthly Update - November 2024

Check out the November 2024 Power BI update to learn about new features.