Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Calling all Data Engineers! Fabric Data Engineer (Exam DP-700) live sessions are back! Starting October 16th. Sign up.

Reply
roncruiser
Post Patron
Post Patron

Creating Index within Index

Hi, I'm struggling to create a redundant index column and another index column which references the redundant index column.

 

I've grouped a set of data using the Group By function and within each Group By table I need the following two index columns.

 

Example>

Index1, Index2

0,0

0,1

0,2

0,3

0,4

1,0

1,1

1,2

1,3

1,4

2,0

2,1

2,2

2,3

2,4

 

Note: The first index columns actually has 10 repeats but I used 5 to shorten this example.

Thanks... 

1 ACCEPTED SOLUTION
ziying35
Impactful Individual
Impactful Individual

Hi, @roncruiser 

The complete code, I've changed it, you just need to follow the picture below to modify the parameters to achieve your desired effect.

2.png

 

 

// output
let
    Source = Excel.CurrentWorkbook(){[Name="Table2"]}[Content],
    chType = Table.TransformColumnTypes(Source,{{"Delay", type number}}),
    fx = (tbl as table, Coarse_start as number, Fine_start as number, Fine_end as number)=>
       let
         sortedTbl = Table.Sort(tbl, {"Delay", 0}),
         rows = List.Buffer(Table.ToRows(sortedTbl)),
         n = List.Count(rows),
         gen = List.Generate(
                   ()=>{0, {}, 0},//{counter, {new_list}, delay_counter}
                   each _{0}<=n,
                   each let count = Fine_end+1-Fine_start,
                            Index1 = _{0},
                            Coarse = Number.IntegerDivide(_{0}, count)+Coarse_start,
                            Fine = Number.Mod(_{0}, count)+Fine_start
                        in  {_{0}+1, {Index1, Coarse, Fine}, _{0}},
                   each List.InsertRange(rows{_{2}}, 4, _{1})
          ),
        toTbl = Table.FromRows(
                     List.Skip(gen), 
                     List.InsertRange(
                           Table.ColumnNames(sortedTbl), 
                           4, 
                           {"Index1", "Coarse", "Fine"}
                      )
                )
      in
        toTbl,
    group = Table.Group(chType, "Ref", {"t", each fx(_, 3, 4, 11)})[t],
    result = Table.Combine(group)
in
    result

 

View solution in original post

40 REPLIES 40
ziying35
Impactful Individual
Impactful Individual

Hi, @roncruiser 

Change the code for the query called Example in the sample file you uploaded to the Google Drive to the following:

 

// Example
let
    Source = Excel.CurrentWorkbook(){[Name="Table2"]}[Content],
    fnRec = (tbl)=> List.Transform({0..Table.RowCount(tbl)}, each [Index1=_, Index2=Number.IntegerDivide(_, 10), Index3=Number.Mod(_, 10)]),
    fx = Function.ScalarVector(
            type function(rec as text) as text,
            (tbl)=>
                 let t=Table.Buffer(tbl) in fnRec(t)
          ),
    group = Table.Group(Source, {"Ref"}, {"SubIndexes", each Table.AddColumn(_, "rec", (r)=>fx(r))})[SubIndexes],
    cmbTbls = Table.Combine(group),
    expd = Table.ExpandRecordColumn(cmbTbls, "rec", {"Index1", "Index2", "Index3"})
in
    expd

 

 

The result of the code run is shown below:

4.png

@ziying35 

 

Almost!  The trick is getting the Delay column to Sort Ascending per each Ref.  Then add the index columns.

 

  1. Sort Delay column Ascending per each Ref.  The lowest Delay values per each Ref is the 0 starting point for each added index column.
  2. Then add index columns.  The trick here is the Coarse starting value may change, and the Fine range may change.  This is tricky.  I'm working on it but I keep running into dead ends.  

I've added a before and after example in a previous post.

 

 

ImkeF
Community Champion
Community Champion

Hi @roncruiser ,

please check this screencast instruction for it : https://www.youtube.com/watch?v=-3KFZaYImEY 

 

Imke Feldmann (The BIccountant)

If you liked my solution, please give it a thumbs up. And if I did answer your question, please mark this post as a solution. Thanks!

How to integrate M-code into your solution -- How to get your questions answered quickly -- How to provide sample data -- Check out more PBI- learning resources here -- Performance Tipps for M-queries

@ImkeF 

 

Thanks for the Quick respose but not quite what I was looking for. Please correct me.  The solution you provided adds an index column to each Grouped by table.  I've already had that. 

 

I need an addtional two columns per GroupBy table.  For example.

 

Within each already created GroupBy table:

 

Index1(have),Index2(need),Index3(need)

0,0,0

1,0,1

2,0,2

3,0,3

4,0,4

5,1,0

6,1,1

7,1,2

8,1,3

9,1,4

10,2,0

11,2,1

12,2,2

13,2,3

14,2,4

 

Hope that clarifies better, maybe you did give the solution and I don't see it.

 

ImkeF
Community Champion
Community Champion

Hi @roncruiser ,

it's hard for me what you're after here.

But the figures you've provided can be achieved by the following code:

 

let
    Source = {0..14},
    #"Converted to Table" = Table.FromList(Source, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
    #"Inserted Integer-Division" = Table.AddColumn(#"Converted to Table", "Integer-Division", each Number.IntegerDivide([Column1], 5), Int64.Type),
    #"Inserted Modulo" = Table.AddColumn(#"Inserted Integer-Division", "Modulo", each Number.Mod([Column1], 5), type number)
in
    #"Inserted Modulo"

 

Imke Feldmann (The BIccountant)

If you liked my solution, please give it a thumbs up. And if I did answer your question, please mark this post as a solution. Thanks!

How to integrate M-code into your solution -- How to get your questions answered quickly -- How to provide sample data -- Check out more PBI- learning resources here -- Performance Tipps for M-queries

@roncruiser - perhaps you could provide some specific data with expected results that would help us see what you are doing. To me, it seems you can just add an index, then add another sub-index of 0-4 using this code.

let
    Source = {0..10},
    #"Converted to Table" = Table.FromList(Source, Splitter.SplitByNothing(), {"First Index"}, null, ExtraValues.Error),
    #"Added Custom" = Table.AddColumn(#"Converted to Table", "Second Index", each {1..4}),
    #"Expanded Second Index" = Table.ExpandListColumn(#"Added Custom", "Second Index")
in
    #"Expanded Second Index"

edhans_0-1598296274527.png

But without data, we are just guessing. See links at bottom for providing actual data.

 

How to use M code provided in a blank query:
1) In Power Query, select New Source, then Blank Query
2) On the Home ribbon, select "Advanced Editor" button
3) Remove everything you see, then paste the M code I've given you in that box.
4) Press Done
5) See this article if you need help using this M code in your model.

 

How to get good help fast. Help us help you.
How to Get Your Question Answered Quickly
How to provide sample data in the Power BI Forum

 



Did I answer your question? Mark my post as a solution!
Did my answers help arrive at a solution? Give it a kudos by clicking the Thumbs Up!

DAX is for Analysis. Power Query is for Data Modeling


Proud to be a Super User!

MCSA: BI Reporting

@edhans 

@ImkeF 

 

Here's a link to the file containing the data I showed in my example.

https://drive.google.com/file/d/15bMAH979oPxViF9yH8W-ho1XsBPx8Yoz/view?usp=sharing

 

Hope you can help.  I'm stumped. 

 

-Ron

Can you explain the 3 index columns? Why is the first one 0,0,0, and the 2nd one 1,0,1, and the 9th one 9,0,9, and the 10th one 10,1,0?

I don't see any logic to use.



Did I answer your question? Mark my post as a solution!
Did my answers help arrive at a solution? Give it a kudos by clicking the Thumbs Up!

DAX is for Analysis. Power Query is for Data Modeling


Proud to be a Super User!

MCSA: BI Reporting

@edhans 

 

Good question.

 

In short, the columns add fine tune controls to a time delay mechanism.

 

Column Index1 simply is an index colum to add an absolute delay control per "Ref".  We can then work other calculations based on the values in that column.  (Column Index1 is nice to have.  It's well documented on how to add this index column per table.)

 

Columns Index2 and Index3 work together.  (These two columns are valuable and are Must Have columns.)

Index2 is a Coarse (time) delay knob.

Index3 is a Fine (time) Step delay knob.

 

The 3 columns allow us to see a delay parameter against the data we visualize.  Moreso with columns Index2 and Index3.

 

Maybe something to anchor off is Index3.  There are 10 Fine Steps per each Coarse delay.

 

(With the caveat that the quantity of steps may change per coarse.  The Fine Step start value may change as well.  Our default is 10 Fine Steps per coarse with the steps starting at 0 and ending at 9.  Sometimes we use 6 Fine Steps per coarse with the fine steps starting 3 and ending 8.  12 Fine Steps per coarse with the fine steps starting at 4 and ending at 15. )

 

 

ImkeF
Community Champion
Community Champion

Hi @roncruiser ,

I still don't understand how the data you've provided should actually match together.

Could you please provide sample data that have a reall "before and after"-scenario that allows the business logic to follow up?

 

Imke Feldmann (The BIccountant)

If you liked my solution, please give it a thumbs up. And if I did answer your question, please mark this post as a solution. Thanks!

How to integrate M-code into your solution -- How to get your questions answered quickly -- How to provide sample data -- Check out more PBI- learning resources here -- Performance Tipps for M-queries

@ImkeF Sure.

 

Sure.  This file contains before and after data.  Each on it's own tab.

 

The "after" sample has been truncated to include just a few Refs

 

https://drive.google.com/file/d/15Ep2jzlHm-q5qC7bM3e5XDDfaPk07vwC/view?usp=sharing

ziying35
Impactful Individual
Impactful Individual

@roncruiser 

try this:

// Before
let
    Source = Excel.CurrentWorkbook(){[Name="Table2"]}[Content],
    chType = Table.TransformColumnTypes(Source,{{"Delay", type number}}),
    fnRec = (tbl)=> List.Transform({0..Table.RowCount(tbl)}, each [Index1=_, Coarse=Number.IntegerDivide(_, 10), Fine=Number.Mod(_, 10)]),
    fx = Function.ScalarVector(
            type function(rec as text) as text,
            (tbl)=>
                 let t=Table.Buffer(tbl) in fnRec(t)
          ),
    group = Table.Group(chType, {"Ref"}, {"SubIndexes", each Table.AddColumn( Table.Sort(_, {"Delay", 0}), "rec", (r)=>fx(r))})[SubIndexes],
    cmbTbls = Table.Combine(group),
    expd = Table.ExpandRecordColumn(cmbTbls, "rec", {"Index1", "Coarse", "Fine"})
in
    expd

 

@ziying35   Sure.   This looks good. 

 

  • So, default settings for the Coarse start value is 0, and the Fine range is 0-9.
  •  Is it possible add functionality to set the Coarse Value to start at n start value and adjust the Fine range from n to +n?
    • For example: Coarse start value at 3.  With Fine range from 3-9 steps.

 

70% of the time we use default settings, but some times we vary the Coarse and Fine range parameters.

 

Thanks again for the work... I have to settle down and digest your code....

 

ziying35
Impactful Individual
Impactful Individual

@roncruiser 

 

 

// sample
let
    Source = {0..100},
    n = List.Count(Source),
    gen = List.Generate(
               ()=>{0, {}},
               each _{0}<=n, 
               each let Coarse=Number.IntegerDivide(_{0}, 10)+3
                    in  if _{1}{3}? =null or _{1}{3}=9 then {_{0}+1, {0, _{0}, Coarse , 3}}
                    else {_{0}+1, {_{1}{0}+1, _{0}, Coarse, _{1}{3}+1}},
               each List.Skip(_{1})
          ),
    toTbl = Table.FromRows(List.Skip(gen), {"Index1", "Coarse", "Fine"})
in
    toTbl

 

5.png

 

 

You see if I understand what you're saying, and if that's what you want, then I'll go ahead and write the whole thing.

What I'm trying to determine is how many times I need to repeat the same Coarse value, now I'm writing the code to repeat it 10 times, if the number of times the Coarse value needs to be repeated is variable, then under what conditions does it change?

@ziying35 Thank you.   Here are a few examples I created.

 

Annotation 2020-08-26 092707_version_examples.jpg

 

Thank you.

 

ziying35
Impactful Individual
Impactful Individual

Hi, @roncruiser 

The complete code, I've changed it, you just need to follow the picture below to modify the parameters to achieve your desired effect.

2.png

 

 

// output
let
    Source = Excel.CurrentWorkbook(){[Name="Table2"]}[Content],
    chType = Table.TransformColumnTypes(Source,{{"Delay", type number}}),
    fx = (tbl as table, Coarse_start as number, Fine_start as number, Fine_end as number)=>
       let
         sortedTbl = Table.Sort(tbl, {"Delay", 0}),
         rows = List.Buffer(Table.ToRows(sortedTbl)),
         n = List.Count(rows),
         gen = List.Generate(
                   ()=>{0, {}, 0},//{counter, {new_list}, delay_counter}
                   each _{0}<=n,
                   each let count = Fine_end+1-Fine_start,
                            Index1 = _{0},
                            Coarse = Number.IntegerDivide(_{0}, count)+Coarse_start,
                            Fine = Number.Mod(_{0}, count)+Fine_start
                        in  {_{0}+1, {Index1, Coarse, Fine}, _{0}},
                   each List.InsertRange(rows{_{2}}, 4, _{1})
          ),
        toTbl = Table.FromRows(
                     List.Skip(gen), 
                     List.InsertRange(
                           Table.ColumnNames(sortedTbl), 
                           4, 
                           {"Index1", "Coarse", "Fine"}
                      )
                )
      in
        toTbl,
    group = Table.Group(chType, "Ref", {"t", each fx(_, 3, 4, 11)})[t],
    result = Table.Combine(group)
in
    result

 

Adding control for Coarse Start Value and the Fine Step Range allows control of the time delay sensitivity for each Ref step. This allows finer control of the delay which ultimately allows a wider optimization range. 

ziying35
Impactful Individual
Impactful Individual

@roncruiser 

I'm working on the code. I'll repost it  when I'm done

ImkeF
Community Champion
Community Champion

Hi @roncruiser ,

the error must come from an earlier step or some changes you've made to the function code.
Can you share the full M-code that you've used please?

 

Imke Feldmann (The BIccountant)

If you liked my solution, please give it a thumbs up. And if I did answer your question, please mark this post as a solution. Thanks!

How to integrate M-code into your solution -- How to get your questions answered quickly -- How to provide sample data -- Check out more PBI- learning resources here -- Performance Tipps for M-queries

@ImkeF  Sure.  Thanks. The code I used is basically a copy and paste of your code direct from the web page.  Pasted directly into a blank query.

 

Maybe something changes?  I'm running the latest version of the Exel office "MS Office Pro Plus".

I tried to substite the real source data but I get the same error.  The error always occurrs at row "Added Custom".  Up until that particular line, all lines show properly.

 

let
Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("ddtBbtxGEAXQu3idAKyuZje5laO5hJH7XyMyWyNrWC8LwwnxIwR5tkf/p/jr14/tx18/+naObfv9V18/Pv/290///vXrR5TU34FYq7GGWNZYItZrrCO219iO2KixgdissYnYUWMHYmeNnfrPWxX+DjoIQhIBipBFACOkEeAIeQRAQiIBkpBJACWkEmAJuQRgQjINMk0yDTKNv0f0m0QyDTJNMg0yTTINMk0yDTJNMg0yTTINMk0yDTJNMgmZlExCJiWTkEn++aU/wCSTkEnJJGRSMgmZlExCJiWTkEnJJGRSMh0yXTIdMl0yHTJdMh0ynZ8t+nCRTIdMl0yHTJdMh0yXTIdMl0yHTJfMDpldMjtkdsnskNkls0Nml8wOmZ2f+/rgl8wOmV0yO2R2yeyQ2SWzQ2aXzIDMkMyAzJDMgMyQzIDMkMyAzJDMgMzg92T6pkwyAzJDMgMyQzIDMkMyEzJTMhMyUzITMlMyEzJTMhMyUzITMlMyEzKT3y/rG2bJTMhMyUzITMkckLmebXELQuZ6VoKQuZ4d9yBkrmdv5y0ImevZ+XYLQuZ69va4BSFzPXt7O16DkLmePR63oMrMsYLxEoTM9ewjuH8PnpC5npUgZK5nJQiZ61kJQuZ69hH850l+BSFzPXsGz2cQMtezx+N8DULmevbxFc+XIGTO+Sf49hWEzHn8+Xd8PJ5BFc0TwfXruFTNjdGKw6q5VRs2za3SsGhuVYY9c6swrJlbdWHL3CoLS+ZWVdgxt4rC8o/2/z/lHyAUQft3+Uf7d/lH+3f5R/t3+Uf7d/lH+3f5R/t3+Uf7Z/kPtH+W/0D7Z/kPtH+W/0D7Z/kPtH+W/0D7Z/kPtH+W/0D7Z/kPtH+W/0D7Z/kPtH+W/0D7Z/kPtH+W/0D7Z/kPtH+W/0D7Z/kPtH+W/0D7Z/kPtH+W/0D7Z/kPtH+W/0D7Z/kPtH+W/0D7Z/kPtH+W/0D7Z/kPtH+W/0D7Z/kPtH+W/0D7Z/kPtH+W/0D7Z/kPtH+W/0D7Z/kPtH+W/0D7Z/kPtH+W/0D7Z/kPtH+W/0D7Z/kPtH+W/0D7Z/kPtH+W/0D7Z/kPtH+W/0D7Z/kPtH+W/0D7Z/kPtH+W/0D7Z/kPtH+W/0D7Z/kPtH+W/0D7fy3//RmsMq/l/ytYZV7L/1ewyryW//EM4lvmF5mPXyIrWGVey//H11/BKvNa/p9BtP/ju0yf75/BKnN8l+mPZ7DKHN9k+ng8PoNV5vgu8/MrWGWObzL9/StYZVat7f0WrDLXo+3n9QE2/wSrzPXo4yvegmgzx5+v+PgTrDKr1q6v+Hg8g2j/16P3n/v7y1dE+1/l/+d8f/2KVWaV/xKsMqv8v8/Ha7DKrPJfglVmlf8SrDKr/JdglVnlvwSrzGf5f9yCKJongm2rMs/uf0+iam7BJLrm1phE2dySSbTNrTOJurntTKJvboNJFM5tMonGuYmoYQtY+0BJag0IGnEOoJH2gKCRBoGgkRaBoJEmgaCRNoGgkUaBoJFWgaCRZoGgEXaBtRXUJIwajbAMrLWgJmHUaIRtYO0FNQmjRiOsA2sxqEkYNRphH1ibQU3CqNEIC8FaDWoSRkkjbARrN6hJDWs0wkqwloOahFHSCDvB2g5qEkZJIywFaz2oSRgljbAVrP2gJmHUaYS1YC0INQmjTiPsBdFphMFgjQg1CaNOI0wGa0aoSRh1GmE0WENCSWI1WEtCTcJopxF2g7Ul1CSMdhphOVhrQk1qoqYRtoO1J9QkjHYaYT1Yi0JNwminEfaDtSnUJIwGjbAgrFWhJmE0aIQNYe0KNQmjQSOsCGtZqEkYDRphR1jbQk3CaNAIS8JaF2oSRpNG2BLWvlCTMJo0wpqwFoaahNGkEfaEtTHUpP5nD42wKKyVoSZhNGmETWHtDDUJo4NGWBXW0lCTMDpohF1hbQ01CaODRlgW1tpQkzA6aIRtIQ4aYVyIg0ZYF+KkEeaFOGmEfSFOGmFgiJNGWBjipBEmhjhphI0hThphZIiTRlgZgjNDw8wQ3BkSO0PjzpDYGRp3htSVAXeGxM7QuDMkdobGnSGxMzTuDImdoXFnSOwMjTtDYmdo3BkSO0PjzpDYGRp3hsTO0LgzJHaGxp0hsTM07gyJnaFxZ0jsDI07Q2JnaNwZEjtD486Q2Bkad4bEztC4M6TuD7gzpA4QuDMkLxBopBME7gypGwTuDKkjBO4MqSsE7gypMwTuDKk7BO4MqUME7gyJnaFxZ0jsDI07Q2JnaNwZEjtD486Q2Bkad4bEztC4MyR2hsadIbEzNO4MiZ2hcWdI7AyNO0NiZ2jcGRI7Q+POkNgZGneGvO0M68253z8+/8nfP61gr0G+Z5e3mWEl+arKbWZYSb+rMpHkyyq3mWEl+bbKbWZYSb6ucpsZVpLvq9xmhpXkCyu3mWEl+S7RDiK/gZc7kai0S4mHeLmLiad4ucuJx3i5C4rneLlLigd5OUTFk7wcsuJRXg5h8Swvh7R4mJdDWjzNy8HfU9Qa0uJ5Xg5p8UAvh7R4opdDWjzSyyktnunllBYP9XJKy+/pTWn5Tb0pLb+rN6Xlt/Um/wik1pSW39ib0vI7e1NafmvvkNb/vLcnLb+5d0iL53t5SIsHfHlIiyd8eUiLR3x5SItnfHnwE4tah7R4ypeHtHjMl6e0eM6Xp7R40JentHjSl6e0eNSXp7R41pentHjYl6e0eNqXp7R43Jcnv8Gg1iktHvj1TVo88eubtHjk1zdp8cyvb9LioV/fpMVTv75Ji8d+fZMWz/36Ji0e/PVNWjz56xu/H6RWSItnfz2kxcO/HtLi6V8PafH4r4e0eP7XQ1o8AOwhLb7/10NafAOwh7T4DmAPafEtwN6kxfcAe5MW3wTsTVp8F7A3aR3UatLi+4C9SYtvBPYmrevTtEaldX2aHm/3qLTWe4ElKi2+GdhTWten6Uc0XqNsW3o7sKe0Pkf8c3+NSutzxb9HpfU543+L/vsf", BinaryEncoding.Base64), Compression.Deflate)),

let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [Iteration = _t, Frequency = _t, Ref = _t, Delay = _t, Capture1 = _t, Capture2 = _t]),
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Iteration", Int64.Type}, {"Frequency", Int64.Type}, {"Ref", Int64.Type}, {"Delay", Int64.Type}, {"Capture1", type text}, {"Capture2", type text}}),
fnTransformation = (myPartition as table) => let
#"Sorted Rows" = Table.Buffer(Table.Sort(myPartition,{{"Delay", Order.Ascending}})),
#"Added Index" = Table.AddIndexColumn(#"Sorted Rows", "Index", 0, 1, Int64.Type),
#"Inserted Integer-Division" = Table.AddColumn(#"Added Index", "Coarse", each Number.IntegerDivide([Index], 10), Int64.Type),
#"Inserted Modulo" = Table.AddColumn(#"Inserted Integer-Division", "Fine", each Number.Mod([Index], 10), type number)
in
#"Inserted Modulo",
#"Grouped Rows" = Table.Group(#"Changed Type", {"Ref"}, {{"All", each _, type table [Iteration=nullable number, Frequency=nullable number, Ref=nullable number, Delay=nullable number, Capture1=nullable text, Capture2=nullable text]}}),
#"Added Custom" = Table.AddColumn(#"Grouped Rows", "ApplyFunction", each fnTransformation([All])),  // Error occurrs here
#"Expanded ApplyFunction" = Table.ExpandTableColumn(#"Added Custom", "ApplyFunction", {"Iteration", "Frequency", "Delay", "Capture1", "Capture2", "Index", "Coarse", "Fine"}, {"Iteration", "Frequency", "Delay", "Capture1", "Capture2", "Index", "Coarse", "Fine"})
in
#"Expanded ApplyFunction"

 

Thanks for the responsiveness @ImkeF 

 

 

Helpful resources

Announcements
FabCon Global Hackathon Carousel

FabCon Global Hackathon

Join the Fabric FabCon Global Hackathon—running virtually through Nov 3. Open to all skill levels. $10,000 in prizes!

October Power BI Update Carousel

Power BI Monthly Update - October 2025

Check out the October 2025 Power BI update to learn about new features.

FabCon Atlanta 2026 carousel

FabCon Atlanta 2026

Join us at FabCon Atlanta, March 16-20, for the ultimate Fabric, Power BI, AI and SQL community-led event. Save $200 with code FABCOMM.

Top Kudoed Authors