Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
6 years ago
Solved

Dynamic date data retreival

Hello all,

Im trying to alter the SQL query retreiving all of my data to only retreive Today - 4 days and everything after. This is because im using this specific dataflow to refresh without hitting our SQL server too hard.

the SQL querry is LONG and has countless if statements because powerquery. i wouldnt really know where to insert my statement.
Can anyone help me with this specific instance? 

Context: This query is used in my source table of my dataflow. This dataflow only needs to show very recent data which we want to have "live". I know its not actually going to be live, but thats not the point, we need it refreshed at specific points of the day so my collegues have more to work off. a few days of data will reduce the time allot in compared to the 1400 days its trying to load now, which takes 25 minutes. 

Sql.Database(#"Database Server", #"Database Name", [Query="#(lf)#(lf)select#(lf)#(lf)Leg.""Legnr"", #(lf)Leg.""SalesOrder"",#(lf)ROW_NUMBER() OVER (PARTITION BY Leg.SalesOrder ORDER BY Legnr) AS SalesOrderLine,#(lf)Leg.""LegStatus"", #(lf)Leg.""Planned"",#(lf)Leg.""SalesInvoice"", #(lf)isnull(Leg.EDIProvider,'987654321') as EDIProvider,#(lf)Leg.""Distance"", #(lf)Leg.""OrderLeg"", #(lf)Leg.""PlanGroup"" as 'FK PlanGroup', #(lf)Leg.""FinDate"", #(lf)Leg.""TotalAmount"", #(lf)Leg.""Costs"", #(lf)Leg.""CMRReceived"",#(lf)Leg.[EmailSalesInvoice],#(lf)Leg.[Reference],#(lf)Leg.""CreationUser"",#(lf)Leg.[Cargo],#(lf)--Leg.BeginActivity,#(lf)--leg.EndActivity,#(lf)isnull(B.""Address"",(b.Activitynr + 900000000)) as BeginAddress, #(lf)B.""Date"" as BeginDate, #(lf)B.""DateTill"" as BeginDateTill, #(lf)B.""Time"" as BeginTime, #(lf)B.""TimeTill"" as BeginTimeTill,#(lf)B.""ResourceCombination"" as BeginResourceCombination, #(lf)B.""TripID"" as BeginTripID, #(lf)B.""StartDate"" as BeginStartDate, #(lf)B.""StartTime"" as BeginStartTime, #(lf)B.""EndDate"" as BeginEndDate, #(lf)B.""EndTime"" as BeginEndTime, #(lf)B.""ActivityDuration"" as BeginActivityDuration,#(lf)isnull(E.""Address"",(e.activitynr + 800000000)) as EndAddress, #(lf)E.""Date"" as EndDate, #(lf)E.""DateTill"" as EndDateTill, #(lf)E.""Time"" as EndTime, #(lf)E.""TimeTill"" as EndTimeTill, #(lf)E.""ResourceCombination"" as EndResourceCombination, #(lf)E.""TripID"" as EndTripID, #(lf)E.""StartDate"" as EndStartDate, #(lf)E.""StartTime"" as EndStartTime, #(lf)E.""EndDate"" as EndEndDate, #(lf)E.""EndTime"" as EndEndTime, #(lf)E.""ActivityDuration"" as EndActivityDuration,#(lf)leg.CreationDate,#(lf)Sum(Cargo.""Weight"") as Weight, #(lf)Sum(Cargo.""LoadingMeter"") as LoadingMeter, #(lf)Sum(Cargo.""PalletPlaces"") as PalletPlaces,#(lf)CASE WHEN legnr in (SELECT DISTINCT [leg] AS [pl_orders]#(lf)#(tab)#(tab)#(tab)#(tab)#(tab)FROM SO_Cargo #(lf)#(tab)#(tab)#(tab)#(tab)#(tab)WHERE Unit = 78) THEN 'pl' ELSE NULL END AS [pl order] -- unit 78 = pl#(lf)from SO_Leg as Leg#(lf)left join SO_Activity as B on leg.BeginActivity = B.Activitynr#(lf)left join SO_Activity as E on leg.EndActivity = E.Activitynr#(lf)left join so_cargo as cargo on leg.[legnr] = cargo.Leg#(lf)where FinDate >= DATEADD(year,-3,convert (date,DATEADD(YEAR, DATEDIFF(YEAR,0,GETDATE()) + 1, 0))) #(lf)#(lf)group by #(lf)Leg.""Legnr"", #(lf)Leg.""SalesOrder"",#(lf)Leg.""LegType"", #(lf)Leg.""LegStatus"", #(lf)Leg.""Planned"",#(lf)Leg.""SalesInvoice"", #(lf)Leg.EDIProvider,#(lf)Leg.""Distance"", #(lf)Leg.""OrderLeg"", #(lf)Leg.""PlanGroup"", #(lf)Leg.""FinDate"", #(lf)Leg.""TotalAmount"", #(lf)Leg.""Costs"", #(lf)Leg.""CMRReceived"",#(lf)Leg.[EmailSalesInvoice],#(lf)Leg.[Reference],#(lf)Leg.""CreationUser"",#(lf)Leg.Cargo,#(lf)--Leg.BeginActivity,#(lf)--Leg.EndActivity,#(lf)leg.CreationDate,#(lf)B.Activitynr,#(lf)B.""ActivityKind"", #(lf)B.""Address"", #(lf)B.""Date"", #(lf)B.""DateTill"", #(lf)B.""Time"", #(lf)B.""TimeTill"", #(lf)B.""ResourceCombination"", #(lf)B.""TripID"", #(lf)B.""StartDate"", #(lf)B.""StartTime"", #(lf)B.""EndDate"", #(lf)B.""EndTime"", #(lf)B.""ActivityDuration"",#(lf)E.""ActivityKind"", #(lf)E.""Address"", #(lf)E.""Date"", #(lf)E.""DateTill"", #(lf)E.""Time"", #(lf)E.""TimeTill"",#(lf)E.""ResourceCombination"", #(lf)E.""TripID"", #(lf)E.""StartDate"", #(lf)E.""StartTime"", #(lf)E.""EndDate"", #(lf)E.""EndTime"", #(lf)E.""ActivityDuration"",#(lf)E.Activitynr#(lf)#(lf)#(lf)#(lf)#(lf)"])

 

  • edhans's avatar
    edhans
    6 years ago

    Here is your actual SQL statement. You'll need to head over to a SQL forum to get someone to help. Someone here might be able to assist, but a SQL forum as a bunch of people that can assist.

    SELECT Leg."legnr", 
           Leg."salesorder", 
           Row_number() 
             OVER ( 
               partition BY Leg.salesorder 
               ORDER BY legnr)                               AS SalesOrderLine, 
           Leg."legstatus", 
           Leg."planned", 
           Leg."salesinvoice", 
           Isnull(Leg.ediprovider, '987654321')              AS EDIProvider, 
           Leg."distance", 
           Leg."orderleg", 
           Leg."plangroup"                                   AS 'FK PlanGroup', 
           Leg."findate", 
           Leg."totalamount", 
           Leg."costs", 
           Leg."cmrreceived", 
           Leg.[emailsalesinvoice], 
           Leg.[reference], 
           Leg."creationuser", 
           Leg.[cargo], 
           --Leg.BeginActivity, 
           --leg.EndActivity, 
           Isnull(B."address", ( b.activitynr + 900000000 )) AS BeginAddress, 
           B."date"                                          AS BeginDate, 
           B."datetill"                                      AS BeginDateTill, 
           B."time"                                          AS BeginTime, 
           B."timetill"                                      AS BeginTimeTill, 
           B."resourcecombination"                           AS 
           BeginResourceCombination, 
           B."tripid"                                        AS BeginTripID, 
           B."startdate"                                     AS BeginStartDate, 
           B."starttime"                                     AS BeginStartTime, 
           B."enddate"                                       AS BeginEndDate, 
           B."endtime"                                       AS BeginEndTime, 
           B."activityduration"                              AS 
           BeginActivityDuration, 
           Isnull(E."address", ( e.activitynr + 800000000 )) AS EndAddress, 
           E."date"                                          AS EndDate, 
           E."datetill"                                      AS EndDateTill, 
           E."time"                                          AS EndTime, 
           E."timetill"                                      AS EndTimeTill, 
           E."resourcecombination"                           AS 
           EndResourceCombination, 
           E."tripid"                                        AS EndTripID, 
           E."startdate"                                     AS EndStartDate, 
           E."starttime"                                     AS EndStartTime, 
           E."enddate"                                       AS EndEndDate, 
           E."endtime"                                       AS EndEndTime, 
           E."activityduration"                              AS EndActivityDuration, 
           leg.creationdate, 
           Sum(Cargo."weight")                               AS Weight, 
           Sum(Cargo."loadingmeter")                         AS LoadingMeter, 
           Sum(Cargo."palletplaces")                         AS PalletPlaces, 
           CASE 
             WHEN legnr IN (SELECT DISTINCT [leg] AS [pl_orders] 
                            FROM   so_cargo 
                            WHERE  unit = 78) THEN 'pl' 
             ELSE NULL 
           END                                               AS [pl order] 
    -- unit 78 = pl 
    FROM   so_leg AS Leg 
           LEFT JOIN so_activity AS B 
                  ON leg.beginactivity = B.activitynr 
           LEFT JOIN so_activity AS E 
                  ON leg.endactivity = E.activitynr 
           LEFT JOIN so_cargo AS cargo 
                  ON leg.[legnr] = cargo.leg 
    WHERE  findate >= Dateadd(year, -3, CONVERT (DATE, Dateadd(year, Datediff(year, 
                                                                     0, 
                                                                     Getdate()) + 1, 
                                                                         0))) 
    GROUP  BY Leg."legnr", 
              Leg."salesorder", 
              Leg."legtype", 
              Leg."legstatus", 
              Leg."planned", 
              Leg."salesinvoice", 
              Leg.ediprovider, 
              Leg."distance", 
              Leg."orderleg", 
              Leg."plangroup", 
              Leg."findate", 
              Leg."totalamount", 
              Leg."costs", 
              Leg."cmrreceived", 
              Leg.[emailsalesinvoice], 
              Leg.[reference], 
              Leg."creationuser", 
              Leg.cargo, 
              --Leg.BeginActivity, 
              --Leg.EndActivity, 
              leg.creationdate, 
              B.activitynr, 
              B."activitykind", 
              B."address", 
              B."date", 
              B."datetill", 
              B."time", 
              B."timetill", 
              B."resourcecombination", 
              B."tripid", 
              B."startdate", 
              B."starttime", 
              B."enddate", 
              B."endtime", 
              B."activityduration", 
              E."activitykind", 
              E."address", 
              E."date", 
              E."datetill", 
              E."time", 
              E."timetill", 
              E."resourcecombination", 
              E."tripid", 
              E."startdate", 
              E."starttime", 
              E."enddate", 
              E."endtime", 
              E."activityduration", 
              E.activitynr 

     

    Or, post some sample data and tell us what you are trying to do and we can start from scratch with Power Query. 

    You should rarely/never use that advanced SQL dialog box. Either create a View on the SQL server, or do it in Power Query. Doing it via an advanced SQL statement can cause permissions issue and prevents simple edits. I could adjust an M statement to only get the last 4 days of info in 2-3 lines that would fold back to the server. Nothing after an advanced SQL statement entry will allow folding.

3 Replies

  • Greg_Deckler's avatar
    Greg_Deckler
    Community Champion

    Anonymous - It looks like you are using a SQL Statement in your query. This is done in the Advanced portion of the Source step. I think that you would want to paste this SQL code instead and modify it to only get the last 4 days.

    • Anonymous's avatar
      Anonymous
      Not applicable

      Greg_Deckler wrote:

      Anonymous - It looks like you are using a SQL Statement in your query. This is done in the Advanced portion of the Source step. I think that you would want to paste this SQL code instead and modify it to only get the last 4 days.


      This was done by a  previous engineer, ive been looking for a while now, is there a way to get this query up again? 

      • edhans's avatar
        edhans
        Community Champion

        Here is your actual SQL statement. You'll need to head over to a SQL forum to get someone to help. Someone here might be able to assist, but a SQL forum as a bunch of people that can assist.

        SELECT Leg."legnr", 
               Leg."salesorder", 
               Row_number() 
                 OVER ( 
                   partition BY Leg.salesorder 
                   ORDER BY legnr)                               AS SalesOrderLine, 
               Leg."legstatus", 
               Leg."planned", 
               Leg."salesinvoice", 
               Isnull(Leg.ediprovider, '987654321')              AS EDIProvider, 
               Leg."distance", 
               Leg."orderleg", 
               Leg."plangroup"                                   AS 'FK PlanGroup', 
               Leg."findate", 
               Leg."totalamount", 
               Leg."costs", 
               Leg."cmrreceived", 
               Leg.[emailsalesinvoice], 
               Leg.[reference], 
               Leg."creationuser", 
               Leg.[cargo], 
               --Leg.BeginActivity, 
               --leg.EndActivity, 
               Isnull(B."address", ( b.activitynr + 900000000 )) AS BeginAddress, 
               B."date"                                          AS BeginDate, 
               B."datetill"                                      AS BeginDateTill, 
               B."time"                                          AS BeginTime, 
               B."timetill"                                      AS BeginTimeTill, 
               B."resourcecombination"                           AS 
               BeginResourceCombination, 
               B."tripid"                                        AS BeginTripID, 
               B."startdate"                                     AS BeginStartDate, 
               B."starttime"                                     AS BeginStartTime, 
               B."enddate"                                       AS BeginEndDate, 
               B."endtime"                                       AS BeginEndTime, 
               B."activityduration"                              AS 
               BeginActivityDuration, 
               Isnull(E."address", ( e.activitynr + 800000000 )) AS EndAddress, 
               E."date"                                          AS EndDate, 
               E."datetill"                                      AS EndDateTill, 
               E."time"                                          AS EndTime, 
               E."timetill"                                      AS EndTimeTill, 
               E."resourcecombination"                           AS 
               EndResourceCombination, 
               E."tripid"                                        AS EndTripID, 
               E."startdate"                                     AS EndStartDate, 
               E."starttime"                                     AS EndStartTime, 
               E."enddate"                                       AS EndEndDate, 
               E."endtime"                                       AS EndEndTime, 
               E."activityduration"                              AS EndActivityDuration, 
               leg.creationdate, 
               Sum(Cargo."weight")                               AS Weight, 
               Sum(Cargo."loadingmeter")                         AS LoadingMeter, 
               Sum(Cargo."palletplaces")                         AS PalletPlaces, 
               CASE 
                 WHEN legnr IN (SELECT DISTINCT [leg] AS [pl_orders] 
                                FROM   so_cargo 
                                WHERE  unit = 78) THEN 'pl' 
                 ELSE NULL 
               END                                               AS [pl order] 
        -- unit 78 = pl 
        FROM   so_leg AS Leg 
               LEFT JOIN so_activity AS B 
                      ON leg.beginactivity = B.activitynr 
               LEFT JOIN so_activity AS E 
                      ON leg.endactivity = E.activitynr 
               LEFT JOIN so_cargo AS cargo 
                      ON leg.[legnr] = cargo.leg 
        WHERE  findate >= Dateadd(year, -3, CONVERT (DATE, Dateadd(year, Datediff(year, 
                                                                         0, 
                                                                         Getdate()) + 1, 
                                                                             0))) 
        GROUP  BY Leg."legnr", 
                  Leg."salesorder", 
                  Leg."legtype", 
                  Leg."legstatus", 
                  Leg."planned", 
                  Leg."salesinvoice", 
                  Leg.ediprovider, 
                  Leg."distance", 
                  Leg."orderleg", 
                  Leg."plangroup", 
                  Leg."findate", 
                  Leg."totalamount", 
                  Leg."costs", 
                  Leg."cmrreceived", 
                  Leg.[emailsalesinvoice], 
                  Leg.[reference], 
                  Leg."creationuser", 
                  Leg.cargo, 
                  --Leg.BeginActivity, 
                  --Leg.EndActivity, 
                  leg.creationdate, 
                  B.activitynr, 
                  B."activitykind", 
                  B."address", 
                  B."date", 
                  B."datetill", 
                  B."time", 
                  B."timetill", 
                  B."resourcecombination", 
                  B."tripid", 
                  B."startdate", 
                  B."starttime", 
                  B."enddate", 
                  B."endtime", 
                  B."activityduration", 
                  E."activitykind", 
                  E."address", 
                  E."date", 
                  E."datetill", 
                  E."time", 
                  E."timetill", 
                  E."resourcecombination", 
                  E."tripid", 
                  E."startdate", 
                  E."starttime", 
                  E."enddate", 
                  E."endtime", 
                  E."activityduration", 
                  E.activitynr 

         

        Or, post some sample data and tell us what you are trying to do and we can start from scratch with Power Query. 

        You should rarely/never use that advanced SQL dialog box. Either create a View on the SQL server, or do it in Power Query. Doing it via an advanced SQL statement can cause permissions issue and prevents simple edits. I could adjust an M statement to only get the last 4 days of info in 2-3 lines that would fold back to the server. Nothing after an advanced SQL statement entry will allow folding.