Forum Discussion

jaryszek's avatar
jaryszek
Super User
1 year ago
Solved

How to connect to git lfs file on github?

Hello, 

I am trying to get raw file from github using link like:

https://raw.githubusercontent.com/myCompany/cost-mgmt-repo/BranchName/CostManagement/Dim_ManagementGroups.csv


I am getting:

version https://git-lfs.github.com/spec/v1
oid sha256:3b70bcd441cd74d876c58f9aef3e25c8822cb8bfe5904ed3b6dcaa344445
size 44476

I have code like here:

        AddColumnWithTable = Table.AddColumn(ExpandedColumn, "ResultTable", each 
            let
                a = Csv.Document(
                    Web.Contents(paramBaseUrlRawGithub, [
                        RelativePath = [RelativePath],
                        Query = [token = [Token]]
                    ]),
                    [Delimiter = ",", QuoteStyle = QuoteStyle.Csv]
                ),
                b = Table.PromoteHeaders(a, [PromoteAllScalars=true])
            in
                b
            )   

 

this is paramBaseUrlRawGithub:
https://raw.githubusercontent.com/ 

How to get it?

Best,
Jacek 
 

  • Anonymous's avatar
    Anonymous
    1 year ago

    Hi jaryszek,

     

    Thank you for reaching out to the Microsoft Fabric Forum Community.

     

    If the issue still persists, I’d recommend raising a support ticket with Microsoft. The support team can look into the backend and provide more in-depth assistance tailored to your environment.


    https://learn.microsoft.com/en-us/power-bi/support/create-support-ticket


    If this solution helped, please consider marking the response as accepted and giving it a thumbs-up so others can benefit as well.

     

    Best regards,
    Prasanna Kumar

     

8 Replies

  • Anonymous's avatar
    Anonymous
    Not applicable

    Hi jaryszek,

     

    Thank you for reaching out to the Microsoft Fabric Forum Community.

     

    You're getting a Git LFS pointer file instead of actual CSV content because the file is stored using Git Large File Storage (LFS). raw.githubusercontent.com only returns the pointer file, not the actual data.

     

    Best fix: Remove the file from Git LFS and commit it as a regular file.

    Alternative: Host the CSV elsewhere (e.g. SharePoint, Azure Blob, or a direct-download server).

     

    If this solution helped, please consider marking the response as accepted and giving it a thumbs-up so others can benefit as well.

     

    Best regards,
    Prasanna Kumar

     

    • jaryszek's avatar
      jaryszek
      Super User

      thanks it helped if you want to change your whole model but still is not a great solution. 
      It has to be LFS file by design it is higher than 100 mb and I am looking to get this file using power query and provided sha.

      What about Custom Git Connector? 

      I have one, maybe here i could post the statement like here 

      https://gist.github.com/fkraeutli/66fa741d9a8c2a6a238a01d17ed0edc5

      I have Github connector like here:

      // GitHub Connector for Power BI (Standard GitHub OAuth)
      [Version = "1.0.0"]
      section GitHub;
      
      client_id = "minenumber";
      client_secret = "minenumber";
      
      [DataSource.Kind="GitHub", Publish="GitHub.Publish"]
      shared GitHub.Contents = (relativePath as text, optional folderPath as text, optional branchName as text) =>
          let
              baseUrl = "https://api.github.com/repos/Company/",
              refBranch = if branchName <> null then "?ref=" & branchName else "?ref=develop",
              url = relativePath & "/contents/" & folderPath & refBranch,
      
              source = Json.Document(Web.Contents(baseUrl, [
                  RelativePath = url,
                  Headers = [
                      #"Authorization" = "Bearer " & Extension.CurrentCredential()[access_token],
                      #"User-Agent" = "PowerQuery"
                  ]
              ]))
          in
              source;
      
      GitHub = [
          Authentication = [
              OAuth = [
                  StartLogin = (resourceUrl, state, display) =>
                      let
                          authorizeUrl = "https://github.com/login/oauth/authorize?" &
                              "client_id=" & client_id &
                              "&redirect_uri=https://oauth.powerbi.com/views/oauthredirect.html" &
                              "&scope=repo" &
                              "&state=" & state
                      in
                          [
                              LoginUri = authorizeUrl,
                              CallbackUri = "https://oauth.powerbi.com/views/oauthredirect.html",
                              WindowHeight = 720,
                              WindowWidth = 1024,
                              Context = null
                          ],
      
                  FinishLogin = (context, callbackUri, state) =>
                      let
                          parts = Uri.Parts(callbackUri)[Query],
                          code = parts[code],
                          access_token_response = Json.Document(Web.Contents("https://github.com/login/oauth/access_token", [
                              Content = Text.ToBinary("client_id=" & client_id & "&client_secret=" & client_secret & "&code=" & code),
                              Headers = [
                                  #"Content-Type" = "application/x-www-form-urlencoded",
                                  Accept = "application/json"
                              ]
                          ])),
                          access_token = access_token_response[access_token]
                      in
                          [
                              access_token = access_token
                          ],
      
                  Refresh = (resourceUrl, refresh_token) => error "GitHub does not support refresh tokens.",
                  TestConnection = (access_token_record) => {"https://api.github.com/user"},
                  AccessToken = (access_token_record) => "Bearer " & access_token_record[access_token]
              ]
          ]
      ];
      
      GitHub.Publish = [
          Beta = true,
          Category = "Other",
          ButtonText = { Extension.LoadString("ButtonTitle"), Extension.LoadString("ButtonHelp") },
          LearnMoreUrl = "https://powerbi.microsoft.com/",
          SourceImage = GitHub.Icons,
          SourceTypeImage = GitHub.Icons
      ];
      
      GitHub.Icons = [
          Icon16 = { Extension.Contents("GitHub16.png"), Extension.Contents("GitHub20.png"), Extension.Contents("GitHub24.png"), Extension.Contents("GitHub32.png") },
          Icon32 = { Extension.Contents("GitHub32.png"), Extension.Contents("GitHub40.png"), Extension.Contents("GitHub48.png"), Extension.Contents("GitHub64.png") }
      ];


      Still need help on this!
      Jacek

      • jaryszek's avatar
        jaryszek
        Super User

        Ok i tried using this tip:
        https://learn.microsoft.com/en-us/power-query/handling-resource-path

        my code changed to implement it :

        shared GitHub.Contents = Value.ReplaceType(GitHubContentsImpl,
        type function (
        repo as (text meta [DataSource.Path = false]),
        filePath as (text meta [DataSource.Path = false]),
        optional branch as nullable text
        ) as any
        );

        but it didnt work. 

        It is still asking twice about credentials for each repository. It is possible to turn this off?

        Best,
        Jacek




  • ok so i managed to add new funtion (it has to have the same parameters like in the first one - this is a huge Microsoft bug like always!!!: https://www.youtube.com/watch?v=U9lqXCiGa08 ) 

    [DataSource.Kind="GitHub", Publish="GitHub.Publish"]
    shared GitHub.LfsDownloadUrl = (relativePath as text, optional folderPath as text, optional branchName as text) =>
        let
            sizeNumber = Number.FromText(folderPath),
            body = Json.FromValue([
                operation = "download",
                transfer = {"basic"},
                objects = {
                    [ oid = relativePath, size = sizeNumber ]
                }
            ]),
            UrlLfs = "MyCompany" & "/" & branchName & ".git/info/lfs/objects/batch",
            result = Json.Document(Web.Contents("https://github.com", [
                RelativePath = UrlLfs,
                Content = body,
                Headers = [
                    #"Authorization" = "Bearer " & Extension.CurrentCredential()[access_token],
                    #"Content-Type" = "application/vnd.git-lfs+json",
                    Accept = "application/vnd.git-lfs+json",
                    #"User-Agent" = "PowerQuery"
                ]
            ])),
            url = result[objects]{0}[actions][download][href]
        in
            url;

     

    and this work, i got fully downaloded link in power querY:


    The issue is that i need to add credentials for this file always...why this is not using my GitHub credentials? 

    Best,
    Jacek

     

  • Anonymous's avatar
    Anonymous
    Not applicable

    Hi jaryszek,

     

    Thank you for reaching out to the Microsoft Fabric Forum Community.

     

    If the issue still persists, I’d recommend raising a support ticket with Microsoft. The support team can look into the backend and provide more in-depth assistance tailored to your environment.


    https://learn.microsoft.com/en-us/power-bi/support/create-support-ticket


    If this solution helped, please consider marking the response as accepted and giving it a thumbs-up so others can benefit as well.

     

    Best regards,
    Prasanna Kumar

     

  • Anonymous's avatar
    Anonymous
    Not applicable

    Hi jaryszek,

     

    Just checking, have you had a chance to open a support ticket, as suggested. If so, we'd love to hear the current status or any updates from that.

    If the issue was resolved through the support ticket, it would be great if you could share the solution here as well. It could really help other community members find answers more quickly.

     

    Warm regards,
    Prasanna Kumar

  • Anonymous's avatar
    Anonymous
    Not applicable

    Hi jaryszek ,

     

    Thank you for reaching out to the Microsoft Fabric Forum Community.

     

    As we haven’t heard back from you and there are no further queries at this time, we’ll proceed to close this thread for now.
    Should you need any additional assistance, please feel free to start a new thread we’re always happy to help.

     

    Best regards,
    Prasanna Kumar