Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
4 years ago
Solved

Query Folding with Databricks.Catalogs Connector

Hi all,   I'm using Databricks.Catalogs connector to get data from Databricks Database Tables in the import mode (using the online Dataflows).  The problem is that my query is not folded even thou...
Need Help
  • Anonymous's avatar
    Anonymous
    3 years ago

    I found a great solution and does not break Databricks Query Folding 

     

    use Databricks.Query instead of Databricks.Catalogs

     

    If you want to lightweight your loading process

     

    create a parameter  like date_filter

     

    let
    Source = Databricks.Query("warehouses" , [] ),
    part1 = "select * from TYPE YOUR TABLE",
    part2 = " WHERE snapshot_date > "&"'"&date_filter&"'",
    MyQuery = Source(part1 & part2)
in
    MyQuery

     

v-jingzhang's avatar
v-jingzhang
Community Support
4 years ago

Hi Anonymous 

 

Can you show the M code of this query? You can open Advanced Editor and find all code there. Or you can use Query folding indicators to check which steps are preventing your query from folding. Reorder them if necessary to increase folding.

 

Best Regards,
Community Support Team _ Jing

Anonymous's avatar
Anonymous
Not applicable
4 years ago

Hello Jing,

 

thank you for your message.

 

  • the M code is:
let
Source = Databricks.Catalogs(#"Server Hostname", #"HTTP Path"),
  Navigation = Source{[Name = #"Database Name", Kind = "Database"]}[Data],
  #"Navigation 1" = Navigation{[Name = #"Schema Name", Kind = "Schema"]}[Data],
  #"Navigation 2" = #"Navigation 1"{[Name = #"Table Name", Kind = "Table"]}[Data]
in
  #"Navigation 2"
 
Unfortunately, I'm strungling to understand which step migh prevent the query folding, as there is no other data transformation besides the unpacking of the content of the [Data] columns into the final table.
 
 
Kind regards,
Boris.