Forum Discussion
DataflowGen2 on MS Fabric: Optimize column profiling on millions of rows (1-5mil)
Hello, I have datasets residing on hive_metastore of Azure Databricks storage, in the range of 1-5 millions of records. I wish to use the Column Profiling feature on the entire dataset after load...
This is something that we don't have full control over. It relies on 3 specific components:
- Data source: and the resources available for it
- Connector: owned by the Databricks team
- Power Query editor: it effectively runs the queries that the connector tells it to run and it tries to later cache the results if they're fully computed
I can pass your feedback to the Databricks team, but there's nothing beyond what you're doing today that can impact the performance of the profiling for such scenario.
miguel
Community Admin
Community AdminThis is something that we don't have full control over. It relies on 3 specific components:
- Data source: and the resources available for it
- Connector: owned by the Databricks team
- Power Query editor: it effectively runs the queries that the connector tells it to run and it tries to later cache the results if they're fully computed
I can pass your feedback to the Databricks team, but there's nothing beyond what you're doing today that can impact the performance of the profiling for such scenario.