Forum Discussion
Automating Data Wrangler output
- Anonymous2 years ago
Hi Cornelia,
If I understand you correctly, you'd like to describe a dataset automatically/programatically, the way data wrangler does in python notebooks. I don't know that there is an api to use the data wrangler programatically.
what I can suggest is either using the dataframe description capabilites in spark:
https://spark.apache.org/docs/3.1.1/api/python/reference/api/pyspark.sql.DataFrame.describe.html
or generate the description directly in the dataset (assuming we're talking about pbi datasets = semantic models) using MCode
https://learn.microsoft.com/en-us/powerquery-m/table-schema
https://learn.microsoft.com/en-us/powerquery-m/table-profile
hope this helps!
Thank you for your help! 😁
Even if Data Wrangler seems pretty good at generating python code for describing data, I ended up creating some custom pyspark functions - easier to modify and automate.