Record all the applied steps in Dataflow/power query editor with Dates-Activity logs
Hi Team, I have a table in dataflow and applied some transformations on that . table name:Empdetails initial table is in below snap i have applied some transformations on that like below #"Renamed columns" = Table.RenameColumns(#"Navigation 2", {{"nam$e", "name"}, {"ag*e", "age"}, {"gen_der", "gender"}, {"sala@@ry", "salary"}, {"id_?", "id"}}), #"Uppercased text" = Table.TransformColumns(#"Renamed columns", {{"location", each Text.Upper(_), type nullable text}}), #"Split column by delimiter" = Table.SplitColumn(#"Uppercased text", "name", Splitter.SplitTextByDelimiter(" "), {"name.1", "name.2"}), #"Renamed columns 1" = Table.RenameColumns(#"Split column by delimiter", {{"name.1", "FirstName"}, {"name.2", "LastName"}}), #"Capitalized each word" = Table.TransformColumns(#"Renamed columns 1", {{"FirstName", each Text.Proper(_), type nullable text}}), #"Capitalized each word 1" = Table.TransformColumns(#"Capitalized each word", {{"LastName", each Text.Proper(_), type nullable text}}) in #"Capitalized each word 1" final Dataflow table is like below Applied transformations are: Transformations 1-Renaming column Transformations 2-Upper case Transformations 3-Split column Transformations 4-Renaming column Transformations 5- Capitalization I have no issue with code and transforamtions. so in adavnce query editor we will get all the M code, but here i need to record the activity log time and date Eg: Transformations 1-Renaming column 9/1/2023 18:05 Transformations 2-Upper case 9/1/2023 18:06 Transformations 3-Split column 9/1/2023 18:07 Transformations 4-Renaming column 9/1/2023 18:08 Transformations 5- Capitalization 9/1/2023 18:10 like this, if in future if i apply any change in any one of the transformations, then i need to capture that date and time and person name who changed that transformation, I know we dont have this option in Dataflow/PQE as of now, but i need to achieve this. after creating dataflow, i will call that dataflow into the pipeline and after that by using script activity we may achiieve that(i heard it some where), but i am not getting any idea to achieve this by using script activity can some body help me to get out of this. TIA
getting started with data factory
hi, i thought i'd start learning a bit about data factory by trying to do something i want instead of what some of the tutorials show. so shown below i believew i've used what i understand is a studio to create a factory. what i'd like to do now is to use some power query code that i used elsewhere as a source of data. and land that data on prem in sql. i suppose if i had to land it in the cloud that would be ok instead. but i dont see much in the way of flexibility here. how would i start? the power query has its origins in a python script that extracts stock values from yahoo finance. can data factory be used to do something like this without getting ssis involved?
Dataflow Sink is not working for Lakehouse/Warehouse
I am building a pipeline with Copy Activity, then Dataflow Activity. Source of Dataflow is Lakehouse Table which is selected fine. But When adding Destination as Lakehouse, it kept searching for connection details. Then I routed sink to Azure SQL DB which worked fine. By when we can expect Dataflow Sink working as Lakehouse? Thanks in advance
