Write sql query when source is a lakehouse table in dataflow gen2

Hi pbi_taken ,

You can write sql queries to filter the tables in lakehouse.
First you need to create a Dataflow Gen2 then select lakehouse as the destination, publish it and find that lakehouse then create a new notebook and write sql query to filter the tables. If you want to save the filtered results you need to use python query.

%%sql
SELECT *
FROM lakehouse_table
WHERE column_name = 'desired_value'

%%pyspark
# Read the original table
df = spark.read.table("original_table_name")

# Apply the filter
filtered_df = df.filter(df["column_name"] == "desired_value")

# Save the filtered data as a new table
filtered_df.write.mode("overwrite").saveAsTable("original_table_name")

Best Regards,

Neeko Tang

If this post  helps, then please consider Accept it as the solution  to help the other members find it more quickly.