About Pipeline Query

Hi kye1, 

THe Lookup activity does not support queries for lakehouses, only Tables and Files. 

 I recommend you vote for this idea: Query a lakehouse using LookUP Activity - Microsoft Fabric Community

If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.

You can use Azure SQL Db connector and connect to the lakehouse via SQL anlaytics endpoint and then use look up activity and query 

Hi kye1 ,

There is an option in Copy activity , are you looking for something like this?

Regards,

Srisakthi

Thanks tayloramy  and NandanHegde. You guys do help me a lot

Hi kye1 ,

Thank you for reaching out to the Microsoft Fabric Community Forum.

Could you please let us know if the issue has been resolved? I wanted to check if you had the opportunity to review the information provided by  NandanHegde and tayloramy  . If you still require support, please let us know, we are happy to assist you.

Thank you.

Hi kye1 ,

We haven’t received an update from you in some time. Could you please let us know if the issue has been resolved?
If you still require support, please let us know, we are happy to assist you.

Thank you.

In Fabric Data Pipeline → Lookup activity:

When you connect Lakehouse and select Table

Fabric does NOT show a “Query” option

👉 Because Lookup with Lakehouse only supports table-level access, not custom SQL queries.

Important Limitation (Key Point)

🔴 Lookup activity in Fabric does NOT support SQL queries on Lakehouse tables

That’s why:

You only see Table

No Query textbox appears

You cannot write SELECT MAX(watermark) directly

---

How to Handle Watermark for Incremental Load (Correct Ways)

✅ Option 1: Use Lakehouse Table with Single Row

Best & simplest approach.

Design your watermark table like this:

watermark_value

2024-12-01 00:00:00

Then in Lookup:

Source: Lakehouse

Table: watermark_table

First row only: ✅ enabled

You can access it using:

@activity('Lookup_Watermark').output.firstRow.watermark_value

✔ No query needed

✔ Fully supported

---

✅ Option 2: Use Notebook activity (Recommended for flexibility)

If you need:

MAX()

filtering

complex logic

Then:

1. Use Notebook activity

2. Run Spark SQL:

SELECT MAX(watermark_value) AS watermark FROM watermark_table

3. Return the value

4. Use it in the pipeline via notebook output

✔ Most flexible

✔ Best practice for complex logic

---

✅ Option 3: Use Warehouse instead of Lakehouse

If your watermark table is in a Fabric Warehouse:

Lookup → Source = Warehouse

You will see “Query”

You can write SQL normally

---

Summary (Quick Answer)

Scenario Query Visible?

Lakehouse + Lookup ❌ No

Warehouse + Lookup ✅ Yes

Notebook + Lakehouse ✅ Yes

---

Recommended Architecture for Incremental Load

✔ Watermark table = 1 row only

✔ Use Lookup (table) OR Notebook

❌ Do not expect SQL Query option in Lakehouse Lookup

How you want to update watermark

I’ll design the exact incremental pipeline flow for you step by step 🚀