Runtime2.0 - appendOnly tableFeature + Copy Data activity incompatible

Hi RLWBG​,

Your reading is mostly right. One clarification first: seeing appendOnly in tableFeatures does not necessarily mean the table is append-only. With writer version 7, Delta lists the legacy features explicitly, so appendOnly can appear in the list while the property itself is off. What matters for tools like the Copy activity is whether their Delta writer supports the protocol version and every feature in that list (deletion vectors are the usual blocker). You can check what you actually have:

DESCRIBE DETAIL my_table;          -- minReaderVersion, minWriterVersion, tableFeatures
SHOW TBLPROPERTIES my_table;       -- delta.appendOnly should be false (or absent)

Practical options, from least to most change:

# Option 1: create tables the Copy activity can write to (per table, in your DDL)
spark.sql("""
CREATE TABLE my_table (id INT, name STRING)
USING DELTA
TBLPROPERTIES (
  'delta.minReaderVersion' = '1',
  'delta.minWriterVersion' = '2',
  'delta.enableDeletionVectors' = 'false'
)
""")

# Option 2: set the defaults for the session/environment instead of every DDL
spark.conf.set("spark.databricks.delta.properties.defaults.enableDeletionVectors", "false")
spark.conf.set("spark.databricks.delta.properties.defaults.minReaderVersion", "1")
spark.conf.set("spark.databricks.delta.properties.defaults.minWriterVersion", "2")

Option 3, which I usually prefer: keep the newer protocol and do the upsert in the notebook with MERGE, and use the Copy activity only to land raw data in a staging table (append/overwrite). That way you keep deletion vectors and other Runtime 2.0 benefits, and you are not tied to the Copy activity's writer support.

On the Lookup permission error that disappears after a few minutes: that pattern matches the SQL analytics endpoint metadata sync not having caught up with a brand-new table yet, rather than a real permission issue. Creating tables once up front (not on every run), or adding a short retry on the Lookup (Retry = 2-3, interval 60s) after the refresh call, usually removes it.

So my overall approach would be:

DESCRIBE DETAIL to see protocol + features
-> Copy upsert target: create with writer v2 / no deletion vectors
   (per DDL or session defaults)
-> Better: Copy -> staging, MERGE in notebook
-> Lookup: create tables once + retry after SQL endpoint refresh

AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.