Dataframes affected by changes to their source!?

[EricCicconetti]

Looking for someone who knows more about dataframes to confirm what exactly they are in terms of a Fabric notebook. They look and feel like a temp table that you can build and manipulate but I've recently discovered that must not be the case. I have a notebook that was going to update two tables, the primary target table Warehouse and a secondary logging table called WarehouseSource. Both tables are being updated based on changes that have happened between Warehouse and its original source so I have dataframes that start off by looking at that table, but from there I built dataframes independently that would be used in a merge statement with each table. In my final merge I was altering Warehouse first and then WarehouseSource and was running into my WarehouseSource changes not happening and discovered that after the Warehouse merge happened my dataframe meant to update WarehouseSource was suddenly empty. After some back and forth with ChatGPT it was saying dataframes in Fabric aren't actually data themselves but a query template for the data, so by changing the original source I broke my WarehouseSource dataframe. Ultimately it was an easy fix, I merge WarehouseSource first and then Warehouse and it works how I want, but it was a jarring discovery to realize that dataframes are not what I thought they were.

[arabalca · Accepted solution]

Hola @EricCicconetti ,

Los DataFrames no son tablas temporales ni copias de datos, sino una definición lógica de cómo recuperarlos.
Esto significa que se ejecutan en tiempo de ejecución y siempre reflejan el estado actual de la fuente. Por eso, si modificas la tabla base antes de usar otro DataFrame dependiente, el resultado puede cambiar o incluso quedar vacío, exactamente lo que experimentaste.
Se utilizan principalmente como motor de transformación de datos para:

• filtrar, unir y agregar datos;
• crear canalizaciones ETL/ELT;
• preparar datos para análisis.

Dentro de la arquitectura Medallion:

• Bronce: ingesta de datos sin procesar
• Plata: transformación y limpieza;
• Oro: conjuntos de datos listos para el negocio.

Si no estás trabajando con este enfoque, vale la pena revisarlo, ya que ayuda a estructurar el uso de DataFrames correctamente y evitar situaciones como esta.

Si mi comentario te ayudó a resolver tu pregunta, te agradecería mucho que lo marcaras como la solución aceptada. Esto ayuda a otros con el mismo problema y también me motiva a seguir contribuyendo. Muchas gracias. Lo aprecio mucho.