Forum Discussion
Using GUIDs
There is no record-count limit where a GUID suddenly stops being stable. What likely changed is execution behaviour, not data size. In distributed engines such as Spark (including environments like Databricks/Microsoft Fabric), functions that generate GUIDs or UUIDs are non-deterministic. If the dataframe is recomputed (due to cache eviction, lineage re-evaluation, optimisation changes, cluster restart, or engine version update), the GUID column is recalculated and new values appear. That explains why it worked earlier and then changed after a platform/runtime update or caching behaviour shift in november. The fix is to materialise immediately after generation (write to storage or checkpoint) instead of relying on cache to freeze non-deterministic columns.
Super UserThere is no record-count limit where a GUID suddenly stops being stable. What likely changed is execution behaviour, not data size. In distributed engines such as Spark (including environments like Databricks/Microsoft Fabric), functions that generate GUIDs or UUIDs are non-deterministic. If the dataframe is recomputed (due to cache eviction, lineage re-evaluation, optimisation changes, cluster restart, or engine version update), the GUID column is recalculated and new values appear. That explains why it worked earlier and then changed after a platform/runtime update or caching behaviour shift in november. The fix is to materialise immediately after generation (write to storage or checkpoint) instead of relying on cache to freeze non-deterministic columns.