Forum Discussion

fredforest's avatar
fredforest
New Member
2 years ago
Solved

Data types in Delta

In the olden days of data warehousing, a developper/modeler would be careful to optimize data types (especially on large fact tables) for a variety of reasons. I'm wondering if there's still a reaso...
data engineering
data warehouse
  • AndyDDC's avatar
    AndyDDC
    2 years ago

    Well, I did offer some advice as to why we still want to optimise data types.  The engine will provide the resources necesarry to process data operations based on data types and precision.  The bigger the data type and precision, the bigger the resources are used to process.  If you keep your data type sizes to a minimum, this will help the engine only provision what it needs.  

     

    Also, if we define our tables with data type sizes in Delta, it will stick to the data type size (e.g. cannot declare a string as 100 and then insert 200 chars in)

AndyDDC's avatar
AndyDDC
Icon for Most Valuable Professional rankMost Valuable Professional
2 years ago

That's a great question fredforest personally I'm going to keep optimising data types even in this new world of delta and parquet. One of the best practices I worked with in synapse serverless (which the lakehouse endpoint and warehouse endpoint are built from) was optimising data types. Reason for this was that the data types of the columns being used in a workload would directly impact the amount of cluster resources was given to process the workload. The larger the data type, the more resources were provided. Also with joining data together across different delta tables, still can't beat integers.