IFCO Reduces dbt Job Runtime by Over 60% with Databricks Optimization

IFCO's data team achieved a core job runtime reduction of over 60% by optimizing their dbt workflows on Databricks. The company, which employs over 2,000 people globally and serves more than 50 countries, operates in Pullach and Munich, Germany, where it has over 350 employees.

Before optimization, IFCO's daily runtime for processing data was approximately 7 hours, which has now been reduced to 2 hours and 20 minutes. This change resulted in a 58% reduction in daily compute costs. The data team processes billions of raw tracking events daily, scanning around 25 billion rows per run and recomputing only 3-5% of their asset pool.

Key to this optimization was the retirement of nightly full refreshes and the introduction of a semantic layer, which allowed for more efficient data handling. The IFCO Data Team noted that the largest gains came not from increasing cluster size but from minimizing the amount of work done—specifically, by touching fewer rows and rebuilding tables less frequently.

The team emphasized the challenges of turning billions of raw tracking events into reliable KPIs, citing issues with data volume, late arrivals, and the need to correct historical reports. Their goal is to provide accurate estimates quickly and refine them as late data arrives, without the need for nightly reprocessing. This optimization follows previous enhancements by Databricks, including the introduction of the databricks-dbt-factory tool, which supports better integration of dbt projects within the Databricks environment.