FlowCompile: An Optimizing Compiler for Structured LLM Workflows

Problem
This paper addresses the optimization of structured LLM (Large Language Model) workflows, which involve specialized sub-agents executing tasks based on a predefined graph. The challenge lies in selecting configurations for each sub-agent to balance accuracy and latency, given the combinatorial design space of model choices, reasoning budgets, and workflow structures. Existing methods primarily treat this optimization as a routing problem, focusing on real-time configuration selection based on accuracy-latency objectives established during training. The authors propose a novel approach that leverages compilation techniques to optimize workflows before deployment, a gap in the current literature that has not been thoroughly explored. This work is presented as a preprint and has not yet undergone peer review.

Method
The core technical contribution is FlowCompile, a structured LLM workflow compiler that performs compile-time design space exploration. FlowCompile decomposes workflows into sub-agents and profiles each under various configurations to gather performance metrics. It employs a structure-aware proxy to estimate the overall accuracy and latency of the workflow based on these measurements. The optimization process identifies a diverse set of high-quality configurations in a single compile-time pass, eliminating the need for retraining or online adaptation. The authors do not disclose specific details regarding the architecture of the sub-agents or the exact nature of the profiling metrics, but they emphasize the efficiency of their compile-time approach compared to traditional methods.

Results
FlowCompile demonstrates significant performance improvements over existing heuristic and routing-based optimization methods. In experiments across various workflows and benchmarks, it achieves up to a 6.4x speedup compared to baseline configurations. The results indicate that the compiled configuration set not only enhances runtime efficiency but also serves as a reusable optimization artifact, allowing for flexible deployment under different runtime preferences. The paper provides quantitative comparisons against named baselines, although specific benchmark names and detailed performance metrics are not included in the summary.

Limitations
The authors acknowledge that their approach may not generalize to all types of workflows, particularly those with highly dynamic or unpredictable characteristics. They also note that the profiling process may introduce overhead, which could offset some of the performance gains in certain scenarios. Additionally, the reliance on compile-time optimization may limit adaptability to real-time changes in workload or user requirements, a consideration not explicitly addressed in the paper.

Why it matters
The implications of FlowCompile are significant for the deployment of structured LLM workflows in production environments. By providing a systematic method for optimizing workflows at compile time, this work enables the creation of reusable configurations that can adapt to varying accuracy-latency trade-offs. This approach not only enhances the efficiency of LLM applications but also lays the groundwork for future research into compiler-based optimizations in machine learning, potentially influencing the design of more sophisticated LLM systems and workflows.

Authors: Junyan Li, Zhang-Wei Hong, Maohao Shen, Yang Zhang, Chuang Gan
Source: arXiv:2605.13647
URL: https://arxiv.org/abs/2605.13647v1