Classifier-Augmented Generation for Structured Workflow Prediction

AI summary

Overview

Research area: Natural language processing applied to structured workflow authoring — specifically translating natural-language descriptions into executable ETL (Extract, Transform, Load) pipelines for IBM DataStage.

Technical level: Advanced. The paper assumes familiarity with LLM prompting strategies (few-shot prompting, ReAct-style agents, tool calling), multi-label classification, and ETL tooling.

One-sentence scope: The paper introduces Classifier-Augmented Generation (CAG), a modular pipeline that decomposes a user utterance, retrieves candidate workflow stages via a small classifier plus keyword matching, and then uses an LLM to predict stages, edges, and stage properties — evaluated against single-prompt and agentic baselines on 1010 utterances, 54 non-linear flows, and 308 flows with 1410 properties.

What This Paper Is About

Graphical ETL tools such as IBM DataStage let users assemble data workflows visually, but configuring stages and their dozens of properties is slow and demands deep tool knowledge. This paper builds a system that takes a plain natural-language description and produces a complete, executable workflow — predicting which stages are needed, how they connect, and how each stage is configured. The goal is to reduce manual authoring effort for both novice and expert users while keeping the pipeline interpretable and validatable.

Key Contributions

1. An end-to-end system for natural-language-driven ETL authoring that generates the full workflow — stage selection, non-linear edge layout, and stage property configuration — rather than only a linear skeleton. The authors state this is the first system with a detailed evaluation across stage prediction, edge layout, and property generation for this task.

2. The Classifier-Augmented Generation (CAG) architecture, which combines utterance decomposition, a small classifier used as a candidate-retrieval tool, keyword/synonym matching over the utterance, and stage-specific few-shot prompting to produce the final multi-label stage prediction.

3. A detailed component-wise evaluation comparing CAG against a strong single-prompt baseline and an agentic (ReAct-style) baseline on stage prediction, reporting accuracy overall, on single-stage utterances, and on multi-stage utterances, plus separate evaluations for edge prediction and property prediction.

4. Practical grounding in a production tool, including a multi-dimensional validation strategy (property name checks, type coercion, inter-property dependency expressions, and external consistency checks) and deployment inside IBM DataStage.

Main Findings

• CAG outperforms both baselines on stage prediction. Total accuracy on 1010 flows: llama-3.3-70b-instruct reached 97.2% (98.6% on single-stage utterances, 94.2% on multi-stage utterances) and llama-4-maverick-17b-128e reached 97.7% (99.0% / 94.8%). The single-prompt baseline reached 96.4% (llama-3.3-70b) and the agentic baseline only 69.3% (llama-3.3-70b).

• CAG is cheaper as well as more accurate. It uses 4,000–4,700 tokens per request on average depending on the model, a reduction of approximately 66% versus the single-prompt baseline's roughly 14,000 tokens per request. The conclusions state more than 60% token reduction, and roughly 90% lower overall cost when deployed with a smaller model.

• The agentic approach underperformed. Reasons identified: granularity mismatch between generated sub-utterances and the functional scope of DataStage stages, semantic similarity between stages such as split_subrecord and split_vector, and occasional classification failures. The authors also report that attempts to fix this by supplying decomposition examples, stage descriptions, or a candidate-suggestion function were misinterpreted by the LLMs and reduced accuracy further.

• The classifier itself is highly accurate on single-label data. The classification tool was trained on 2,697 (utterance, operator) single-label pairs covering 138 semantic labels (derived by merging close variants among the 142 total stages), split into 2,133 training and 564 test samples, yielding macro-Precision 98.6%, macro-Recall 97.8%, and macro-F1 98.1%. Both RoBERTa-large and IBM's slate-125m-english-rtrvr reached about 98% accuracy, with RoBERTa marginally better but slower; slate was chosen for speed.

• Edge prediction is the weakest link. On 54 non-linear flows with 6–14 stages (average 8.1), the best model (llama-3.3-70b-instruct) achieved 73% structural similarity and 37% exact match. Other models: granite 41%/4%, llama-4-maverick 42%/15%, llama-3.2-3b 31%/0%.

• Property prediction is strong, partly because of the sub-utterance decomposition. Results across 308 flows with 1410 properties: llama-3.3-70b-instruct precision 92%, recall 81%, F1 0.86; granite 93%/71%/0.81; llama-3.2-3b 88%/72%/0.79; llama-4-maverick 94%/67%/0.78. The conclusions summarize property prediction as achieving 90% accuracy, and full-flow generation as reaching over 70% structural similarity.

• Utterance segmentation for edges is near-perfect. Once the task list and stage descriptions are known, the LLM segments the utterance into sub-utterances with over 99% accuracy.

• Validation is multi-layered. Generated properties are dropped if the name is not valid for the stage, if the value cannot be coerced to the declared type, if inter-property dependency metadata is violated (for example, in the Column Generator stage, Options/Column to Generate is only valid when Options/Column Method is set to Explicit), or if external consistency checks such as matching real connection names fail.

• A stated inconsistency in the paper's tables: Tables 1–3 report a "granite-3.1-8b-instruct" model while Tables 4 and 5 report "granite-3.3-8b-instruct."

Methodology in Plain English

The researchers first extracted 142 DataStage stages (90 of them datasource connectors) along with their descriptions, properties, types, default values, and availability conditions. Each stage has between 1 and 111 properties, averaging 27.6.

They then compared three ways of predicting which stages a workflow needs:

1. Single prompt: put all 142 stages, one-line descriptions, and 142 few-shot examples into one prompt — about 14,000 tokens per request.
2. Agentic: let the LLM act as a ReAct-style agent that decomposes the utterance itself and calls a stage classification tool. The tool is a small classifier trained on single-label (utterance, stage) pairs.
3. CAG: separate those roles. The LLM splits the utterance into single-stage sub-utterances in a dedicated step with its own few-shot examples; the classifier labels each sub-utterance; a keyword/synonym scan over the original utterance adds missed candidates. The resulting candidate set is passed back to the LLM with one-line descriptions and about 40 few-shot examples that mention at least one candidate stage, so the LLM makes the final multi-label decision over a much smaller space.

Once stages are predicted, repeated stages get unique names with a suffix, the utterance is re-segmented against the known task list, and the LLM predicts the directed edges. A validation pass checks each node's edge count against DataStage input/output cardinality constraints, splitting overloaded nodes where unambiguous or removing excess edges otherwise. In parallel, properties are predicted per stage using only that stage's sub-utterance, its name, a list of supported properties with one-line descriptions, and a one-shot example, followed by the validation layers described above.

Why This Matters

Impact on research. The paper shows that a small, fast classifier used for candidate retrieval can beat both a giant all-in-one prompt and an autonomous agent on a structured prediction task, while costing far fewer tokens. That is a useful counterpoint to the assumption that agentic LLM loops are the natural solution for tool-augmented generation, and it offers a concrete failure analysis for why the agentic variant struggled (granularity mismatch, semantic similarity between stages, prompt misinterpretation).

Real-world applications:

• ETL and data integration authoring. Data engineers describing a pipeline in prose and getting a draft DataStage job, including stage properties, that they review rather than build.
• Onboarding and accessibility. Novice users who do not know which stage or which of dozens of properties to pick benefit from a generated starting point.
• Expert productivity. Experts working with repeated pipeline patterns get auto-filled configurations and only need a lightweight review.
• Other structured automation domains. Any platform with operator-level metadata (transformation descriptions and property schemas) could adopt the same decompose-classify-prompt-validate pattern.

Industry relevance. The system is already integrated into IBM DataStage, a production ETL tool, which the authors cite as evidence of practicality and scalability. The reported economics — a smaller model at roughly 90% lower overall cost with comparable accuracy — speak directly to deployment decisions.

Future Directions

1. Improving edge prediction. Exact match of 37% and structural similarity of 73% leave drafts that still need manual correction. The authors propose hybrid architectures that combine LLM semantic reasoning with geometric deep learning methods such as graph neural networks (GNNs) for flow topology.

2. Moving beyond single-label classifier training. The classifier was trained only on single-label (utterance, stage) pairs because sufficiently large multi-label training data was unavailable, so relevant candidate stages can be excluded entirely and errors propagate; a real multi-label classifier is an open problem.

3. Fuzzy matching and correction of schema elements. The current validation logic assumes table and column names are either correct or ignorable; misspelled or unavailable schema references are not corrected.

4. Portability and generalization. Prompt formats and examples are tuned per model family, which may limit cross-architecture or cross-provider portability, and the evaluation inputs reflect internally sourced usage data, so adaptation to other domains or user populations will likely need further refinement.

Target Audience

Researchers and practitioners working on LLM-assisted code or configuration generation, especially those building natural-language interfaces to structured tools; applied scientists at data-integration and ETL vendors; and NLP researchers interested in comparing classifier-augmented retrieval, single-prompt baselines, and agentic tool-use pipelines on a real structured prediction task. The paper is most valuable to readers who already understand prompting and classification pipelines and want a careful, component-level account of what works and where it breaks.

Authors’ abstract

ETL (Extract, Transform, Load) tools such as IBM DataStage allow users to visually assemble complex data workflows, but configuring stages and their properties remains time consuming and requires deep tool knowledge. We propose a system that translates natural language descriptions into executable workflows, automatically predicting both the structure and detailed configuration of the flow. At its core lies a Classifier-Augmented Generation (CAG) approach that combines utterance decomposition with a classifier and stage-specific few-shot prompting to produce accurate stage predictions. These stages are then connected into non-linear workflows using edge prediction, and stage properties are inferred from sub-utterance context. We compare CAG against strong single-prompt and agentic baselines, showing improved accuracy and efficiency, while substantially reducing token usage. Our architecture is modular, interpretable, and capable of end-to-end workflow generation, including robust validation steps. To our knowledge, this is the first system with a detailed evaluation across stage prediction, edge layout, and property generation for natural-language-driven ETL authoring.