GEPA and SkillOpt Optimization: When Reflective Prompting and Skill Optimization Help on Locally Served Reasoning Models

Executive Summary & Core Discovery

Reflective prompt and skill optimization techniques like GEPA and SkillOpt have shown significant gains on frontier and mid-size models. However, their effectiveness on locally served reasoning models remains an open question. This study evaluates these techniques across two reasoning-native target-serving configurations: DeepSeek V4 Flash through DwarfStar and Qwen3.8-27B through LM Studio. The research reveals that optimization effects are task- and method-dependent, with validation gains potentially reversing on held-out evaluation. These findings suggest that textual headroom for a model-task pair and the observed effect of a text-space optimizer are more informative when decomposed into content/reasoning, answer-realization/evaluator alignment, and serving-completion/reliability.

Architectural Background & Theoretical Foundations

The study focuses on two reflective prompt and skill optimization techniques: GEPA and SkillOpt. GEPA is a reflective prompt optimization technique that involves iterative refinement of prompts based on model feedback. SkillOpt, on the other hand, is a skill optimization technique that involves iterative refinement of model skills based on task performance. These techniques have shown significant gains on frontier and mid-size models, but their effectiveness on locally served reasoning models has not been thoroughly evaluated.

The study evaluates these techniques across two reasoning-native target-serving configurations: DeepSeek V4 Flash through DwarfStar and Qwen3.8-27B through LM Studio. DeepSeek V4 Flash is a 27B parameter model that has been optimized for reasoning tasks, while Qwen3.8-27B is a 27B parameter model that has been optimized for a wide range of tasks. Both models are served locally, which allows for more control over the optimization process and evaluation of the techniques.

Benchmark Architecture & Evaluation Methodology

The study evaluates the same held-out tasks and four conditions – minimal baseline, GEPA prompt, SkillOpt skill, and manually engineered prompt – on HotpotQA (n=500 per condition) and MATH-500 (n=250). HotpotQA is a question-answering dataset that requires multi-hop reasoning, while MATH-500 is a math problem-solving dataset that requires complex reasoning. The study uses exact match as the primary evaluation metric, which measures the percentage of tasks where the model’s answer exactly matches the ground truth.

The study finds that GEPA shows significant gains on MATH-500, raising exact match from 0.828 to 0.904 for Qwen3.8-27B and from 0.852 to 0.908 for DeepSeek V4 Flash. However, GEPA’s validation gain reverses into held-out degradation on HotpotQA. SkillOpt shows no significant gains on either dataset. These findings suggest that optimization effects are task-dependent and that validation gains may reverse on held-out evaluation.

Empirical Findings & Granular Performance Analysis

The study reveals that GEPA shows significant gains on MATH-500, raising exact match from 0.828 to 0.904 for Qwen3.8-27B and from 0.852 to 0.908 for DeepSeek V4 Flash. However, GEPA’s validation gain reverses into held-out degradation on HotpotQA. SkillOpt shows no significant gains on either dataset. These findings suggest that optimization effects are task-dependent and that validation gains may reverse on held-out evaluation.

The study also finds that GEPA increases token consumption but reduces latency on MATH-500. This suggests that lower response time does not necessarily imply lower token cost. These findings have important implications for the deployment of reflective prompt and skill optimization techniques in locally served reasoning models.

In-Depth Analysis of Core Limitations & Vulnerabilities

The study identifies four core limitations of reflective prompt and skill optimization techniques like GEPA and SkillOpt on locally served reasoning models.

1. Task-Dependent Gains

Reflective prompt and skill optimization techniques like GEPA and SkillOpt show significant gains on some tasks but not others. The study finds that GEPA shows significant gains on MATH-500 but not on HotpotQA. This suggests that optimization effects are task-dependent and that validation gains may reverse on held-out evaluation.

2. Validation Challenges

The study finds that validation gains may reverse on held-out evaluation. GEPA’s validation gain on HotpotQA reverses into held-out degradation. This suggests that validation gains may not be indicative of held-out performance and that careful evaluation is necessary.

3. Token Consumption

The study finds that GEPA increases token consumption on MATH-500. This suggests that optimization techniques may have unintended consequences and that careful evaluation of token consumption is necessary.

4. Serving Sensitivity

The study finds that optimization effects may be sensitive to serving configurations. GEPA shows different effects on DeepSeek V4 Flash and Qwen3.8-27B. This suggests that optimization effects may be sensitive to serving configurations and that careful evaluation is necessary.

Enterprise Production & Red-Teaming Blueprint

The findings of this study have important implications for the deployment of reflective prompt and skill optimization techniques in locally served reasoning models. Organizations should carefully evaluate the task-dependency of optimization effects and the potential for validation gains to reverse on held-out evaluation. They should also carefully evaluate token consumption and serving sensitivity. Additionally, organizations should consider using JudgeMyAI’s automated evaluation layer to ensure the robustness and reliability of their locally served reasoning models.

Technical FAQ (Granular & Non-Repetitive)

Reflective prompt and skill optimization techniques like GEPA and SkillOpt show task-dependent performance gains on locally served reasoning models, with validation gains potentially reversing on held-out evaluations. Optimization effects are more informative when decomposed into content/reasoning, answer-realization/evaluator alignment, and serving-completion/reliability.