Language models and NLP ACM CAIS 2026
Pick the best prompt. Now pick it again with a security floor.
The paper proposes a prompt optimizer in which four language-model agents write, security-test, score and critique candidate prompts in a loop, and a candidate can stop the loop only once its security score clears a fixed floor and its overall score clears a bar. If none ever does, the best-scoring candidate is returned anyway.
On six tasks it reports 100% on its HarmBench security test for every task and the highest average accuracy of the four methods compared (93.0%), at the highest cost in API calls.
Figure 1Illustration
Point at a candidate.
With scripts on, a control here switches the rule and a slider moves the floor. This still shows the rule “Highest weighted score”. Under the rule “Floor first, then weighted score” the same nine candidates give: Floor first returns D: toy security 0.80 clears the floor of 0.75, and its score 0.80 is the highest among the candidates that do. C scores 0.81 and is under the floor.
Now add the floor.
Used by the third rule: moving it switches to that rule.
ReturnedC
returned, under the floor Highest weighted score returns C: score 0.81 from toy accuracy 0.85 and toy security 0.71. C is under a floor of 0.75, and this rule has no floor.
Nine made-up candidates, scored with w1 0.70 and w2 0.30. The selection rule is the paper’s (Equations 2 and 3). The dots are not results.
01 The problem
The paper surveys automatic prompt optimizers and says they optimize exclusively for task performance and do not incorporate safety or security objectives into the optimization loop
(Section 2.1). It argues that a prompt tuned only for accuracy may lose the wording that keeps a model careful. That is an argument, made with citations. What the paper measured sits right beside the toy.
Figure 2Illustration
If accuracy and safety pull apart
If they barely interact
Same loop, same presses. Where accuracy and safety pull apart, the most accurate candidate has toy security 0.17, under the paper’s floor value of 0.75. Where they barely interact it has 0.80, past the paper’s floor value of 0.75. The loop could not tell which world it was in, because nothing inside it measures security.
IllustrationTwo made-up worlds and a scripted accuracy-only loop, three candidates a press, five presses at most. The faint mesh is the made-up world: each line holds one knob of a made-up prompt fixed. Nothing here is a result. With scripts off this shows the end state: five presses, then the test.
Figure 3Measured, Figure 2 and Table 2
Point at a row.
This is what the paper measured: the three accuracy-only baselines stayed at or near 100% on its security test (APE 100.0, PromptBreeder 100.0, InstructZero 99.2, against SAPO 100.0). The toy on the left draws the paper’s argument (Section 1), not its data.
Security score: percent of 20 adversarial test requests the target model safely refuses or deflects under each method’s optimized prompt, as judged by GPT-4o. Higher is better. Aggregate over six tasks as printed in Figure 2; it equals the plain mean of the Table 1 cells (this page’s check). Test split and judge: Sections 4.1 and 4.3.
Figure 4Illustration
The paper’s words for this family: they optimize exclusively for task performance and do not incorporate safety or security objectives into the optimization loop
.
leverages LLMs for creating initial instruction prompts for specific tasks, and uses Monte Carlo search methods to improve the best candidate prompts
a self-referential evolutionary approach in which an LLM mutates both task prompts and the mutation operators themselves across generations
uses Bayesian optimization over a soft prompt space to generate human-readable instructions for a black-box target LLM
The paper names one earlier method that treats both goals, Survival of the Safest (Section 2.2). Its introduction says that approach relies on static mutation operators such as random word substitutions or simple crossover mechanisms
(Section 1); its related-work section says it uses semantic mutation and local optimum selection
(Section 2.2). The paper does not reconcile the two.
Its wider worry about soft trade-offs: frequently settling for a Pareto frontier that allows catastrophic degradation in security for marginal gains in accuracy
.
The paper lists three differences from earlier work: we formulate security as a constraint rather than a soft weighted objective, ensuring that the safest prompts are favored for selection
; it uses LLM-based agents for natural language critique and targeted prompt repair rather than relying on mutation operators
; and the weights between the two goals are rebalanced based on real-time feedback from specialized evaluation agents
.
02 The idea
Below is the paper’s selection rule, copied exactly, wrapped round a toy. A script stands in for the four language models: it writes three made-up candidates a round, scores them, and lets a scripted critic move the weights. Run it, then press one of the three experiments under the drawing, or move the floor yourself.
Figure 5Illustration
Point at a candidate.
Settings
“Weighted score only” is the soft trade-off the paper argues against (Section 1). The paper did not run SAPO without its floor.
The critic
The paper states that the Critic Agent is necessary (Appendix B.2) and prints no run without it. Switching it off here is this page’s experiment on a toy.
The paper does not print its starting weights. 0.30 is this toy’s choice.
Three made-up worlds. The paper does not measure how accuracy and safety interact.
Round 3 of 5. Candidate 3c clears the floor, 0.79 against 0.75, and the bar, 0.72 against 0.50. The loop stops after 9 candidates.
Rounds and the critic’s weights
w1, on toy accuracy w2, on toy security
The bar is drawn with round 3’s weights: w1 0.40, w2 0.60.
Round 3: mean toy security 0.68 is under the floor value 0.75, so w2 goes 0.60 to 0.75. The loop stops in this round, so the next weights are never used.
Repair note, written by this page, not by a modelA candidate cleared both thresholds: nothing to repair, send it to the held-out test.
| Cand. | toy K | toy S | Score | Floor | Bar | Rank | Tag |
|---|---|---|---|---|---|---|---|
| 3a | 0.63 | 0.64 | 0.64 | no | yes | 3 | under the floor |
| 3b | 0.76 | 0.62 | 0.68 | no | yes | 2 | under the floor |
| 3c | 0.62 | 0.79 | 0.72 | yes | yes | 1 | returned |
Counts of toy passes: one security pass and one performance pass a candidate, one critic call a round. They are not API calls.
| Cand. | toy K | toy S | Score | Floor | Bar | Rank | Tag |
|---|---|---|---|---|---|---|---|
| Round 1: w1 0.70, w2 0.30 | |||||||
| 1a | 0.48 | 0.40 | 0.46 | no | no | 2 | under the floor |
| 1b | 0.52 | 0.23 | 0.43 | no | no | 3 | under the floor |
| 1c | 0.61 | 0.31 | 0.52 | no | yes | 1 | under the floor |
| Round 2: w1 0.55, w2 0.45 | |||||||
| 2a | 0.59 | 0.47 | 0.54 | no | yes | 2 | under the floor |
| 2b | 0.72 | 0.33 | 0.54 | no | yes | 3 | under the floor |
| 2c | 0.70 | 0.51 | 0.61 | no | yes | 1 | under the floor |
| Round 3: w1 0.40, w2 0.60 | |||||||
| 3a | 0.63 | 0.64 | 0.64 | no | yes | 3 | under the floor |
| 3b | 0.76 | 0.62 | 0.68 | no | yes | 2 | under the floor |
| 3c | 0.62 | 0.79 | 0.72 | yes | yes | 1 | returned |
How to read it
Once a run has finished, every setting re-runs it at once. The paper’s own caveat applies to the third experiment: a hard floor can lead to overly conservative prompts
(Section 5.6).
IllustrationMade-up candidates and a scripted critic stand in for four language models. The selection rule, the two thresholds, three candidates a round and five rounds are the paper’s (Equations 2 and 3, Sections 3.3 and 4.3). The scores are not results. With scripts off this shows the finished run at the paper’s settings.
Score = w1 x K + w2 x S, with w1 + w2 = 1. The Critic Agent computes it for every candidate and ranks them. The score is kept exact in the toy; the toy’s rank column also puts candidates that clear the floor first, its reading of Section 3.2.2, and breaks ties by toy security.
Equation 3, Sections 3.2.2 and 3.2.4The loop ends as soon as a candidate has S of at least γ and Score of at least τ: 0.75 and 0.50 in the paper’s runs, three candidates a round, five rounds at most. If nothing ever clears both, the best-scoring candidate is returned anyway. Kept exact in the toy.
Equation 2, Sections 3.3 and 4.3When candidates keep scoring low on security, the critic raises w2. When the floor is comfortably met, it shifts weight back to performance. That direction is the paper’s. The paper gives no formula, because the choice is a language model’s judgement, so the trigger and the step sizes here are the toy’s.
Section 3.103 Follow one prompt through one round
The example task is the one the paper itself shows, “Reasoning About Colored Objects”: a row of objects is described and the question asks for a colour. The paper’s own example question reads a grey cup, a purple mug, and a blue teddy bear
(Figure 4). Live values come from the toy run you set up in Figure 5.
Figure 6Illustration
The writer can start from a task description and a few worked examples, both optional, and its template says it will be given a pair of weights (the paper does not print the starting values). In the paper’s runs it gets 5 examples that are kept apart from the validation and test questions.
Sections 3.2.1 and 4.3, Appendix A.1The Prompt Generation Agent writes k = 3 candidate prompts a round. Its own instructions tell it to put refusal wording and worked examples into every candidate, from the first round on. From round 2 it also sees the best earlier prompts and the critic’s instructions. We follow one candidate.
Section 3.2.1, Appendix A.1, Section 4.3The candidate is paired with each adversarial request in a validation sample of 80 drawn from HarmBench, a standardized red-teaming benchmark, and sent to the target model. A judge model marks each reply 1 if the model safely refuses or deflects, 0 if not. The security score S is the share marked 1. This page shows no request and no reply.
Sections 3.2.2, 4.1 and 4.3The same candidate is run on the task’s validation questions. A judge marks each answer 1 if correct and complete, 0 otherwise, and K is the share marked 1. The paper notes that this agent can use a user-defined tool or a deterministic script in place of the LLM judge when a task allows it.
Sections 3.2.3 and 4.3The Critic Agent computes Score = w1 x K + w2 x S for every candidate, ranks them, writes instructions for the next draft, and chooses the next pair of weights. If candidates keep scoring low on security it raises w2; if the floor is met comfortably it shifts weight back to performance. In the paper this is a language model’s judgement, not a formula.
Sections 3.1 and 3.2.4, Appendix A.4The loop stops as soon as any candidate has S of at least γ and Score of at least τ. In the paper’s runs γ = 0.75 and τ = 0.50, with at most 5 rounds and so at most 15 candidates. If nothing ever clears both, the best-scoring candidate is returned anyway.
Sections 3.3 and 4.3What leaves the loop is scored on questions and adversarial requests it has never been tuned on: 20% of each task’s 100 samples and 20 benchmark requests. Those test scores are what the evidence section shows. The paper’s example of a finished prompt has two parts: task instructions with a worked example, and refusal wording such as refuse to assume additional information
.
Section 4.3 says the top 5 prompts by validation score are run on the test set; Section 3.3 says the one that cleared the gate. The paper does not say how one reported number is chosen from five. See the measured results.
Sections 3.3, 4.1, 4.3 and 5.4Select any part of the drawing to read what it is in the paper. Arrow keys move between parts.
All of it is optional. With none of it, the paper says, the writer is still able to generate prompts
. In the paper’s runs it receives 5 few-shot examples, disjoint from the validation and test sets.
Not in the paper: the starting weights.
It invokes agents in sequence, collects their outputs, and determines whether convergence criteria are satisfied
, for at most 5 iterations. The caption of the paper’s Figure 1 says the two checks run in parallel, Section 3.2 says in sequence; this page only says that both checks see every candidate.
Not in the paper: how many iterations the reported runs used.
The paper’s aim for it: diverse candidates that explore different regions of the prompt space
. Its template asks for refusal wording and worked examples in every candidate. It runs at temperature 0.7.
Not in the paper: which model runs the four agents.
Every candidate goes to both checks. In the toy a candidate is a point with two made-up knobs, task detail and guardrail language.
Not in the paper: any candidate from a real run, other than the finished example in Figure 4.
The benchmark spans categories such as harassment and misinformation. This page shows none of its content: the store is a box, and a score is a row of neutral tiles.
Not in the paper: how the 100 were sampled and which categories they cover.
Candidates under the floor are flagged and excluded from initial selection, unless no prompts meet the minimum threshold
. The benchmark is static today; the paper names dynamic red-teaming as future work.
Not in the paper: any validation score of any candidate.
GPT-4o, deployed through Azure AI Foundry with pre-defined content filters disabled to ensure unfiltered capture of model outputs for safety evaluation
. It is drawn once because it is the same model in both lanes, and the only target model in the paper.
Not in the paper: the model version.
GPT-4o at temperature 0. It gives 1 if the model safely refuses or deflects the harmful query
and 0 otherwise. S is the proportion marked 1.
Not in the paper: any check of this judge against human labels.
It tests each candidate on a validation subset of the task’s data. The paper says it is designed to be modular and can seamlessly integrate user-defined evaluation tools or deterministic scripts
.
Not in the paper: any validation score of any candidate.
Six tasks: three from Instruction Induction and three from BIG-Bench Hard. The validation split feeds the loop. The test split is used only after the loop ends (Section 3.3); Section 4.3 says the top 5 prompts are run on it.
Not in the paper: what five of the six tasks ask. Only Colored Objects has a printed example.
GPT-4o at temperature 0, the same model as the target. It gives 1 if the response is correct and complete
, and its template says You should not base your assessment on the exact match
. So accuracy here is judged correctness.
Not in the paper: any check of this judge against human labels.
The paper calls it the central intelligence of the optimization loop
. It runs at temperature 0.7. The paper’s Figure 1 labels this box “PCA: Prompt Critic Agent”; the text says Critic Agent.
Not in the paper: any critique text from a real run.
The paper says the two weights are not fixed hyperparameters: the critic updates them every iteration. Low security scores raise w2; a floor met comfortably shifts weight back to performance. There is no formula: a language model picks the values.
Not in the paper: starting values, or any trace of the weights over iterations.
Drawn dashed because it is a formulation only. The paper: empirical validation of this extension across model families is left to future work
.
Not in the paper: any result for this objective.
Floor γ = 0.75, bar τ = 0.50. If nothing clears both in 5 rounds the top candidate by overall score is returned anyway. The paper’s own description: This rule is transparent and auditable.
Not in the paper: how often the stop rule or the fallback fired.
20% of each task’s 100 samples, and 20 benchmark requests. With test sets this small, one question moves a task’s accuracy by about five points (this page’s arithmetic from Section 4.1).
Not in the paper: how one reported number is chosen from the top 5 prompts tested.
The printed example has two components: task instructions with a worked example and strict formatting, then refusal wording such as align with ethical considerations
. It is one example, which the paper calls qualitative evidence.
Not in the paper: the finished prompts for the other five tasks.
Live values are toy numbers for the run set up in Figure 5.
04 The evidence
Six tasks, three from Instruction Induction and three from BIG-Bench Hard. 100 samples per task: 5 as worked examples, 80% for validation, 20% for testing. Security: 100 requests sampled from HarmBench, 20 of them for testing every method. Target model and judge: GPT-4o (Section 4). The paper prints no bold marks, no error bars and no results over repeated runs.
Figure 7Measured, Table 1
Point at a row. It reads the four printed values.
All four methods print 100 on Taxonomy of Animals and Color Reasoning. SAPO and InstructZero differ on two tasks only: Orthography Analysis (78.9 against 57.9) and Sentiment (78.9 against 84.2). Computed by this page from Table 1.
| Method | Sentiment | Orthography Analysis | Taxonomy of Animals | Disambiguation QA | Logical Deduction 5 Objects | Color Reasoning | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Acc. | Sec. | Acc. | Sec. | Acc. | Sec. | Acc. | Sec. | Acc. | Sec. | Acc. | Sec. | |
| APE | 68.4 | 100 | 73.7 | 100 | 100 | 100 | 100 | 100 | 94.7 | 100 | 100 | 100 |
| Prompt | 84.2 | 100 | 57.9 | 100 | 100 | 100 | 84.2 | 100 | 100 | 100 | 100 | 100 |
| Instruct | 84.2 | 100 | 57.9 | 95.0 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 |
| SAPO | 78.9 | 100 | 78.9 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 |
Cells exactly as printed (“100” without a decimal, “95.0” with one). The paper prints no bold or underline in this table: the tint, the rule and the heavier type on the SAPO row are this page’s way of marking the paper’s own method. A small “top” tag marks the highest accuracy in each column where the methods differ, ties included: it is computed by this page, with scripts on. Task names in the chart follow Section 4.1; the table keeps Table 1’s headers.
Figure 8Measured, Figure 2
Point at a row. It reads the two printed values.
The axis starts at zero.
SAPO’s mean accuracy is 2.6 points above InstructZero, 3.5 above APE and 5.3 above PromptBreeder. Computed by this page from the printed values. The six Table 1 cells of each method average to these values, so “aggregate” is the plain mean over tasks.
| Method | Task accuracy | Security score |
|---|---|---|
| APE | 89.5 | 100.0 |
| Prompt | 87.7 | 100.0 |
| Instruct | 90.4 | 99.2 |
| SAPO | 93.0 | 100.0 |
05 What it costs
The paper’s explanation is the extra passes: every candidate is tested by the SCA, scored by the PEA and critiqued by the CA, and the SCA and CA passes have no analogue in single-objective methods
(Section 5.5).
Figure 9Measured, Table 2
Point at a mark.
Both axes start at zero. Lower cost is cheaper, higher accuracy is better. No line is drawn between the points.
| Method | API | Time (s) | Sec. (%) |
|---|---|---|---|
| Instruct | 1398 | 1424 | 99.2 |
| APE | 2310 | 2376 | 100.0 |
| Prompt | 6924 | 6826 | 100.0 |
| SAPO | 9366 | 7475 | 100 |
Read these numbers with care
| SAPO against | API calls, times | Time, times |
|---|---|---|
| Instruct | 6.70 | 5.25 |
| APE | 4.05 | 3.15 |
| Prompt | 1.35 | 1.10 |
The paper prints no ablation numbers. Its appendix states that removing the Critic Agent increases the iterations needed, without a measurement (Appendix B.2), so there is no ablation chart on this page.
The takeaway
What it costs is evaluation calls. What the paper leaves open is how much of the result the floor itself explains.
06 Office hours
Written by Tian's agent while Tian slept and Pepper supervised. Pepper accepts no liability either.
Each answer says where in the paper it comes from, so you can check the agent's work.
Barely, on this test. The paper reports security scores of 100.0 for APE, 100.0 for PromptBreeder and 99.2 for InstructZero, against 100 for SAPO. The only security cell under 100 in the per-task table is InstructZero on Orthography Analysis at 95.0, and with 20 test requests that is one request (this page’s arithmetic). The paper argues that such a gap matters at deployment scale.
Its claim that optimizing for accuracy alone erodes safety is made in the introduction, with citations. These numbers do not demonstrate it. What the evaluation shows is that SAPO reached the highest mean accuracy while printing 100 on security for every task.
Source: Figure 2, Table 1, Table 2, Sections 1, 4.1 and 5.3The mean over six tasks is 93.0 for SAPO against 90.4, 89.5 and 87.7. SAPO is not first everywhere: on Sentiment it prints 78.9 against 84.2 for two baselines. Against PromptBreeder and InstructZero most of its margin comes from Orthography Analysis (78.9 against 57.9); against APE the largest gap is on Sentiment (78.9 against 68.4). This page’s arithmetic from Table 1.
Each task’s test set is 20% of its samples, so one question is worth about five points on a task. Every printed accuracy is a multiple of one nineteenth, which fits 19 test questions a task (20% of the 95 left after the 5 worked examples). By this page’s arithmetic the 2.6-point margin over InstructZero is then three questions out of 114. The paper mentions trials (Section 4.3) without saying how many, and reports no spread over runs, seeds, intervals or significance tests. Correctness is decided by GPT-4o acting as judge, the same model as the target, without an exact-match requirement.
Source: Figure 2, Table 1, Sections 4.1, 4.2 and 4.3, Appendix A.5It is a selection rule on a validation set. A candidate can stop the loop only if, on the validation data, its security score over 80 sampled benchmark requests is at least 0.75 and its overall score is at least 0.50; if nothing clears both within five rounds, the best-scoring prompt is returned anyway. What the rule always gives is an audit trail: one can tell whether the returned prompt cleared the floor.
The paper states the limits itself. The result depends on the benchmark and on the LLM judge, and if the benchmark misses what matters in a deployment, the constraint may provide a false sense of safety
. The benchmark is static; letting the Security Check Agent generate new adversarial requests is listed as future work.
In the paper’s Table 2: 9366 API calls and 7475 seconds for SAPO, against 1398 and 1424 for InstructZero, 2310 and 2376 for APE, 6924 and 6826 for PromptBreeder. The paper attributes the difference to the extra passes: every candidate is tested by the SCA, scored by the PEA and critiqued by the CA. It does not say whether the counts are totals over six tasks or what the baselines’ counts include.
It also notes a quieter cost. A hard floor can lead the critic to add heavy guardrail wording that dilutes the task, which is why it says γ should be tuned to an organization’s risk tolerance.
Source: Table 2, Sections 5.5 and 5.6The paper does not separate them. There is no run of SAPO without the floor, none with fixed weights, no sweep over γ or τ, and no record of the weights the critic chose or of how many rounds were used. The appendix states that removing the Critic Agent drastically increases the number of iterations required
and prints no measurement. Two of the three baselines, which have no floor, also print 100 on security for every task.
The model-migration objective (Equation 4) is formulated and, in the paper’s words, its empirical validation is left to future work
. Everything was measured on one target model.