Using Different AI Models to Critique and Evaluate Each Other

There is a temptation with AI to keep asking the same model to improve its own work.

Generate something, review it, improve it, then review it again.

That can work, especially with the stronger reasoning models. But there is a point where you are still asking the same system to inspect a line of thinking it already believes in.

For bigger jobs or those that involve fundamental reasoning of a task, the better move is to give the work to another model and ask for a second opinion.

My first real cross-model eureka moment

I first started doing this properly while working on MyTourneyTime.

At the time I was still very new to using AI models seriously, never mind deliberately using different models together. My main platform was Abacus AI, which made it easy to move between different model families, so I started taking planning and architecture work from one model and asking another to critique it.

That was a bit of a eureka moment.

The second model did not necessarily dismantle the first answer. Quite often it agreed with most of it. But it might notice a missing assumption, an unnecessarily complicated section, or a decision that looked sensible from one angle but less convincing from another.

The value was not in trying to manufacture an argument between two models. It was in getting a genuinely fresh perspective.

One strong model is usually enough

I do not think every task needs this.

Strong models such as Sol, Fable or Astra are perfectly capable of handling most of the work I give it (most don't require models as strong as those named). If I am making normal website changes, researching something fairly straightforward, working on copy, or improving a page layout, there is usually little value in putting every answer through several different models.

You could probably squeeze out a slightly better result, but you could also turn ordinary work into a committee meeting.

Cross-model evaluation becomes more interesting when the task is foundational, interpretive or difficult to test objectively.

That might include:

• architecture;
• high-level planning;
• system or skill design;
• decisions with several plausible answers;
• work that will influence a lot of later work;
• situations where a bad assumption could become expensive to undo.

Those are the areas where a second reasoning path can justify the extra effort.

Different model families give you fresher eyes

If I am going to ask for a second opinion, I would rather it came from a different model family.

Using another model from the same family can still help, but you are more likely to keep moving down broadly similar reasoning paths. Different models have different tendencies, strengths and instincts around how they approach a problem.

It is much closer to asking another person to look at your work.

You are not necessarily looking for somebody to disagree with you. You are looking for a fresh pair of eyes.

I used this recently while building a Skill Architect: a skill designed to help create better AI skills. Most of the original thinking from Sol on a very high reasoning level stood up well, which in itself was useful validation.

But when I brought another model, Fable, into the process, it picked up a few places where the design risked becoming more complicated than it needed to be.

They were not major failures. They were the kind of smaller weaknesses that become easier to see when the reviewer did not produce the original answer.

Agreement can be useful too

The idea of AI judges and evaluators can sound more adversarial than it needs to be.

Quite often I actually want the models to agree.

If one capable model proposes an approach and another capable model from a different family independently reviews it and finds most of it sound, that gives me more confidence in the answer.

Where the evaluator finds something useful, I have also found the original model is usually quite willing to accept the criticism. You can take the critique back, ask it to reconsider the point, and often it will recognise something it missed and revise the work.

If they genuinely disagree, the human becomes more important. At that point I can decide which argument I find more convincing or, if the decision really matters, bring in a third model.

By then, though, you are often well into the finer details.

Give the evaluator enough context, but not necessarily the same framing

There is a big difference between asking:

“What do you think of this?”

and giving another model a genuine evaluation job.

The evaluator needs enough context to understand what is being worked on. I will normally explain the broad task, what the first model was asked to consider, and then give it the answer or plan that was produced.

But I would not always give it the entire original brief word for word.

Sometimes that can work against what I am trying to achieve.

If the original framing has already missed something important, giving the second model exactly the same framing risks anchoring it to the same assumptions. I want the evaluator to understand the problem, but I also want it to have enough freedom to stand back and ask whether we are solving the right problem in the first place.

So the second model might get:

• an outline of the overall goal;
• the important constraints or context;
• an outline of what the first model was asked to do;
• the answer, plan or architecture it produced.

Then I want it to consider both the quality of that answer and the wider task around it.

It might find incorrect assumptions, missing considerations, unnecessary complexity, weak reasoning or areas of unjustified confidence. It might also spot something that neither I nor the first model had included in the original framing.

That broader perspective is part of the value.

Sometimes the useful evaluation is simply that the original answer is already strong. Other times the most useful thing the evaluator does is question something that everybody had previously assumed was settled.

Know when to stop

The obvious danger is that you can evaluate forever.

One model produces something. Another critiques it. The first revises it. A third model critiques the revision, and suddenly you are burning a lot of tokens looking for an answer that is fractionally better.

The amount of evaluation has to match the importance of the work.

For most of what I am doing, once two strong models have explored the main weaknesses and there is broad agreement, I am probably done. If you are working on something where failure carries much greater consequences, the level of checking should obviously rise with it.

There is nearly always another improvement available somewhere. That does not mean it is worth chasing.

This becomes more interesting with evaluator agents

What interests me most is where this leads.

So far, I have often been the person manually moving work between models. One produces something, I send it somewhere else, that model critiques it, then I return the criticism to the first one.

Technically, I am the human in the loop throughout that process.

But I am not necessarily contributing much expertise at every handoff.

If two capable systems are designing an AI skill using documentation from OpenAI, Anthropic and other sources, they already know far more about the technical detail than I do. Much of my role is orchestration.

That makes the next step fairly obvious: a more generative system where a team of agents can work towards a substantial goal, break the problem down, evaluate each other's work and revise where they come up short before presenting the result to me.

Ideally, what comes back is already very good and only needs a bit of human refinement.

Human in the loop does not have to mean human in every handoff

I still think the human needs to be in the loop. I am less convinced that the human needs to perform every individual action.

The human should retain the larger goal, important constraints, taste and final judgement.

That becomes particularly important where there is no objective acceptance criterion. A maths problem can have a right answer. A design decision often cannot. At some point somebody has to decide whether the thing actually feels right.

There are also moments where even a capable model can suddenly head in the wrong direction.

I had this recently while working on an animated website hero. We had spent a long time deliberately keeping the moving subject separate from the background because that separation was fundamental to the implementation.

Later in the process, the agent casually suggested combining them into a single asset.

It was a small mistake, but blindly following it would have started undoing a lot of the work we had already done.

That is why I still want visibility into what the system is doing.

I want to be able to look into the tank

The analogy I keep coming back to is a fish tank.

If I build a more autonomous agent system, I do not want to have my hands in the tank constantly moving everything around. That rather defeats the point.

But I do want clear glass.

I want to be able to see what the system is trying to achieve, what decisions have been made, where different agents disagree and why. If something looks wrong, I want to be able to step in, query the reasoning or steer it back towards the goal.

That oversight should not become another job in itself.

Too little visibility and you lose control. Too much and you have simply rebuilt manual work with more agents.

Finding the useful middle ground is going to be an important part of building genuinely effective generative systems.

Fix the system rather than patching every output

Visibility also helps when the same kind of failure keeps happening.

If an autonomous workflow repeatedly produces work that is nowhere near what you want, there comes a point where correcting the output is no longer the answer.

The problem may be in the setup.

The goal might be unclear. The agents may have weak context. The evaluator may be measuring the wrong thing. Important constraints may not be persistent enough.

At that point, the better move is to improve the system producing the answers rather than patch every answer individually.

A simple version you can use now

You do not need a multi-agent platform to get most of the benefit.

Start with one strong model and get the best answer you can.

Then take the proposed answer to a capable model from another family. Give it enough context to understand the overall task and the constraints that matter, but consider whether repeating the entire original framing would unnecessarily anchor its thinking.

Ask it to evaluate the work rather than simply replace it.

You want it looking for weaknesses in the answer, but also thinking more broadly about whether the approach itself has missed anything important.

Take the useful criticism back to the original model and let it reconsider the solution.

If the two substantially converge, stop.

If they disagree on something important, make the judgement yourself or bring in another perspective.

That simple manual workflow is already useful today. What makes it more interesting is that it also looks like a small-scale version of where agent systems are heading: different models creating, challenging, validating and improving work before the human needs to intervene.