Building a Skill Architect to Design Better AI Skills

The problem was not the individual skills

I had already made a few AI skills that worked well enough to convince me the idea was useful.

Some though had started drifting away from what I originally wanted, and fixing that usually meant going back into the instructions and adding another correction, another exception or another version of don't do this. The skill might improve again, but the underlying document was becoming harder to reason about.

Eventually I started wondering whether I was fixing the wrong thing.

Until then, the process had been fairly straightforward: explain what I wanted to a capable chat model, discuss it for a while, then turn the result into a skill. ChatGPT itself makes skill creation approachable in much the same way; start with a conversation and build from there.

I wanted to see whether I could push that process further.

A skill for making skills

The idea became a Skill Architect.

Instead of starting every new skill with a general-purpose chat and rediscovering the same design questions, I wanted something whose job was already understood:

That doesn't mean it should turn every tiny personal utility into an engineering project. Avoiding that became one of the most important parts of the design.

If I want a simple skill that performs one small repeatable job, the process should stay simple. The Architect should recognise that quickly, ask only for information that genuinely changes the design, and get on with it.

A more consequential skill might justify research, stronger testing, tool controls or a much clearer evidence trail.

The architecture eventually settled around a phrase that captures that balance quite well:

Cheap thinking, proportional evidence.

Thinking carefully about what sort of skill is needed is relatively cheap. The more expensive work — deep research, formal evaluation, extensive documentation or stronger controls — should only appear when the job earns it.

Using different models to challenge the design

The Skill Architect itself was built through a deliberately model-heavy process.

I started with a long research pass looking at what good reusable AI skills actually consist of: OpenAI's current approach, Anthropic's work on Agent Skills, progressive disclosure, triggering, evaluation, tool use, failure handling, instruction design and the point at which something should not be a skill at all.

That produced a research report of roughly 17,000 words.

I then passed the research into Codex and asked it to design the architecture rather than immediately build the skill. I wanted the first output to explain the system we should create, not simply generate a very large SKILL.md.

Codex produced a detailed implementation specification. I passed both the research and that specification to Claude Fable and asked it to attack the design.

That critique highlighted three areas that changed the next version substantially.

The first was proportionality. The initial architecture risked making every skill go through the same heavyweight process, regardless of whether it was a personal shortcut or something much more consequential.

The second was user elicitation. For subjective skills, the user may be one of the most important sources of truth. A writing skill, for example, cannot discover someone's preferred voice purely from external research.

The third was principle discovery. The system was becoming good at classifying, evaluating and documenting skills, but it needed a better way to discover the small number of governing ideas that can replace dozens of smaller rules.

Those criticisms went back into Codex rather than simply being accepted as instructions. Codex assessed them against the original research and produced a second architecture.

Only after that did I ask it to build the actual Skill Architect.

Finding principles instead of collecting rules

One of the ideas that came out of that back-and-forth was that a good skill should not just become a longer and longer list of instructions.

If something goes wrong, the easiest response is to add another rule. That can work for a while, but eventually the skill becomes difficult to reason about because every new instruction is compensating for the last problem.

The better question is whether several of those problems point to the same underlying principle.

That principle might be about how much freedom the model should have, what the user is authoritative about, when the skill should stop and ask, or what trade-off it should consistently resolve.

The aim is to get to a smaller number of strong ideas that guide behaviour across lots of situations, then only keep the extra rules that are genuinely necessary.

That gives the Skill Architect another job beyond simply assembling instructions: it should help work out what the skill is actually trying to preserve.

Sometimes the answer should be no

Another part I wanted to keep was the ability for the Architect to decide:

This should not be a skill.

Some jobs are better handled by a normal prompt. Others belong in project instructions, deterministic code, a tool integration or a larger workflow.

At the same time, I didn't want the test to become overly academic. A personal skill can be worthwhile simply because it saves repeated prompting or gives me a predictable way to perform a job I do regularly. It doesn't have to unlock some ability that ChatGPT could never perform without it.

I now think about the value of a skill across several different areas:

• Capability — it can do something the base agent struggles to do.
• Consistency — it does the same job in the same way each time.
• Compression — a tiny invocation can replace a much longer explanation.
• Defaults — decisions about behaviour are already made.
• Reliability — known failure modes can be corrected once rather than in every conversation.

A deliberately silly first test

I wanted the first test to be something where I had enough personal knowledge to spot a bad answer quickly.

So I made a Liverpool FC skill.

I give it a footballer's surname and it works out whether that person played for Liverpool. If they did, it gives me some basic information and tries to find the crowd song associated with them.

The invocation can be as small as:

$lfc McManaman

It found Steve McManaman and his old crowd song without any problem.

I then started trying more awkward examples. It correctly rejected players who never played for Liverpool, handled Rafael Benítez as a manager rather than inventing a playing career, and coped reasonably with misspellings and ambiguous surnames.

Karl-Heinz Riedle was a more difficult test. It found the player but initially couldn't find his song. I knew one existed from memory, (To a Bananarama tune. An ear-worm that's stuck with me since his beautiful chip at Elland Road, many years ago!), yet even when I did a quick Google search myself it was difficult to surface. That was a useful reminder that a skill doesn't somehow gain perfect knowledge just because its instructions are well designed.

The interesting part was the experience of using it. I could type a tiny command and a surname and get the behaviour I wanted without explaining the Liverpool check, the player identification, the information format and the crowd-song search every time.

The Skill Architect had also created the skill from a very short description without turning the process into anything more complicated than it needed to be.

For an early test, that was encouraging.

A more useful test: choosing the right Codex level

The next skill is much closer to something I expect to use regularly.

I called it codex-level.

I give it a prompt I am about to send to Codex and it recommends the model and reasoning level it thinks is appropriate.

The purpose isn't to find the cheapest possible option at all costs. I mainly want to avoid automatically reaching for an expensive reasoning level when a simpler task doesn't need it.

Early testing has already produced some interesting behaviour. The skill often prefers Terra Medium for ordinary implementation work, but it has moved to Sol when a task contained more unresolved design judgement. It has also recommended lighter or higher reasoning independently when the task warranted it.

I don't yet know whether its calibration is actually better than simply asking a normal ChatGPT agent each time. There may not even be a huge performance difference.

The potential value is having a stable decision framework available every time, with the behaviour already packaged into a short invocation.

I'll cover that skill separately once I've used it enough to have something useful to say about whether the recommendations hold up.

What the first tests do and don't prove

The early results are promising, but they don't prove that the Skill Architect creates better skills.

A long research report, several frontier models critiquing each other and a carefully designed SKILL.md can still produce something that looks sophisticated without being particularly useful.

The Architect has been designed to resist the easiest response: adding another prohibition to the prompt. Instead, it should work out whether the failure points to a broader principle, a local exception, bad evaluation, missing knowledge or a problem with the underlying architecture.

That behaviour will probably tell me more about its value than whether the first version of every skill is perfect.

Why I'm not sharing it yet

I'm deliberately not publishing the Skill Architect at this stage.

At the moment it is an experiment that has produced a couple of encouraging results. I need more time using it before I know whether the process genuinely produces skills that stay useful and easier to maintain.

If it does, I'll look at sharing it.

For now, its immediate value is much simpler: when I decide I want to make a new skill, I no longer have to begin by working out how to design one.

There is already something in the toolbox whose job is to think about that problem.