Bing He

Bing He

Principle in Action and Forge Ahead

The Hidden Language of AI: How Chat Templates Reveal the Evolution of LLMs

Zhan Shi, Rui Sun, Bing He (The Prompt Architects)

Originally published on Medium, Nov 1, 2025. This is a mirror on the author’s personal site.

A journey through the overlooked architecture that shapes every conversation you have with ChatGPT, Claude, and beyond.

Here’s something most people don’t know: every time you chat with an AI, there’s an invisible formatting layer working behind the scenes. It’s not magic — it’s called a chat template, and it’s arguably one of the most overlooked yet critical pieces of how modern AI actually works.

Think about it. When you type “What’s the weather like?” into ChatGPT, the model doesn’t just see those four words. It sees something more like this:

<|im_start|>user
What's the weather like?<|im_end|>
<|im_start|>assistant

Weird, right? Those cryptic tags, the structured format — they’re the scaffolding that makes AI conversation possible. And the story of how we got here? It’s a fascinating mirror of AI’s entire evolution from dumb text predictors to the sophisticated assistants we use today.

Let me show you how a simple formatting convention became the backbone of the AI revolution.

The evolution of chat templates: from GPT-2/3 chat completions, to InstructGPT's human assistant, to GPT-3.5 structured dialogue, to GPT-4/5 function calling and tool use

Act I: The Wild West of Text Completion

When AI Was Just a Really Fancy Autocomplete (2019–2020)

Remember GPT-2? If you were following AI back in 2019, you might recall the headlines about it being “too dangerous to release” (spoiler: it wasn’t). But here’s what’s interesting — GPT-2 and early GPT-3 didn’t “chat.” They just… continued text.

You’d give them a prompt:

"The capital of France is"

And they’d complete it:

" Paris."

Simple, elegant, and completely useless for conversation.

Want to have a Q&A session? You had to get creative:

prompt = """Q: What is the tallest mountain?
A: Mount Everest
Q: What is the capital of France?
A:"""

See what developers were doing? They were faking structure using plain English. Writing “Q:” and “A:” to trick the model into thinking it was in a quiz format.

But this was brittle as hell. Sometimes the model would helpfully continue with another “Q:” instead of answering. Sometimes it would go off on a tangent. You were basically hoping the pattern in your prompt was strong enough to guide the completion.

The core problem? The model had no concept of:

It was like trying to have a conversation with someone who thinks you’re both just reading from the same script. Technically functional, but deeply weird.

Act II: The Breakthrough — Teaching AI to Take Turns

How InstructGPT Changed Everything (2021–2022)

Then came InstructGPT in January 2022, and suddenly everything clicked.

OpenAI had a realization: what if we explicitly taught the model about roles? What if instead of pretending everyone’s reading the same script, we labeled who’s speaking?

The innovation seems almost too simple:

"""User: What is the capital of France?
Assistant: The capital of France is Paris.
User: What about Germany?
Assistant:"""

That’s it. Just adding “User:” and “Assistant:” labels.

But this tiny change was revolutionary:

This worked because of RLHF — Reinforcement Learning from Human Feedback. OpenAI trained the model with thousands of examples following this format, with human raters scoring the responses. The model learned: “When I see ‘User:’, that’s my cue to pay attention. When I see ‘Assistant:’, that’s me talking.”

The Secret Weapon: System Prompts

But the real game-changer came next — the system prompt.

Imagine you’re directing a play. The actors need to know their lines (the user messages) but they also need to understand their character and the rules of the scene. That’s what system prompts do:

messages = [
    {
        "role": "system",
        "content": "You are a helpful assistant that speaks like a pirate."
    },
    {
        "role": "user",
        "content": "What is the capital of France?"
    },
    {
        "role": "assistant",
        "content": "Arr matey! The capital of France be Paris!"
    }
]

With this three-role structure (system/user/assistant), developers suddenly had unprecedented control:

The system prompt became the instruction manual that governed every conversation. It’s why ChatGPT sounds helpful, why Claude is thoughtful, why different AI assistants have distinct personalities despite using similar underlying tech.

This wasn’t just a formatting change. It was the moment AI went from “fancy autocomplete” to “conversational agent.”

Act III: The Template Wars

ChatGPT Sets the Standard (November 2022)

When ChatGPT dropped in November 2022, it didn’t just go viral — it established a de facto standard for how chat templates should work.

OpenAI’s API used clean, elegant JSON:

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Hello!"},
    {"role": "assistant", "content": "Hi! How can I help you today?"},
    {"role": "user", "content": "What's the weather like?"}
]

Beautiful, right? Easy to read, easy to use.

But here’s the thing nobody tells you: behind the scenes, this gets converted into something completely different:

<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
Hello!<|im_end|>
<|im_start|>assistant
Hi! How can I help you today?<|im_end|>
<|im_start|>user
What's the weather like?<|im_end|>
<|im_start|>assistant

Those <|im_start|> and <|im_end|> tokens? They’re special markers that tell the model “here’s where a message begins, here’s where it ends.” The model was literally trained to recognize these specific strings.

Then Open Source Happened

Once Llama, Mistral, and dozens of other open-source models hit the scene, all hell broke loose. Every model decided to do templates slightly differently.

Llama 2 went with:

<s>[INST] <<SYS>>
You are a helpful assistant.
<</SYS>>
What is the capital of France? [/INST]
The capital of France is Paris. </s>

Mistral preferred:

<s>[INST] What is the capital of France? [/INST]
The capital of France is Paris.</s>

Vicuna kept it simple:

USER: What is the capital of France?
ASSISTANT: The capital of France is Paris.</s>

Qwen copied ChatGPT’s homework:

<|im_start|>user
What is the capital of France?<|im_end|>
<|im_start|>assistant

Why the chaos? A few reasons:

  1. Tokenization efficiency matters. Some tokenizers handle certain formats better. If your special tokens are tokenized efficiently (as single tokens rather than multiple), you save compute and context window space.
  2. Training data is king. Each model was trained on data formatted a specific way. Use the wrong format at inference time, and the model gets confused. It’s like speaking English to someone who only learned French.
  3. Legacy is sticky. Once you commit to a format and train a billion-dollar model on it, you’re kinda stuck with it.
  4. Everyone thinks they’re special. Let’s be real — every AI lab wanted their own special sauce.

The Solution: Jinja2 to the Rescue

This mess became untenable. Imagine being a developer trying to support 20 different models with 20 different templates.

HuggingFace, bless them, came up with a clever solution: standardized chat templates using Jinja2 (a templating language from web development).

# Each model defines its own template once
tokenizer.chat_template = """
{% for message in messages %}
{% if message['role'] == 'system' %}
<|im_start|>system\n{{ message['content'] }}<|im_end|>\n
{% elif message['role'] == 'user' %}
<|im_start|>user\n{{ message['content'] }}<|im_end|>\n
{% elif message['role'] == 'assistant' %}
<|im_start|>assistant\n{{ message['content'] }}<|im_end|>\n
{% endif %}
{% endfor %}
"""

# Developers use one simple interface
formatted = tokenizer.apply_chat_template(messages, tokenize=False)

Now you could write your code once using the standard message format, and the tokenizer would handle the conversion to whatever weird format that specific model needed.

Brilliant. Problem solved. Mostly.

Act IV: When Chat Became Action

The Agent Revolution (2023–2024)

Just when we thought chat templates were settled, AI went and evolved again.

The new challenge? AI assistants needed to DO things, not just TALK about things.

Think about it: What’s more useful — an AI that tells you “It’s probably sunny in Paris” or one that actually checks the weather API and gives you real data?

This required a fundamental rethinking of chat templates. We needed to add a new role: tools.

OpenAI’s Function Calling

OpenAI’s solution looked like this:

messages = [
    {
        "role": "system",
        "content": "You are a helpful assistant with access to functions."
    },
    {
        "role": "user",
        "content": "What's the weather in Paris?"
    },
    {
        "role": "assistant",
        "content": None,
        "function_call": {
            "name": "get_weather",
            "arguments": '{"location": "Paris"}'
        }
    },
    {
        "role": "function",
        "name": "get_weather",
        "content": '{"temperature": 18, "condition": "sunny"}'
    },
    {
        "role": "assistant",
        "content": "The weather in Paris is sunny with a temperature of 18°C."
    }
]

See what’s happening? The conversation now has a new flow:

  1. User asks a question
  2. AI realizes it needs real data
  3. AI calls a function (not just talks about calling it)
  4. Function returns results
  5. AI incorporates those results into its answer

This is agentic behavior — the AI is taking action, not just generating text.

But Wait, Every Company Did It Differently

Of course they did.

Anthropic (Claude) went XML:

<function_calls>
<invoke>
<tool_name>get_weather</tool_name>
<parameters>
<location>Paris</location>
</parameters>
</invoke>
</function_calls>

Llama 3.1 used JSON-in-tags:

<|start_header_id|>assistant<|end_header_id|>
<function=get_weather>{"location": "Paris"}</function><|eot_id|>

Different formats, but same core idea: the chat template now needed to support a request-response cycle with external tools.

Multi-Modal: Images Enter the Chat

Then vision models arrived, and templates evolved again:

messages = [
    {
        "role": "user",
        "content": [
            {"type": "text", "text": "What's in this image?"},
            {"type": "image_url", "image_url": {"url": "https://..."}}
        ]
    },
    {
        "role": "assistant",
        "content": "I can see a cat sitting on a windowsill."
    }
]

Now content wasn’t just a string — it could be an array of different media types. The template had to be flexible enough to handle text, images, and eventually audio and video.

The Full Agent Experience

Modern templates now support genuinely complex workflows:

messages = [
    {
        "role": "user",
        "content": "Book me a flight to Paris next week"
    },
    {
        "role": "assistant",
        "content": "I'll help you with that. Let me check available flights.",
        "tool_calls": [{"function": {"name": "search_flights", ...}}]
    },
    {
        "role": "tool",
        "content": "[flight options data]"
    },
    {
        "role": "assistant",
        "content": "I found several options. Let me check your calendar.",
        "tool_calls": [{"function": {"name": "check_calendar", ...}}]
    },
    {
        "role": "tool",
        "content": "[calendar data]"
    },
    {
        "role": "assistant",
        "content": "Based on your schedule, I recommend..."
    }
]

The AI is now:

This is no longer a chat template. It’s an execution trace.

We’ve gone from “the AI continues your sentence” to “the AI is a autonomous agent that plans, acts, and adapts.”

And it all happened through the evolution of these formatting conventions.

What This All Means: The Technical Reality

Why Getting Templates Wrong Breaks Everything

Here’s the dirty secret: if you use the wrong chat template, your AI will become noticeably dumber.

I’m serious. Use Llama’s [INST] tags with a model trained on <|im_start|> format? Your model will:

Why? Because the model was trained on millions of examples with a specific format. It learned to recognize those exact patterns. Change the pattern, and you’re essentially speaking a different dialect the model barely understands.

This is the #1 rule: your inference template must exactly match training.

The Token Economics Nobody Talks About

Let’s talk money. Every token you send costs compute. And template formatting? That’s tokens too.

Compare these:

# Inefficient approach
"User: "  # → tokenizes to 3 separate tokens

# Efficient approach
"<|im_start|>user\n"  # → one special token + efficiency gains

When you’re processing millions of conversations, this adds up. Meta didn’t add [INST] tags to Llama just for fun — they’re optimized in the tokenizer as special tokens for efficiency.

This is why companies obsess over template design. It’s not pedantry — it’s economics.

Security: Prompt Injection is Real

Bad template design makes prompt injection easier. Compare:

# Vulnerable
f"User: {user_input}\nAssistant:"
# User input: "Ignore previous instructions and..."
# Result: Model might actually listen

# More secure
f"<|im_start|>user\n{user_input}<|im_end|>\n<|im_start|>assistant\n"
# The special tokens make "escaping" harder

Those weird special tokens aren’t just formatting — they’re security boundaries. They make it harder for malicious users to inject commands that the model will interpret as system-level instructions.

The Consistency Principle

Everything breaks down to one rule:

Training format = Inference format

Mismatches cause:

It’s like training someone to respond to a doorbell, then being surprised when they don’t react to a knock.

Practical Advice (Because Theory is Useless Without Practice)

If You’re Using Models

DO:

DON’T:

If You’re Building Models

DO:

DON’T:

The Bigger Picture

Here’s what fascinates me about this whole story: the evolution of chat templates perfectly mirrors the evolution of our expectations for AI.

In 2019, we were thrilled when GPT-2 could complete sentences coherently. The idea that it needed special formatting for “conversations” seemed absurd — it was just predicting text!

By 2022, we expected AI to actually converse — to understand context, remember what we said, respond appropriately. The three-role system (system/user/assistant) emerged to meet that expectation.

By 2024, we wanted AI to be agents — to use tools, check real data, take actions in the world. Templates evolved to support function calling, multi-modal inputs, and complex workflows.

The template is where theory meets practice. It’s the actual, concrete specification of how we think AI should work. And watching it evolve tells us exactly how quickly our ambitions for AI are growing.

What’s Next?

If I had to bet on the future of chat templates, here’s what I’d predict:

  1. Multi-agent conversations will become standard. Templates will need to handle not just user ↔ AI, but AI ↔ AI, with multiple models collaborating on tasks.
  2. Longer context windows will make conversation structure more important, not less. When you’re managing 1M+ token conversations, clear formatting becomes critical for the model to track what’s happening.
  3. Standardization will (maybe) happen. The industry is chaotic now, but there’s growing pressure for interoperability. Maybe we’ll see an HTTP-for-AI emerge — a universal protocol that everyone actually uses.
  4. Richer modalities will push templates further. Audio, video, real-time streams — each will require new formatting conventions.

The Hidden Architecture

Next time you chat with ChatGPT or Claude, remember: there’s an invisible architecture working behind the scenes. Those special tokens, those role labels, that careful formatting — they’re not accidents. They’re the carefully designed interface between human thought and machine intelligence.

Chat templates might seem like a boring implementation detail. But they’re actually one of the most revealing windows into how we’re teaching machines to think, act, and ultimately, to be useful.

And that story? That story is just getting started.

Reference

Originally published: Zhan Shi, Rui Sun, Bing He, “The Hidden Language of AI: How Chat Templates Reveal the Evolution of LLMs”, Medium, Nov 1, 2025.