LLM Fundamentals

Hosted models and open weights

Almost every current chat model is a transformer. It is trained to predict the next token, then tuned to follow instructions. The ideas here — tokens, temperature, messages, cost — do not change when you switch from Groq to OpenAI, Anthropic, or Gemini. The client library and the model name change.

Starter

messages = [
    {"role": "system", "content": "Reply with one short sentence."},
    {"role": "user", "content": "What is a token?"},
]
print("send messages, read text + usage from the response")
print(messages[1]["content"])

Almost every current chat model is a transformer, the architecture from the 2017 paper Attention Is All You Need. It is trained to predict the next token, then tuned to follow instructions.

• Hosted: the provider runs the weights. You send an API key. GPT, Claude, Gemini, and Groq-hosted models work this way.
• Open-weight: the numbers are published. You can download them or call a host that already did. Llama, Qwen, and gpt-oss are in this family.

A practical free path used in many public labs is openai/gpt-oss-120b on Groq, with the groq Python client. Swap the client and the model string and the rest of this page still holds.

import os
from groq import Groq

client = Groq(api_key=os.environ["GROQ_API_KEY"])
reply = client.chat.completions.create(
    model="openai/gpt-oss-120b",
    temperature=0,
    messages=[
        {"role": "system", "content": "Reply with one short sentence."},
        {"role": "user", "content": "What is a token?"},
    ],
)
print(reply.choices[0].message.content)
print(reply.usage)

Count tokens before you guess

The generation loop assigns a score to every token in the vocabulary, turns scores into probabilities, and picks one. Output tokens are usually more expensive than input tokens, and they arrive one by one, so a long answer is both the bill and the wait. Count the system prompt once and multiply by daily calls.

Starter

text = "Refund order A-1001, the parcel arrived broken."
# Rough English guess: ~4 characters per token. Count for real before you budget.
guess = max(1, len(text) // 4)
print(text)
print("rough tokens:", guess)

Billing and the context window are both in tokens. A rough English rule is 1 token ≈ 4 characters, and it is wrong often enough that you should count.

import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
text = "Refund order A-1001, the parcel arrived broken."
print(len(enc.encode(text)), enc.encode(text)[:8])

Generation is a loop: the model assigns a score to every token in its vocabulary, turns scores into probabilities, picks one, appends it, and repeats. Streaming just shows you those tokens as they are chosen.

Temperature, top-p, length, seed

Set max tokens so a rambling answer cannot eat the budget. Two runs at temperature 0.7 will not match. Judge a prompt by a score over many examples, not by one screenshot. A seed helps when you compare two prompts and want less noise.

Starter

def pick(task: str) -> dict:
    if task == "json":
        return {"temperature": 0, "max_tokens": 80}
    return {"temperature": 0.7, "max_tokens": 400}

print(pick("json"))

One task: sort a support ticket

Change one thing at a time: the instruction, then an example, then temperature. If accuracy goes up and tokens go down, keep it. If the prose sounds smarter and the score falls, revert. The same form works for a hospital inbox, an invoice extract, and a review tagged bug / feature / praise.

Starter

SYSTEM = (
    "Sort the ticket. Reply with JSON only: "
    '{"topic":"billing|shipping|other","priority":"low|medium|high"}'
)
ticket = "Charged twice for order A-1001"
print(SYSTEM)
print("USER:", ticket)

Vague instructions fail in public. This task makes the failure obvious: label a ticket as billing, shipping, or other, with a priority, as one JSON object.

SYSTEM = """You sort support tickets.
Return only JSON: {"topic":"billing|shipping|other","priority":"low|medium|high"}.
billing = payment, refund, invoice. shipping = delivery, tracking, damaged parcel.
If unsure, use other and low."""

def messages(text: str):
    return [
        {"role": "system", "content": SYSTEM},
        {"role": "user", "content": text},
    ]

Few-shot means you also show examples. Two or three is enough. Put them in the user message or as earlier user/assistant pairs.

EXAMPLES = """
Ticket: Charged twice. → {"topic":"billing","priority":"high"}
Ticket: Where is parcel A17? → {"topic":"shipping","priority":"medium"}
Ticket: Do you sell gift cards? → {"topic":"other","priority":"low"}
"""

Score the prompt, do not admire it

Keep a handful of questions you never edit the prompt against. Otherwise the bot memorizes your quiz. Exact match is enough for labels. When wording can vary and still be right, you need a looser score, which is what RAGAS-style judges are for later.

Starter

goldens = [("Charged twice", "billing"), ("Where is my parcel?", "shipping")]
preds = ["billing", "billing"]  # second one is wrong
correct = sum(p == g[1] for p, g in zip(preds, goldens))
print(f"{correct}/{len(goldens)}")

GOLDENS = [
    ("Charged twice for order A-1001", "billing", "high"),
    ("Tracking has not moved in 6 days", "shipping", "medium"),
    ("What are your store hours?", "other", "low"),
]

def exact(pred, topic, priority):
    return pred.get("topic") == topic and pred.get("priority") == priority

# correct / len(GOLDENS) is the number you track when you edit the prompt

Change one thing at a time: the instruction, then an example, then temperature. If accuracy jumps from 70% to 90% and tokens drop, keep it. If the prose “sounds smarter” and the score falls, revert.

Context, cost, and latency

Leave room in the window for the answer. Cap max_tokens. Stream when a person is watching. Batch when nobody is. If the same question arrives during an outage, cache it for 30 seconds instead of calling the model on every refresh.

Starter

def dollars(in_tok, out_tok, in_price, out_price):
    return (in_tok * in_price + out_tok * out_price) / 1_000_000

print(round(dollars(400, 30, 0.15, 0.60) * 20_000, 2), "dollars for 20k calls")

The context window must hold the system prompt, examples, the ticket, and the answer. A 128k window is not a reason to paste the wiki. Extra tokens add cost and delay, and they dilute the instruction.

# sketch: dollars ≈ (input_tokens * in_price + output_tokens * out_price) / 1_000_000
def estimate(in_tokens, out_tokens, in_price, out_price):
    return (in_tokens * in_price + out_tokens * out_price) / 1_000_000

Output tokens are often several times more expensive than input tokens, and they arrive one by one, so a long answer is both the bill and the wait. Cap max_tokens. Stream when a human is watching. Batch when nobody is.

Practice

• Count tokens in your system prompt with tiktoken or the provider’s usage field.
• Run the same ticket at temperature 0 and 1. Compare.
• Add one few-shot example that fixes a ticket your prompt gets wrong.
• Compute daily cost for 20,000 calls if each call is 400 input and 30 output tokens.