Generative AI

Machine learning, deep learning, generative AI

Most modern generators are deep-learning models. Text and code use transformers. Pictures often use diffusion: start from noise and remove it until a picture appears. A product can use both layers. A fraud model decides. A language model explains the decision in a sentence a customer can read.

Starter

task = "write a refund email"  # create  -> generative
task = "is this email spam?"     # decide -> classic ML
print("generative" if task.startswith("write") else "predictive")

These three names sit on top of each other. Mixing them up makes every later topic feel harder than it is.

Modern generative systems are mostly deep learning. Text and code models are usually transformers. Image models are often diffusion models. A large language model (LLM) is the text-and-code case: GPT, Claude, Gemini, Llama, Qwen.

An LLM does not look the answer up

A token is often a word or part of a word. “Unbelievable” may be several tokens. You pay, and you hit length limits, in tokens. The loop is the whole trick: score every possible next token, pick one, append it, continue. Streaming is that loop shown live.

Starter

text = "Binary search keeps"
# A tokenizer would split this into word pieces. This starter just shows the loop idea.
pieces = text.split()
print("so far:", pieces)
print("model would append one more piece, then repeat")

Search engines retrieve a page. An LLM predicts the next piece of text, called a token, over and over until the reply is done.

Prompt
  → Tokenization (split text into tokens)
  → Neural network
  → Predict next token
  → Predict next token
  → … until a stop
  → Final response

A token is a chunk of text, often a word or part of a word. “Unbelievable” might be several tokens. You pay, and you hit length limits, in tokens, not in characters.

A chat request is a list of messages

Three roles matter. system sets behavior. user is the request. assistant is earlier output you want the model to continue from. Put system first. A vague “help me” produces a vague reply. A role, a format, and one example produce text a program can read.

Starter

messages = [
    {"role": "system", "content": "You teach Java. One example, then the code."},
    {"role": "user", "content": "What is a for-loop?"},
]
print(messages[0]["role"], "sets the behavior")
print("send this whole list every call; the API does not remember the last one")

The model does not remember your last API call. You send the conversation you want it to see. Each message has a role and content.

{
  "model": "some-model",
  "messages": [
    {"role": "system", "content": "You are a Java programming tutor."},
    {"role": "user", "content": "Explain binary search."},
    {"role": "assistant", "content": "Binary search repeatedly divides a sorted array in half."},
    {"role": "user", "content": "Give me Java code."}
  ]
}

Put the system message first. The quality of this list changes the answer more than most people expect. A vague “help me” produces a vague reply. A role, a format, and one example produce something you can parse.

Temperature: predictable or varied

Low temperature (0 to 0.3) is for labels, JSON, and money. Medium is for normal explanations. High is for titles and brainstorms, and it also raises the chance of drifting. Many teams pin temperature at 0 whenever code will parse the answer.

Starter

def style(task):
    if task in {"invoice", "json", "sql"}:
        return 0.0
    if task == "brainstorm titles":
        return 0.9
    return 0.4

print("ticket label temperature:", style("invoice"))

Temperature controls how willing the model is to pick a less likely next token.

• Low (0 to 0.3): stable, repetitive, good for JSON, classification, banking text, ticket labels.
• Medium (0.4 to 0.7): normal chat and explanations.
• High (0.8 to 1.2): more variety, more risk of drifting, useful for brainstorming titles or marketing lines.

The context window is the model’s desk

If an early “I am coding in Java” message falls off the desk, the model does not know it. A bigger window is not a reason to paste a whole wiki: cost, delay, and noise all get worse, and private text you did not need is now in the request.

Starter

limit = 8000  # tokens, toy number
used = {"system": 400, "history": 2200, "docs": 3000, "question": 80}
left = limit - sum(used.values())
print("tokens left for the answer:", left)

The context window is the maximum number of tokens the model can read in one request: system prompt, history, retrieved documents, and the new question, plus room for the answer.

Imagine a chat that starts with “I am coding in Java.” Twenty turns later you ask “what language am I using?” The model only knows if that earlier message is still inside the window you sent. If you dropped it to save tokens, the model honestly does not know.

You cannot paste a million internal documents into the window. Cost, latency, and noise all get worse, and private text you did not need to send is now in the request.

Hallucination

Three reducers work in practice. Ask the model to abstain. Put the real document in the prompt (RAG). Validate the output: types, tests, or a second check. Generated code is untrusted input. Read it, then run it in a sandbox with no extra network and a time limit.

Starter

answer = "The return window is 90 days."  # sounds fluent
policy = "Returns are allowed for 30 days."
print("trust it?", answer.split()[-2] in policy)  # starter: do not trust without a check

A hallucination is a confident answer that is wrong, unsupported, or invented. It happens because the model is trained to continue plausible text, not to check every fact against a source.

Three practical reducers:

1. Better prompts. Ask it to say “I don’t know” and to quote the source you provided.
2. RAG. Put the real document in the prompt and tell it to use only that.
3. Validation. Check types, run tests, or have a second model score the answer.

When the model has never seen your documents

Do not send every company document. They will not fit, the bill explodes, and irrelevant pages confuse the model. Embed the question and the chunks, keep the closest chunks, and tell the model to use only those. If 80 of 100 questions retrieve the right page and 70 of those 80 answers are right, retrieval is the bigger leak.

Starter

docs = ["Returns: 30 days, unused.", "Shipping: free over $50."]
q = "return window"
hits = [d for d in docs if "return" in q or "30" in d]
print("put only this in the prompt:", hits[0])

An internal Amazon-style employee bot cannot answer from training data about this quarter’s runbooks. The design is retrieval-augmented generation (RAG):

Internal documents
  → extract text
  → chunk into passages
  → embedding model turns each chunk into a vector
  → store vectors in a vector database

User question
  → embed the question
  → find the closest chunks
  → put those chunks in the prompt
  → LLM writes the answer

Why not send every document? A company wiki will not fit the window, the bill would be huge, irrelevant pages confuse the model, and access control gets harder. RAG says: find what this question needs, then generate.

Chunk size is a tradeoff. Too small and you lose the sentence that explains the rule. Too large and you pull in noise and spend tokens. Prefer paragraph or section boundaries, overlap a little so a rule split across a boundary still appears, then measure which size retrieves better. Do not send all 10 retrieved chunks blindly: drop weak scores and leave room in the window for the prompt and the answer.

Embeddings and cosine similarity

Keyword search misses “customer-first leadership” when the page says “start with the customer and work backwards.” Vectors catch that. You store the vectors in a vector database and query with the same embedding model you used to store them. Mixing two embedding models in one index makes the geometry meaningless.

Starter

# Tiny stand-in: count overlapping words. A real embedding model replaces this.
def silly_sim(a, b):
    return len(set(a.lower().split()) & set(b.lower().split()))

q = "customer first leadership"
page = "start with the customer and work backwards"
print("overlap", silly_sim(q, page), "(real vectors would still score this as close)")

An embedding is a list of numbers that stands for meaning. Similar sentences point in a similar direction. Cosine similarity measures that angle. A smaller angle means a closer meaning. Keyword search misses “customer-first leadership” when the page says “start with the customer and work backwards.” Embeddings catch that.

from sentence_transformers import SentenceTransformer
from sentence_transformers.util import cos_sim

documents = [
    "Amazon leaders start with the customer and work backwards.",
    "Amazon employees should take ownership of their projects.",
    "Binary search works efficiently on sorted arrays.",
    "Java uses classes and objects for object-oriented programming.",
]
query = "Tell me about customer-first leadership"
model = SentenceTransformer("all-MiniLM-L6-v2")

doc_vecs = model.encode(documents)
query_vec = model.encode(query)
scores = cos_sim(query_vec, doc_vecs)[0]

for document, score in zip(documents, scores):
    print(f"{score:.4f} -> {document}")

print(documents[int(scores.argmax())])

argmax() returns the index of the largest score, not the score itself. For the top two, sort the indexes and take the first two. With several queries, cos_sim returns one row per query.

Practice

• Write a system, user, and assistant message for a tutor that only answers Java questions.
• Pick temperature for three apps: invoice classification, a story, a SQL generator. Say why.
• Explain why a 500-page PDF should be chunked before it is embedded.
• Given 100 questions, 60 good retrievals, and 55 good answers among those 60, say whether you fix search or generation first.