First LLM Integration

From zero to a working API call — keys, prompts, streaming, and safe defaults.

Interview tip Walk through: pick provider → env vars for keys → one sync call → add streaming → add retries + timeouts.

① What you must know (30 sec)

Your first LLM integration is a backend API call to a provider (OpenAI, Anthropic, Google) with a structured prompt and safe defaults. Master the basics — keys in env vars, system + user messages, parsing the response, handling errors — before adding RAG, agents, or fine-tuning. Always proxy through your server to protect credentials and enforce rate limits.

Analogy: Like adding a payment gateway: start with one successful charge, then add webhooks, retries, and monitoring — not a custom bank on day one.

② How it works

Request flow

Frontend / client→Your API (auth, rate limit)→LLM provider→Stream or JSON response

Client sends user input to your endpoint — never the raw API key
Your server validates input, applies auth, checks quotas
Build messages array: system (role) + user (content) + optional history
Call provider SDK with model, temperature, max_tokens, timeout
Return text to client; log tokens, latency, and errors

Use a fast/cheap model (gpt-4o-mini, Claude Haiku, Gemini Flash) for development; swap to smarter models only where quality demands it.

③ Step-by-step (hands-on)

Step 1 — Create provider account and key

Sign up, generate API key, set billing alerts. Store key in .env locally and in your secrets manager (AWS Secrets Manager, Vault) in production.

Step 2 — Install SDK and hello-world

pip install openai or npm @anthropic-ai/sdk. One function: send "Hello" → print response. Confirm key works before building UI.

Step 3 — Add system prompt

Define persona, output format, and boundaries: "You are a support bot. Answer in 3 bullets. Do not invent refund policies."

Step 4 — Expose via your API

POST /api/chat with body { message }. Validate length, sanitize input, attach user ID for rate limiting. Return { reply, usage }.

Step 5 — Add streaming (optional)

Use SSE or WebSocket so tokens appear incrementally. Flush chunks to client; handle client disconnect to cancel upstream.

Step 6 — Harden for production

Retries with backoff on 429/5xx, 30–60s timeout, structured logging, PII scrubbing in logs, and a circuit breaker or fallback model.

④ Code / config patterns

Concern	Dev default	Production
Model	gpt-4o-mini / Haiku	Tier by task complexity
Temperature	0–0.3 for facts	0 for deterministic; higher for creative
Max tokens	512–1024	Cap to control cost
Retries	None	3 attempts, exponential backoff
Timeout	60s	30s with cancel on client disconnect

// Node.js — minimal chat endpoint
const client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
const res = await client.chat.completions.create({
  model: "gpt-4o-mini",
  messages: [
    { role: "system", content: "You are a helpful assistant." },
    { role: "user", content: userMessage }
  ],
  max_tokens: 512,
  temperature: 0.2
});
return res.choices[0].message.content;

⑤ Production & pitfalls

Pitfall	Why it hurts	Fix
API key in frontend	Key stolen, unlimited spend	Always call LLM from backend; use session auth
No timeout	Hung requests pile up	Set client timeout; cancel on disconnect
Ignoring token usage	Surprise bills	Log prompt + completion tokens per request
No input validation	Prompt injection, huge payloads	Max length, block patterns, rate limit per user
Blind retries on 400	Amplified errors and cost	Retry only 429/5xx with backoff
Logging raw PII	Compliance violations	Redact emails, SSNs before logging

Production tips:

Feature flag to disable LLM calls without deploy
Per-user and global rate limits
Cost dashboard by endpoint and model
Content moderation layer for user-generated prompts

⑥ Interview / on-the-job Q&A

Question	Answer
Why backend-only?	Protects API keys, enables rate limits, audit logs, and prompt templates you control.
What is a system prompt?	Instructions to the model defining role, tone, format, and constraints — not shown as user input.
Streaming vs non-streaming?	Streaming improves perceived latency for chat UIs; batch is fine for background jobs.
What is temperature?	Randomness knob: 0 = deterministic, higher = more creative variation.
How handle 429?	Exponential backoff, queue requests, or route to fallback model/provider.
What to log?	Latency, model, token counts, error codes — not full prompts if they contain secrets.

⑦ Tools & ecosystem

SDKs: openai (Python/Node), anthropic, google-generativeai
Proxies: LiteLLM, Portkey — unified API across providers
Observability: LangSmith, Helicone, OpenTelemetry spans
Secrets: dotenv (dev), AWS/GCP secret managers (prod)

apiopenaistreamingsecurityobservability

⑧ Revision checklist

API key in environment variable, never committed
Hello-world call works from backend script
System prompt defines role and output format
Endpoint behind authentication
Input length and rate limits enforced
Timeout and retry policy documented
Token usage logged per request
Error responses user-friendly without leaking internals
Billing alerts configured on provider dashboard