Beyond the internals — a practical walkthrough of how to actually use large language models in your daily work. Based on Andrej Karpathy's follow-up to his LLM deep dive.
Companion to Part 1: How LLMs Work. All content and examples traced directly to Karpathy's 2025 video.
Karpathy's mental model: ChatGPT is a "one-tab ZIP file" — a highly compressed snapshot of the internet. It read virtually every web page, book, and document up to its training cutoff, roughly 6–12 months ago. What comes back is a probabilistic recollection of that data.
The context window is its working memory — a finite tape of tokens it can see right now. Anything in it is directly accessible. Anything outside it doesn't exist for this conversation. There's no persistent memory between sessions unless you enable it.
There's no live connection to the web by default. The model produces the most statistically likely continuation of your prompt — not a lookup, not a search, not a guarantee.
ChatGPT is the "Original Gangster" — most features, most popular, most polished. But the ecosystem has exploded since 2022. Pick the right tool for the task.
OpenAI's o1, o1 Pro, o3, and o3-mini are a different breed — all model names starting with "o" are thinking models. Before returning an answer, they run an extended internal monologue: exploring approaches, backtracking, trying alternatives.
This emerged from reinforcement learning: the model discovered that deliberation strategies lead to better outcomes on hard problems. It tries different ideas, backtracks, checks its reasoning — much like the inner monologue you have when problem-solving.
Karpathy noted that Claude 3.7 Sonnet (non-thinking) solved a hard coding problem that o1 Pro could not. Model selection isn't always obvious — the right tool depends on the specific task.
By default the model runs on its weights alone — no internet, no live data. Enabling web search means it retrieves pages first, then synthesizes an answer. This costs latency but unlocks real-time information.
The key question: is the model's stale recollection good enough? For well-documented, timeless knowledge — yes. For anything time-sensitive, recent, or niche — enable search or use Perplexity.
Deep Research = extended thinking + web search, run for 5–15 minutes. The model searches dozens of sources in parallel, reasons across them, and produces a structured report — work that would take a human researcher hours.
Karpathy's demo: researching rapamycin and longevity. The model looked at 27+ sources, thought for 5 minutes, and produced a report covering mechanism of action (mTOR inhibition), worm/mouse/human trial data, safety concerns, and ongoing studies.
Both ChatGPT Deep Research (requires $200/mo Pro) and Perplexity's research mode offer this. For literature reviews, competitive analysis, and due diligence — it dramatically lowers the research bar.
Attaching documents transforms the model into a reading assistant. Upload a PDF, paste a chapter, share a spreadsheet — then ask questions, request summaries, or generate conceptual diagrams from the content.
Karpathy's example: reading The Wealth of Nations with Claude. "I'm attaching Chapter 3, Book 1 — please create a conceptual diagram of this chapter." Claude responds with Mermaid code, a diagram markup language that renders as a graph connecting key concepts.
For data, the model can write and run Python to generate charts. But treat it as a very junior data analyst — brilliant at writing the code, but it hallucinated a "1.7 trillion" figure in a chart Karpathy caught. Always scrutinize the numbers, not just the chart shape.
graph TD A[Division of Labor] -->|limited by| B[Extent of Market] B --> C[Local Market] B --> D[Trade Networks] D --> E[Water Transport] D --> F[Land Transport] E -->|lower cost| G[Coastal Cities] G -->|develop first| H[Specialization]
ChatGPT's Advanced Data Analysis wires the model to a live Python runtime. You describe a task in plain language — it writes code, runs it, and shows you the result. No copy-paste, no local setup.
This is the integration of language with computation. Arithmetic, statistics, data cleaning, chart generation — anything Python can do. Upload a CSV and ask for a trend analysis; get a matplotlib chart in seconds.
Karpathy's caution: he caught the model generating a chart with a hallucinated "1.7 trillion" instead of the correct value. The code ran fine; the number was wrong. Treat it like a very capable but unreliable junior — verify the figures, not just the output shape.
Beyond chat, a new class of tools integrates LLMs directly into your code editor. Cursor and Windsurf run Claude or GPT under the hood, operating autonomously across your entire codebase — reading files, writing code, running commands, and iterating.
Cursor's Composer (⌘I) is an autonomous agent loop: describe a task, and it plans, writes files, runs shell commands, reads errors, and loops — asking your confirmation before any destructive action. Karpathy built a React app from scratch in a few minutes.
The model under the hood in Karpathy's setup: Claude 3.7 Sonnet. The key insight is that these tools are most powerful when you understand the model well enough to guide and correct it, not just prompt and hope.
⌘K inline edit ·
⌘L chat sidebar ·
⌘I Composer (agentic)
Karpathy routes roughly half his queries through voice using Super Whisper — his pick among Super Whisper, WhisperFlow, and MacWhisper. Press a hotkey, speak, press again — query transcribed and sent. No typing, no friction.
ChatGPT's Advanced Voice Mode goes further: audio tokens flow directly to and from the model, with no text transcription layer. The result feels genuinely conversational, not a text-to-speech wrapper.
NotebookLM (Google) generates audio podcasts from your documents. Upload papers, books, or notes — it produces a two-host discussion. Karpathy uses it on walks and long drives for passive learning on topics outside his expertise.
Modern LLMs accept images as input — photos, screenshots, scans, diagrams. The model reasons about visual content as fluently as it reasons about text, drawing on training data that included billions of image-text pairs.
Karpathy's examples: uploading a blood test scan for interpretation, pointing a camera at an Aeronet 4 CO2 monitor to identify the device and interpret the 713 PPM reading, and showing a Lord of the Rings map which it correctly identified as Middle-Earth.
Vision is most reliable for well-documented subjects — blood test reference ranges, common consumer devices, famous maps — where training data covers the domain thoroughly. For proprietary or rare objects, expect more hallucination.
By default, every conversation is stateless — the model forgets everything when the tab closes. Two features change this: Memory (ChatGPT auto-saves facts about you across sessions) and Custom Instructions (a persistent system prompt shaping every response).
Karpathy's custom instructions: request educational framing ("be educational whenever you can"), set Korean language formality register for language learning, and share context about his work and interests.
Think of custom instructions as your personal system prompt — it loads before every conversation. Good instructions compress preferences you'd otherwise repeat on every query, making each session feel like it already knows you.
Every tool, model, and resource mentioned in Karpathy's lecture — linked and categorized.
Built from Andrej Karpathy's "How I use LLMs" lecture. All content, examples, and framings traced directly to that source. Interactive visualizations built with AI assistance.