Large Language Models

Scale

LLMs are massive. From 2018 - 2022, model sizes have increased 5000x. OpenAI’s GPT model from June 2018 had 110M parameters; GPT-3 from May 2020 had 175B parameters. LLM providers no longer seem to advertise their parameter counts; GPT-4 was leaked to have 1.8T parameters.

LLMs as Standalone Systems

Unlike LMs that were used as components of larger systems, e.g., machine translation, LLMs are increasingly capable of being a standalone system. Recall that LMs are capable of conditional generation (given a prompt, generate a completion). This allows the same LLM to solve a variety of tasks by changing the prompt, e.g.,

What is a Language Model?

A language model (LM) is a probability distribution over sequences of tokens. Suppose we have a vocabulary \(\mathcal{V}\) of a set of tokens, then a language model \(p\) assigns each sequence of tokens \(x_1, …, x_L \in \mathcal{V} \) a probability.

To assign meaningful probabilities to all sequences requires syntactic knowledge and world knowledge. Given \( \mathcal{V} = \{ \text{ate}, \text{ball}, \text{cheese}, \text{mouse}, \text{the} \} \):

MCP 101

There are 3 key participants in the MCP architecture:

• MCP Host: The AI application that manages one or more MCP clients, e.g., Claude Code, Copilot, etc.
• MCP Client: A component that maintains a connection to an MCP server and obtains context from an MCP server for the MCP host to use.
• MCP Server: A program that provides context to MCP clients.

Chatbot Ethics

Zuckerberg: But if you think something someone is doing is bad and they think it’s really valuable, most of the time in my experience, they’re right and you’re wrong. You just haven’t come up with the framework yet for understanding why the thing they’re doing is valuable and helpful in their life.

A user’s expectations on what is permissible for a bot to do. Grabbing their attention to sell them something, or guiding them out of a slump is fine. Lying, e.g., “come visit me” and using romantic overtones crosses a line. However, Meta has no restrictions against bots telling users they’re real people or proposing real-life social engagements.

Copilot-Powered Scenarios

AI Code Completions

Design Pattern: In-Context Learning

Betting on the LLM’s context window increasing doesn’t pay off. As the input approaches the limits of the context window, inference time and accuracy degrade.

Instead, the typical workflow of in-context learning is:

• Data Pre-processing/Embedding. Compute and store embeddings of the private data in a vector database.
• Prompt Construction/Retrieval. On user input, compile a prompt from a hard-coded template with few-shot examples, information retrieved from external APIs, and a set of relevant documents retrieved from the vector database.
• Prompt Execution/Inference. Submit the compiled prompt to a pre-trained LLM for inference. Can add logging, caching, and validation at this stage.

The in-context learning reduces the AI problem into a data engineering problem.

Notable Benchmarks

Some notable benchmarks in language modeling:

• MMLU: 57 tasks spanning elementary math, US history, computer science, law, and more.
• EleutherAI Eval: Unified framework to test models via zero/few-shot settings on 200 tasks from various evals, including MMLU.
• HELM: Evaluates LLMs across domains; tasks include Q&A, information retrieval, summarization, text classification, etc.
• AlpacaEval: Measures how often a strong LLM (e.g., GPT-4) prefers the output of one model over a reference model.

tldraw.com’s Take on an LLM Canvas

In UX history, chat has tended to precede canvas. From running computers from the command prompt to using the screen as a canvas via a mouse. Mobile phones went from being keypad oriented into a canvas where the finger controls the touchscreen. Where are the canvases for LLMs?

Multi-modal models, e.g., GPT-4 and Gemini, can take image, video, and text inputs, and produce output.

Sample Use Cases