Large Language Models

What is a Language Model? A language model (LM) is a probability distribution over sequences of tokens. Suppose we have a vocabulary \(\mathcal{V}\) of a set of tokens, then a language model \(p\) assigns each sequence of tokens \(x_1, …, x_L \in \mathcal{V} \) a probability. To assign meaningful probabilities to all sequences requires syntactic knowledge and world knowledge. Given \( \mathcal{V} = \{ \text{ate}, \text{ball}, \text{cheese}, \text{mouse}, \text{the} \} \):...

Notable Benchmarks Some notable benchmarks in language modeling: MMLU: 57 tasks spanning elementary math, US history, computer science, law, and more. EleutherAI Eval: Unified framework to test models via zero/few-shot settings on 200 tasks from various evals, including MMLU. HELM: Evaluates LLMs across domains; tasks include Q&A, information retrieval, summarization, text classification, etc. AlpacaEval: Measures how often a strong LLM (e.g., GPT-4) prefers the output of one model over a reference model....

tldraw.com’s Take on an LLM Canvas is a library for creating infinite canvas experiences in React. In UX history, chat has tended to precede canvas. From running computers from the command prompt to using the screen as a canvas via a mouse. Mobile phones went from being keypad oriented into a canvas where the finger controls the touchscreen. Where are the canvases for LLMs? Multi-modal models, e....

LLMs are increasingly here to stay despite the reservations . How can I use them to enhance my capabilities? Building complete applications, e.g., a trivia-like game with Python’s Flask web server. Makes it cheap to prototype in cases where the technology behind the prototype matters much less than the content or problem being solved. As a tutor for new technologies/framework. Although React might be new to you, it’s not new to other people....

was written in a period when NLP practitioners are producing bigger (# of parameters; size of training data) language models (LMs), and pushing the top scores on benchmarks. The paper itself was controversial because it led to Gebru being fired from Google, following disagreements with her managers on conditions (withdraw, or remove Google-affiliated authors) for publishing the paper. A lot changed since mid-2021, when I initially wrote this page....