| Random Link ¯\_(ツ)_/¯ | ||
| Dec 14, 2023 | » | Introduction to LLMs
4 min; updated Dec 17, 2023
What is a Language Model? A language model (LM) is a probability distribution over sequences of tokens. Suppose we have a vocabulary \(\mathcal{V}\) of a set of tokens, then a language model \(p\) assigns each sequence of tokens \(x_1, …, x_L \in \mathcal{V} \) a probability. To assign meaningful probabilities to all sequences requires syntactic knowledge and world knowledge. Given \( \mathcal{V} = \{ \text{ate}, \text{ball}, \text{cheese}, \text{mouse}, \text{the} \} \):... |
| Mar 3, 2021 | » | LLMs: Stochastic Parrots 🦜 and How (Not) to Use Them
10 min; updated Dec 14, 2023
was written in a period when NLP practitioners are producing bigger (# of parameters; size of training data) language models (LMs), and pushing the top scores on benchmarks. The paper itself was controversial because it led to Gebru being fired from Google, following disagreements with her managers on conditions (withdraw, or remove Google-affiliated authors) for publishing the paper. A lot changed since mid-2021, when I initially wrote this page.... |