Map of Contents. Learning path from classical text processing to Transformers and LLMs. Neural network basics live in Machine Learning MOC; agent systems built on LLMs live in Agents MOC.
Foundations
- Glossary (NLP) - Key NLP terms and definitions.
- What is NLP? - NLP is a field of linguistics and machine learning focused on understanding human language.
- Language Modeling & N-grams - Predicting the next word; statistical language models.
- Tokenization - Character, word, and subword tokenization schemes.
Representations
- Word Vectors - Knowledge-based vs. distributional word representations.
- Word2Vec - CBOW and Skip-gram, negative sampling, subsampling, analogies, and limitations.
- Building Embedding Models From Scratch - Word2Vec, autoencoder bottleneck, and Siamese networks, with a PyTorch Skip-Gram snippet.
- Keyword Extraction & Topic Modelling - TF, TF-IDF, RAKE, and topic models.
Neural Sequence Models
- RNNs & CNNs for Text Classification - Improvements over n-grams and remaining limitations.
- RNNs & LSTMs - RNN/LSTM architecture, parameter counting, text generation, and Seq2Seq machine translation.
- Attention & Transformers - Self-attention, multi-head attention, the Transformer encoder/decoder, and BERT pre-training.
Large Language Models
- Large Language Models - Characteristics, scale, and training of LLMs.
- KV Cache - Storing Key and Value vectors of previous tokens to speed up generation.
- LM Benchmarks - Coding, reasoning, and general benchmarks for language models.