Definitions

• LLM (large language model) A neural network with (usually) billions of parameters that is trained on massive text corpora to predict the next token; once trained it can generate, translate, summarize or reason over natural‑language text at runtime  

• GPT (Generative Pre‑trained Transformer) A family of LLMs that follow a specific recipe: (1) Transformer architecture, (2) self‑supervised pre‑training on raw text, and (3) optional task‑specific fine‑tuning. The name originated with OpenAI’s GPT‑1 (2018) and is now used for successive OpenAI releases (GPT‑2, 3, 4o) and for look‑alikes such as BloombergGPT or Salesforce’s EinsteinGPT  

Learning Resources

• Neural Networks: Zero to Hero
  • 2h25m The spelled-out intro to neural networks and backpropagation: building micrograd

3Blue1Brown YouTube Channel

But what is a neural network? | Chapter 1, Deep learning

• neurons have a number representing their activation
• weights are the strength of the connection between neurons
• image recognition was explained as a grid of pixels, that are the neurons. Each pixel will assign a value, its activation, between 0 (white pixel) and 1 (black pixel). It is ‘lit up’ when its activation is high.
• layers of neurons are connected by weights. The weights are the strength of the connection between neurons. The weights are learned during training.

There was an explanation of how number recognition works. The explanation when into details of the different parts of the number start to form patterns.

For example 8 and 9, both have ‘loops’ but the 9 has a single loop at the top and the 8 has two loops.

But what is a GPT? Visual intro to transformers | Chapter 5, Deep Learning

People

• Andrej Karpathy
  • Twitter

  • The hottest new programming language is English. - Andrej Karpathy Jan 24, 2023