In plain language
A piece of text processed by a language model, which may be a word, part of a word or punctuation. The most useful way to understand this concept is to connect it to a real problem rather than memorising a definition.
Why it matters
Token influences how modern digital systems are designed, used or evaluated. Learners should understand its purpose, limitations and relationship to other technologies.
Simple example
Imagine a learner or organisation trying to make a digital service faster, safer or easier to use. Token may be one part of the system that helps information move, decisions happen or results become more relevant.
Questions to ask
- What problem does it solve?
- What input does it need?
- What can go wrong?
- How should a human verify the result?
Continue learning: Explore the course catalogue for a guided lesson and practical activity related to Token.