The transformer is a neural network architecture introduced by Google researchers in the 2017 paper 'Attention Is All You Need'. Its core idea, self-attention, lets a model look at all the tokens in its input at once and learn which ones matter for interpreting each other. GPT, Claude, Gemini, Llama and most other large language models are built on transformers, as are many embedding and reranking models used in search.
What attention does
Earlier language models processed text roughly one word at a time, which made long-range relationships hard to capture. Attention lets each token draw information from every other token in the context, weighted by relevance. In 'The invoice Quillbrook sent was late because it was misaddressed', attention helps the model connect 'it' to 'invoice' rather than 'Quillbrook'.
Transformers also parallelize well on modern hardware, which made it practical to train on very large datasets. That scaling is a large part of why LLMs became capable enough for search and assistants.
Why marketers might care
You do not need to understand the math to do GEO well, but a few practical points follow from the design. Models read context as a whole, so surrounding text changes how a passage is interpreted. Ambiguous pronouns, unclear references and passages that depend on far-away context are harder to use once a passage is lifted out on its own. Clear, explicit writing helps both people and transformers.
Search-specific language systems at Google, such as BERT, are also transformer-based. Those named ranking systems are covered under search engine algorithms.
Example
A passage that depends on its neighbors
A Rookfield help article, from a fictional scheduling company, says 'It supports this on the Pro plan.' Read in place, the meaning is clear from the heading above. Retrieved on its own, the passage is meaningless. Rewriting it as 'Rookfield supports recurring bookings on the Pro plan' makes the passage usable wherever it ends up.
Common mistakes
Assuming the model reads like a person scanning a page
Models process the context they are given all at once. What matters is whether the passage they receive is clear on its own.
Questions
Most large language models are, though researchers continue to explore alternatives and hybrids. Image and audio models use a range of designs, many also transformer-based.
Generative Pre-trained Transformer, a name OpenAI used for its model family.
