What does Transformer mean?

Unlike predecessors like RNNs or LSTMs, the Transformer relies entirely on attention mechanisms to weigh the importance of different parts of the input sequence, regardless of distance. This enables highly parallel computation during training. Core components include encoder and decoder stacks consisting of multiple identical layers, each containing multi-head self-attention and position-wise feed-forward networks. Positional encoding is added to embed sequence order information.

Transformers have become the dominant architecture for numerous NLP tasks, including machine translation, text summarization, and question answering (e.g., models like BERT and GPT are based on Transformers). Their effectiveness stems from capturing complex long-range dependencies efficiently. Their application extends to computer vision, audio processing, and multimodal tasks, driving significant advancements in AI capabilities.