Convolution vs Attention

Convolution and attention are two different techniques that are used in machine learning to process and analyze data.

Convolution is a technique that is used in image processing and computer vision. It works by applying a filter to an image to extract specific features or patterns. For example, you could use a convolutional filter to detect edges in an image or to identify certain shapes.

Attention is a technique that is used in natural language processing and machine translation. It works by allowing a model to focus on specific parts of the input data when making a prediction or decision. For example, an attention mechanism might allow a machine translation model to focus on a specific word in a sentence when translating it into another language, rather than processing the entire sentence at once.

In summary, convolution is a technique that is used to process and analyze images, while attention is a technique that is used to process and analyze text or other sequential data. ___ Kien’s comment: In my opinion, this is the best ELI5 (simple) explanation possible. However, Visual Transformer (ViT) is an actively developing field that integrate the attention mechanism into imaging tasks, so it is not that clear cut in reality.