Focal Transformer: Focal Self-attention for Local-Global Interactions in Vision Transformers

❤️ Become The AI Epiphany Patreon ❤️ ► In this video I cover a new paper coming from Microsoft: “Focal Self-attention for Local-Global Interactions in Vision Transformers“ where they introduce a new transformer layer called focal attention. The main idea is to reduce the complexity but preserve the long-range dependencies. They achieve this by attending to the nearby tokens in a fine-grained manner and to the tokens that are further away they attend their coarsened representations. ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ ✅ Paper: ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ ⌚️ Timetable: