Sparse is Enough in Scaling Transformers (aka Terraformer) | ML Research Paper Explained
#scalingtransformers #terraformer #sparsity
Transformers keep pushing the state of the art in language and other domains, mainly due to their ability to scale to ever more parameters. However, this scaling has made it prohibitively expensive to run a lot of inference requests against a Transformer, both in terms of compute and memory requirements. Scaling Transformers are a new kind of architecture that leverage sparsity in the Transformer blocks to massively speed up inference, and by including additional ideas from other architectures, they create the Terraformer, which is both fast, accurate, and consumes very little memory.
OUTLINE:
0:00 - Intro & Overview
4:10 - Recap: Transformer stack
6:55 - Sparse Feedforward layer
19:20 - Sparse QKV Layer
43:55 - Terraformer architecture
55:05 - Experimental Results & Conclusion
Paper:
Code:
Abstract:
Large Transformer models yield impressive r
2 views
11
2
2 months ago 00:39:45 1
Music to Stop Doomscrolling to | Spiritual Paralysis by Hexsystem (Full Album)
2 months ago 00:01:00 1
Ireland Will Pay you $100K To Move Here
2 months ago 00:04:25 1
Deliver Us (Prince of Egypt) - Cover by Caleb Hyles and Jonathan Young
2 months ago 00:02:31 1
Ann Peebles - I Can’t Stand the Rain (Official Audio)
2 months ago 00:02:11 1
Revolt Production Music - Event Horizon [Epic Music - Powerful Beautiful Orchestral]
2 months ago 00:01:34 1
Kamala Harris phones it in with strange ’Superstar’ joke video to play at Al Smith dinner
2 months ago 00:33:48 1
Igor Stravinsky - The Rite of Spring, arr. for Piano 4 hands (1912) {Ashkenazy/Gavrilov}
2 months ago 00:27:36 1
Gustav Mahler - Symphony No. 3 in D-minor - VI, Langsam. Ruhevoll. Empfunden
2 months ago 00:04:51 1
While My Guitar Gently Weeps || The Beatles || Lyrics Video
2 months ago 03:08:26 1
Supersilent - 1-3
3 months ago 00:05:18 1
Null + Void: Where I Wait feat. Dave Gahan
2 months ago 00:22:43 1
How might LLMs store facts | Chapter 7, Deep Learning
3 months ago 00:01:37 1
[Raggastep] CoffeeMusic - Dubstep Reggae
3 months ago 00:06:31 1
Unreal 5 - Realistic Interactive Water in 5 Minutes
3 months ago 00:05:05 1
🌈 Israel Kamakawiwo’ole ➖ ’Over The Rainbow’ & ’What A Wonderful World’ Medley ➖ 1993 🌈