Galactica: A Large Language Model for Science (Drama & Paper Review)
#ai #galactica #meta
Galactica is a language model trained on a curated corpus of scientific documents, such as papers, knowledge bases, reviews, and other articles. The model can be used in a generative fasion to assist scientific writing, do reference prediction, and much more, including a new approach to do step-by-step reasoning using a clever encoding of intermediate steps. This video explains the paper, but also dives into the drama that ensued once Meta released a public demo of the model.
OUTLINE:
0:00 - Introduction
1:30 - Drama around the public demo
16:00 - Start of paper review
20:30 - Dataset construction and encoding
23:30 - Encoding step-by-step reasoning using a scratchpad
33:00 - Modelling scientific references & citations
35:05 - Prompt Pre-Training
37:10 - Architecture details
38:30 - Experimental results
49:20 - Conclusion
Paper:
Website:
Abstract:
Information overload is a major obstacle to scientific progress. The explosive growth in scientific literature and data has made it ever harder to discover useful insights in a large mass of information. Today scientific knowledge is accessed through search engines, but they are unable to organize scientific knowledge alone. In this paper we introduce Galactica: a large language model that can store, combine and reason about scientific knowledge. We train on a large scientific corpus of papers, reference material, knowledge bases and many other sources. We outperform existing models on a range of scientific tasks. On technical knowledge probes such as LaTeX equations, Galactica outperforms the latest GPT-3 by 68.2% versus 49.0%. Galactica also performs well on reasoning, outperforming Chinchilla on mathematical MMLU by 41.3% to 35.7%, and PaLM 540B on MATH with a score of 20.4% versus 8.8%. It also sets a new state-of-the-art on downstream tasks such as PubMedQA and MedMCQA dev of 77.6% and 52.9%. And despite not being trained on a general corpus, Galactica outperforms BLOOM and OPT-175B on BIG-bench. We believe these results demonstrate the potential for language models as a new interface for science. We open source the model for the benefit of the scientific community.
Authors: Ross Taylor Marcin Kardas Guillem Cucurull Thomas Scialom Anthony Hartshorn Elvis Saravia Andrew Poulton Viktor Kerkez Robert Stojnic
Links:
Homepage:
Merch:
YouTube:
Twitter:
Discord:
LinkedIn:
If you want to support me, the best thing to do is to share out the content :)
If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar:
Patreon:
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
8 views
3
0
2 months ago 00:21:34 1
STAR NATION NEWS 47 @ElenaDanaan VF ~ 16 septembre 2024
2 months ago 08:42:27 1
Филип Дик - “Целитель Галактики“ | АУДИОКНИГА полностью
2 months ago 00:00:28 1
Фейерверк Битва галактик, салют на 7 залпов 0.8 дюйм(а) C001
2 months ago 00:02:02 1
Один в поле Воин ULTRA FUNK
2 months ago 00:00:00 1
Sins of a Solar Empire II - Завоевание Галактики - Серия №11
2 months ago 00:04:16 2
Иван Юрчев - Заветы Мандалора (Звёздные войны. Истории)
2 months ago 00:40:44 1
Вейдер Бессмертный Эпизод 1. [Игрофильм] | ТВ ЗВ
2 months ago 00:08:15 1
ДАРТ ВЕЙДЕР ПРИСОЕДИНЯЕТСЯ К ПОВСТАНЦАМ И МЕНЯЕТ ЦВЕТ КОСТЮМА! ЗВЁЗДНЫЕ ВОЙНЫ
2 months ago 00:00:00 2
Sins of a Solar Empire II - Максимально сложное завоевание Галактики - Серия №10
2 months ago 00:46:29 1
Sam Esmail | Leave the World Behind | Talks at Google
2 months ago 00:57:31 1
[BadComedian] - Вратарь галактики (НАШ ОТВЕТ Стражам и Мстителям за 1 млрд. рублей)
2 months ago 00:00:00 1
Sins of a Solar Empire II - Максимально сложное завоевание Галактики - Серия №9
2 months ago 00:24:04 1
STAR NATION NEWS 46VF @ElenaDanaan ~ 09 septembre2024
2 months ago 00:20:54 1
Петька и Василий Иванович спасают галактику (но есть нюанс)
2 months ago 00:00:00 2
Sins of a Solar Empire II - Максимально сложное завоевание Галактики - Серия №8
2 months ago 00:04:35 1
Rockets - Galactica (1980, Official Video - Version 2)
2 months ago 00:06:54 1
Интервью с Джорджем Лукасом. 7,8 и 9 эпизоды Звёздных войн
2 months ago 00:09:29 1
Интервью с Энтони Дэниелсом, исполнителем роли C-3PO [Comic Con Russia 2015]
2 months ago 00:03:12 1
Интервью с кастом “Мандалорец“ и “Звёздные Войны: Скайуокер. Восход“ | D23 Expo 2019
2 months ago 00:00:00 1
Sins of a Solar Empire II - Максимально сложное завоевание Галактики - Серия №7
2 months ago 00:04:38 1
Дарт Вейдер становится Джедаем и убивает Императора (Канон) | Звёздные Войны
2 months ago 00:04:37 1
Как НА САМОМ ДЕЛЕ выглядел Вейдер с точки зрения СИЛЫ? | Звездные Войны
2 months ago 00:06:50 1
Дарт Вейдер В ОДИНОЧКУ сражается с Флотом и Танками Армии Повстанцев (Канон) | Звездные Войны
2 months ago 00:08:35 1
Как Дарт Вейдер снова нашел свой СИНИЙ МЕЧ (Канон) | Звёздные Войны