Galactica: A Large Language Model for Science (Drama & Paper Review)
#ai #galactica #meta
Galactica is a language model trained on a curated corpus of scientific documents, such as papers, knowledge bases, reviews, and other articles. The model can be used in a generative fasion to assist scientific writing, do reference prediction, and much more, including a new approach to do step-by-step reasoning using a clever encoding of intermediate steps. This video explains the paper, but also dives into the drama that ensued once Meta released a public demo of the model.
OUTLINE:
0:00 - Introduction
1:30 - Drama around the public demo
16:00 - Start of paper review
20:30 - Dataset construction and encoding
23:30 - Encoding step-by-step reasoning using a scratchpad
33:00 - Modelling scientific references & citations
35:05 - Prompt Pre-Training
37:10 - Architecture details
38:30 - Experimental results
49:20 - Conclusion
Paper:
Website:
Abstract:
Information overload is a major obstacle to scientific progress. The explosive growth in scientific literature and data has made it ever harder to discover useful insights in a large mass of information. Today scientific knowledge is accessed through search engines, but they are unable to organize scientific knowledge alone. In this paper we introduce Galactica: a large language model that can store, combine and reason about scientific knowledge. We train on a large scientific corpus of papers, reference material, knowledge bases and many other sources. We outperform existing models on a range of scientific tasks. On technical knowledge probes such as LaTeX equations, Galactica outperforms the latest GPT-3 by 68.2% versus 49.0%. Galactica also performs well on reasoning, outperforming Chinchilla on mathematical MMLU by 41.3% to 35.7%, and PaLM 540B on MATH with a score of 20.4% versus 8.8%. It also sets a new state-of-the-art on downstream tasks such as PubMedQA and MedMCQA dev of 77.6% and 52.9%. And despite not being trained on a general corpus, Galactica outperforms BLOOM and OPT-175B on BIG-bench. We believe these results demonstrate the potential for language models as a new interface for science. We open source the model for the benefit of the scientific community.
Authors: Ross Taylor Marcin Kardas Guillem Cucurull Thomas Scialom Anthony Hartshorn Elvis Saravia Andrew Poulton Viktor Kerkez Robert Stojnic
Links:
Homepage:
Merch:
YouTube:
Twitter:
Discord:
LinkedIn:
If you want to support me, the best thing to do is to share out the content :)
If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar:
Patreon:
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
8 views
3
0
4 weeks ago 00:43:39 1
STAR NATION NEWS 50 VF @ElenaDanaan ~ 14 Octobre 2024
4 weeks ago 00:12:37 1
Чердачный Геймер - Смурфики (Megadrive)(RUS VO)
4 weeks ago 00:18:05 1
Фронтовая жесть. Однажды в далёкой-далёкой галактике...
4 weeks ago 00:11:04 1
ТРАГЕДИЯ ГРАФА ДУКУ - ПОЧЕМУ ОН ОТВЕРНУЛСЯ ОТ ДЖЕДАЕВ? [Полная История]
4 weeks ago 00:06:18 1
Rockets - Astrolights (1978, Official Video)
4 weeks ago 00:04:16 1
Rockets - Venus Rapsody (1978, Official Video)
4 weeks ago 00:04:04 1
Rockets - Some Other Place, Some Other Time (1982, Official Video)
4 weeks ago 00:03:36 1
Rockets - Atomic (1982, Official Video)
1 month ago 00:04:54 1
Rockets - On the Road Again (1978 - Official Video)
1 month ago 00:04:13 2
Battlestar Galactica | Make Your Choice Son
1 month ago 00:04:58 1
Battlestar Galactica | Opening Scene
1 month ago 00:03:54 2
Rockets - Future Game (1982, Official Video)
1 month ago 00:52:55 1
cafe on the moon. 🌒 lofi space vibes
1 month ago 00:11:30 1
Star Wars - The Rule of Two
1 month ago 01:05:45 1
💎PARTE 3 - CONFERENCIA ASCENSIÓN PHOENIX 4 FEBRERO
1 month ago 00:02:09 1
Watchmen Chapter 2 - Exclusive Official Trailer (2024) Matthew Rhys, Katee Sackhoff, Troy Baker
1 month ago 00:00:00 1
СТРИМ ПРОХОЖДЕНИЕ STELLARIS: DLC COSMIC STORMS: Мое место в галактике #16
1 month ago 00:01:00 1
Неужели Это Сокрыто в Нашей Галактике? Млечный Путь #звезды #вселенная #фактыокосмосе #наука #космос
1 month ago 00:18:16 1
STAR NATION NEWS 49 VF @ElenaDanaan ~ 7 octobre 2024
1 month ago 00:00:28 1
Фейерверк Битва галактик, салют на 7 залпов 0.8 дюйм(а) C001
1 month ago 00:28:22 1
Парадокс Новых Галактик
1 month ago 00:28:59 1
MIEMBRO de la ÉLITE: si me VES por la CALLE parezco tan HUMANO como TU (parte 2)
1 month ago 00:20:58 1
Lo que DJIO un MIEMBRO de la ÉLITE sobre LA COSECHA (RESETEO) te dejará sin HABLA
1 month ago 00:11:29 1
«Это не из нашей Вселенной!» JWST обнаружил еще четыре странные галактики, которые старше, чем...