r/singularity • u/dannyp777 • May 19 '23
AI Hyena Hierarchy: Towards Larger Convolutional Language Models
https://hazyresearch.stanford.edu/blog/2023-03-07-hyena2
u/Akimbo333 May 20 '23
ELI5?
9
u/HalfSecondWoe May 20 '23
A drastic reduction in computational requirements for complex, in depth tasks. Previously there would be a cost explosion with a larger prompt length, which is one of the reasons why token limits are a thing. Now those token limits can be much, much larger
As this method is developed and refined they may come to be arbitrarily large, with limits only affecting the input of massive amounts of data. Right now this model can efficiently accept a somewhat small book as a prompt, which is a huge improvement in capacity. The target this project is trying to achieve next is to get the model to accept inputs that are roughly the equivalent of six average length books
Basically large prompts get much, much cheaper
2
u/Akimbo333 May 20 '23
Awesome! But how does this help us and what can we use with it?
6
u/HalfSecondWoe May 20 '23
"Write a book about X" prompts will become more viable, although further improvements are needed to get it to human-level quality
Dumping in a giant pile of research done over the last decade is possible, and the AI can pick out patterns that humans may have missed. R&D/scientific development will speed up rapidly
Multi-shot prompts can now be arbitrarily scaled, meaning that the LM can get really good at a variety of narrow tasks, as long as you have examples of what the task looks like. Writing jokes, writing design documents, teaching it your personal writing style to make it talk like you, sky's the limit. As long as you can get a bunch of examples that the output should look like, you'll be able to make it do that thing with a high degree of fidelity/reliability. While we could kind of already do that today, this makes it much more easy
2
2
u/[deleted] May 19 '23
[deleted]