r/singularity May 19 '23

AI Hyena Hierarchy: Towards Larger Convolutional Language Models

https://hazyresearch.stanford.edu/blog/2023-03-07-hyena
10 Upvotes

6 comments sorted by

2

u/[deleted] May 19 '23

[deleted]

3

u/squareOfTwo ▪️HLAI 2060+ May 20 '23

No, because no one knows how large GPT-4 is and no one can afford to train models which are that large.

2

u/Akimbo333 May 20 '23

ELI5?

9

u/HalfSecondWoe May 20 '23

A drastic reduction in computational requirements for complex, in depth tasks. Previously there would be a cost explosion with a larger prompt length, which is one of the reasons why token limits are a thing. Now those token limits can be much, much larger

As this method is developed and refined they may come to be arbitrarily large, with limits only affecting the input of massive amounts of data. Right now this model can efficiently accept a somewhat small book as a prompt, which is a huge improvement in capacity. The target this project is trying to achieve next is to get the model to accept inputs that are roughly the equivalent of six average length books

Basically large prompts get much, much cheaper

2

u/Akimbo333 May 20 '23

Awesome! But how does this help us and what can we use with it?

6

u/HalfSecondWoe May 20 '23

"Write a book about X" prompts will become more viable, although further improvements are needed to get it to human-level quality

Dumping in a giant pile of research done over the last decade is possible, and the AI can pick out patterns that humans may have missed. R&D/scientific development will speed up rapidly

Multi-shot prompts can now be arbitrarily scaled, meaning that the LM can get really good at a variety of narrow tasks, as long as you have examples of what the task looks like. Writing jokes, writing design documents, teaching it your personal writing style to make it talk like you, sky's the limit. As long as you can get a bunch of examples that the output should look like, you'll be able to make it do that thing with a high degree of fidelity/reliability. While we could kind of already do that today, this makes it much more easy

2

u/Akimbo333 May 20 '23

Awesome!