GPT-4 offered a marginal improvement over the capability of GPT-3. Depending on how you measure it, it would be hard to call it an exponential improvement. Let's say 30% just to put a number on things. A significant improvement.
GPT-3 cost 4.6 million dollars to train. According to a cursory Google search, GPT-4 cost 100 million dollars to train, about 50x as much. Is a 0.3x improvement worth 50x the cost?
That doesn't even touch on the costs of inference. Millions of queries a day requiring heavy duty compute infrastructure. How much capital are we going to waste on the next marginal improvement? Perhaps Microsoft will "only" need another 30% increase in their carbon emissions this year to sustain the growth.
The disconnect between benchmarks and practical use is interesting. I guess it's hard to build accurate benchmarks.
I find gpt-4 and gpt-3 a world apart. Perhaps it's just that it crossed a tipping point that made it useful for _me_ and the next 10x investment in training wouldn't achieve the same effect for me (but perhaps for somebody else?)
>Paradoxically, smaller models require more training to reach the same level of performance.
It's not a paradox at all. If less training to reach the same level of performance was true, that would be a paradox whereby they'd be trained for under a nanosecond to achieve optimal performance/size payoff.
The article mentions youtube as a source of training data, but seems to only be talking about audio transcriptions (text). But, isn't youtube more useful for multimodal training on the video data itself?
What if $7 trillion was spent eliminating economic disparities, enriching our learning institutions, fostering individual and community well-being, and ultimately drawing us away from digital technology and media as the panacea of social-control which is monopolizing our time and attention? Do we really need AGI to solve problems for us, or might the already-existing general intelligence imbued in each of us at birth be enough?
> What if $7 trillion was spent eliminating economic disparities, enriching our learning institutions, fostering individual and community well-being, and ultimately drawing us away from digital technology and media as the panacea of social-control which is monopolizing our time and attention
These are largely not problems money can solve.
Economic disparity is systemic, and changing the system would be nearly free if we knew how to change it to alleviate such disparity.
Community well-being has more to do with involvement than money.
Getting away from electronics is not only free, but will save you money. People just choose not to, and money isn't going to fix that.
Schools could use more funding though. I'm with you there.
>Economic disparity is systemic, and changing the system would be nearly free if we knew how to change it to alleviate such disparity
We already know how to do that and it has already been tried with great success. The reality though is that humans simply aren't ready for it. You can see the hypocrisy in real socialist countries. They could adopt the system and become the heroes of humanity overnight and yet they decided to oppress their citizens and await the collapse of their poorly thought out system instead. If the people claiming to fight against capitalism don't actually want to fight capitalism then expecting people who like capitalism to end it is even more hopeless.
You said changing the system would be nearly free. In theory it is. You could end capitalism with an investment of less than a billion dollars. Probably twenty million if you can get volunteers on board for ten years and even less if you could do it in a country with less expensive bank regulations. Yet I do not see such things happen. Nobody is actually trying. When you tell people the solution all I see is their inner capitalist who rejects the idea because they didn't actually care about their current life being shit, what they wanted is for the remote chance of being rich occurring to them to be worth the sacrifice of a life in poverty to keep them complacent. They would never choose a two times better life if it meant giving up becoming rich. Both the communists and capitalists are willingly harming themselves and making their lives worse and they accept that. It is utterly pointless, but here we are. There is a lever you could flick to end it, but they don't.
In a way the economists are correct. The problem is that people aren't selfish enough.
Isn't the money your talking about intended to be an investment that returns more? I don't think anyone is just giving money away without expecting to at least break even.
Well since we haven't tried spending $7t on those other things, if it works to bring about AGI we'll be stuck with it even if it doesn't really help, and if it doesn't bring about AGI we'll be all out of cash. I'd prefer the opposite strategy personally (especially since I already exist).
And the industry is seeing strong downward pressure on model size.
I don’t see this as good evidence that model parameter increases have reached a limit.
I see it as evidence that compute costs are very high.
GPT-4 offered a marginal improvement over the capability of GPT-3. Depending on how you measure it, it would be hard to call it an exponential improvement. Let's say 30% just to put a number on things. A significant improvement.
GPT-3 cost 4.6 million dollars to train. According to a cursory Google search, GPT-4 cost 100 million dollars to train, about 50x as much. Is a 0.3x improvement worth 50x the cost?
That doesn't even touch on the costs of inference. Millions of queries a day requiring heavy duty compute infrastructure. How much capital are we going to waste on the next marginal improvement? Perhaps Microsoft will "only" need another 30% increase in their carbon emissions this year to sustain the growth.
The disconnect between benchmarks and practical use is interesting. I guess it's hard to build accurate benchmarks.
I find gpt-4 and gpt-3 a world apart. Perhaps it's just that it crossed a tipping point that made it useful for _me_ and the next 10x investment in training wouldn't achieve the same effect for me (but perhaps for somebody else?)
the improvement of gpt4 over even 3.5 is significant and for practical applications these improvements exceed what is indicated by benchmarks
There are many things which earlier models maybe somewhat did, but only large models do reliably to the point of being usable for more than tech demos
Can you give examples of practical applications that have been significantly improved?
Exactly what I was thinking.
He also says
>Paradoxically, smaller models require more training to reach the same level of performance.
It's not a paradox at all. If less training to reach the same level of performance was true, that would be a paradox whereby they'd be trained for under a nanosecond to achieve optimal performance/size payoff.
The article mentions youtube as a source of training data, but seems to only be talking about audio transcriptions (text). But, isn't youtube more useful for multimodal training on the video data itself?
Is this why Sam wants $7 trillion?
What if $7 trillion was spent eliminating economic disparities, enriching our learning institutions, fostering individual and community well-being, and ultimately drawing us away from digital technology and media as the panacea of social-control which is monopolizing our time and attention? Do we really need AGI to solve problems for us, or might the already-existing general intelligence imbued in each of us at birth be enough?
> What if $7 trillion was spent eliminating economic disparities, enriching our learning institutions, fostering individual and community well-being, and ultimately drawing us away from digital technology and media as the panacea of social-control which is monopolizing our time and attention
These are largely not problems money can solve.
Economic disparity is systemic, and changing the system would be nearly free if we knew how to change it to alleviate such disparity.
Community well-being has more to do with involvement than money.
Getting away from electronics is not only free, but will save you money. People just choose not to, and money isn't going to fix that.
Schools could use more funding though. I'm with you there.
I don't know, spending a trillion dollars on housing development would probably solve a lot of problems.
Or a trillion dollars on just teachers. Not "education" but teachers themselves.
Hell the effective altruists would say giving a trillion dollars to people would be pretty effective.
>Economic disparity is systemic, and changing the system would be nearly free if we knew how to change it to alleviate such disparity
We already know how to do that and it has already been tried with great success. The reality though is that humans simply aren't ready for it. You can see the hypocrisy in real socialist countries. They could adopt the system and become the heroes of humanity overnight and yet they decided to oppress their citizens and await the collapse of their poorly thought out system instead. If the people claiming to fight against capitalism don't actually want to fight capitalism then expecting people who like capitalism to end it is even more hopeless.
You said changing the system would be nearly free. In theory it is. You could end capitalism with an investment of less than a billion dollars. Probably twenty million if you can get volunteers on board for ten years and even less if you could do it in a country with less expensive bank regulations. Yet I do not see such things happen. Nobody is actually trying. When you tell people the solution all I see is their inner capitalist who rejects the idea because they didn't actually care about their current life being shit, what they wanted is for the remote chance of being rich occurring to them to be worth the sacrifice of a life in poverty to keep them complacent. They would never choose a two times better life if it meant giving up becoming rich. Both the communists and capitalists are willingly harming themselves and making their lives worse and they accept that. It is utterly pointless, but here we are. There is a lever you could flick to end it, but they don't.
In a way the economists are correct. The problem is that people aren't selfish enough.
Isn't the money your talking about intended to be an investment that returns more? I don't think anyone is just giving money away without expecting to at least break even.
We won’t know the answer to this until we see what AGI brings.
Well since we haven't tried spending $7t on those other things, if it works to bring about AGI we'll be stuck with it even if it doesn't really help, and if it doesn't bring about AGI we'll be all out of cash. I'd prefer the opposite strategy personally (especially since I already exist).
> we'll be all out of cash
Cash is not coal. We're not burning it to get AGI. People will be getting paid to build and run the hardware.
This is a cash grab from the infrastructure players. Always been.
Do we actually think AGI will be used for public good or will it be used to further enrich the 1%?
It’s not an OR. It won’t be binary. Did the internet only enrich the 1%? Or only do public good? No it affected everyone for better and worse.
AGI will ruin the life of every modern worker
And if AGI eventually comes from OpenAI. I'm starting to doubt that.
[dead]