ARTIFICIAL GENERAL INTELLIGENCE
GPT-3: An Artificial General Intelligence (AGI) or just a Strong AI
There always comes a better one after the best and a bigger one after the biggest. This is proved again by the OpenAI’s latest revolutionary general-purpose language model — GPT-3. Just to give a background, language models are the AI algorithms that understand the natural human language and respond accordingly. But the one which we are going to talk about does not only responds but also creates, builds, answers, and summarizes intelligently like a human (better sometimes) after getting input from the user. Is this the next level Artificial General Intelligence (AGI) that people were talking about? Or is it just a strong pre-existing AI?
Just a couple of weeks back, I talked about a language model with a few million parameters from talk of Andrez Karpathy in Germany, and I was thinking that this is a revolution. And now we get this. Not long ago I was talking that AGI is not a thing in near few decades. But now I am hesitating a little. Let me first give you some instances of tasks that GPT can perform. This will help to put things in perspective and get a context of what I am talking about.
This time open AI is offering it as API and that is why we have so many cool apps build around it. Recently I made three posts (first, second, and third) about this beast but I thought, it is worth more than that and hence this writing.
To put things even more in perspective, consider this; a human brain has approximately a hundred billion neurons, which approximately makes about 100 to 500 trillions connections. If we think that increasing the number of hidden layers and the number of parameters is the solution for human-like intelligence, then we are still 1000 leaps behind human intelligence. However, even being 1000x behind human intelligence is a big leap (I believe) and close if we consider the speed of scale-up in the past few decades as evident from the figure below.
The model was trained using 175 billion parameters with below mixture of data.
Above is the mixture of data that was used for training. It is not that the model was simply trained on all data at once blindly without any specific tweaking. The trainers thought about the high-level tasks that humans do in daily life and trained the system for reading comprehension, arithmetic, translation, common sense reasoning, and general Q&A. It is these capabilities when combined together give an effect of an overall intelligent system. These pieces of training are specifically listed in the paper if you really want to get into the depth. Datasets of this size can be trained using large batch sizes but the learning rate has to be small otherwise the training will keep overshooting the global minima which would be resulting in unnecessary time consumption and re-work. It was trained on Nvidia’s v100 GPU provided by Microsoft for the training. Below is the representation of loss over a period of time while increasing the number of parameters.
Analysts estimate that it would have cost almost 14 million dollars for OpenAI to train.
Other interesting things it can do!
The interesting part is that unlike any other machine learning model, GPT-3 is not specific to some particular task or work. This was trained as a general language model and we know a language can contain anything — from a poem to an article and from coding language to questions and answers.
Though the model is just following an algorithm and does not have an understanding or sense of what it is saying and doing unlike us (humans, who understand what we are doing), it can still prove to be a great assistant or maybe the best one.
People are talking …
Some people will still object to the capabilities of GPT-3 and say that this is not equivalent as compared to humans. But when we consider ourselves, even ‘we’ are not great in all tasks. Even though I can speak better Hindi and do better poetry in Hindi than GPT-3, I cannot translate even a word from Hindi to French. Moreover, I cannot create UI blocks as quickly as done by GPT-3. In that manner, this is still a great advancement and a thing to be proud of.
I am really excited about the advancement opportunities that this model has opened. The recent partnership of Azure and Open AI in return for Microsoft’s new and most powerful supercomputer access to open AI will bring some groundbreaking results. All our Alexa, Goggle Home, and Siri will look childish in front of it and who knows after this partnership, Microsoft Cortana gains momentum suddenly. The possibilities can be guessed by a tweet from the CEO of OpenAI, that this is just a start.
The GPT-3 hype is way too much. It’s impressive but it still has serious weaknesses and sometimes makes very silly mistakes. AI is going to change the world, but GPT-3 is just a very early glimpse. We have a lot still to figure out.
Sam Altman (@sama) July 19, 2020
I am thinking …
Think about the translation of imagination that it will bring. ‘Translation if Imagination’ is a term that I just coined while creating this content:) Maybe after the commercialization of this or the next version of this tech, a novice like myself can talk to the model to create some innovative designs that someone is imagining in mind. For example, consider this case when the user asked the system to create a button that looks like a watermelon. The model came up with its code and the button that looked like the one in the image.
Moreover, many-a-times it happens with researchers that they are just not able to recall what to do next. What if the model is provided the data and it tells the researchers about the next step from the context of what has happened in the past. A researcher can directly ask it for the next step something like Iron man asking Jarvis in the famous Marvel’s movie series — Iron Man.
Even though this kind of intelligent assistant is about decades away, we can endlessly talk about the applications of such technology. Its possible usage is only limited to our imagination.