WK: OpenAI recently noted that the compute power required for advanced AI is doubling every 3 and a half months. Are you worried about this?You can read more at Wired.
JP: That’s a really good question. When you scale deep learning, it tends to behave better and to be able to solve a broader task in a better way. So, there's an advantage to scaling. But clearly the rate of progress is not sustainable. If you look at top experiments, each year the cost it going up 10-fold. Right now, an experiment might be in seven figures, but it’s not going to go to nine or ten figures, it’s not possible, nobody can afford that.
It means that at some point we're going to hit the wall. In many ways we already have. Not every area has reached the limit of scaling, but in most places, we're getting to a point where we really need to think in terms of optimization, in terms of cost benefit, and we really need to look at how we get most out of the compute we have. This is the world we are going into.
Facebook AI head foresees field hitting a brick wall
Posted on Thursday, Dec 05 2019 @ 13:24 CET by Thomas De Maesschalck