Can AI Pass Top University Entrance Exams? Insights for Education and Future Work

Selected 100 classic TED talks, lasting 8-15 minutes, covering innovation, personal growth, and future trends. Provides MP3 for streaming, download, and English transcripts to improve listening and speaking skills. Ignite learning enthusiasm with the power of ideas! Below is the collection of 100 classic TED speeches for oral English listening practice. Consistent accumulation will make your English closer to real-life usage!

Today I'm going to talk about AI and us. AI researchers have always said that humans do not need to worry because only many jobs will be taken over by machines. Is that really true? They have also said that AI will create new jobs so those who lose their jobs will find a new one. Of course, but the real question is how many of those who may lose their jobs to AI will be able to land a new one, especially when AI is smart enough to learn better than most of us. Let me ask you a question. How many of you think that AI will pass the entrance examination of a top university by 2020? And many? Okay. So some of you may say that. Of course, yes. Now, singularity is the issue. And some others may say maybe because AI already wants again to top the girl player. And others may say, no, never. That means we do not know the answer yet, right? So that was the reason why I started Todai Robert Project, making an AI which passes the entrance examination of the University of Tokyo, the top university in Japan. So this is our Todai Robert. And of course, the grain of the Robert is working in the remote server. It is now writing 600 words essay, a maritime trade in the 17th century. How does it sound?

Why did I take the entrance exam as a benchmark? Because I thought we had to study the performance of AI in comparison to humans, especially of the skills and expertise which are believed to be acquired only by humans and only through education. To enter Todai, the University of Tokyo, you have to pass two different types of exams. The first one is a national standardized test in multiple choice styles. You have to take seven subjects and achieve high score, I would say like 84 percent or more accuracy rate, to be allowed to take the second stage reading test prepared by Todai. So let me first explain how modern AI works, taking job at the challenge as an example. Here is a typical job at the question, Motos lost Symphony, shares its name with this planet. Interestingly, job at the question always asks, always ends with this something. This planet, this country, this recommendation and so on. In other words, job at the question doesn't ask many different types of questions, but the single type which we call factor questions. By the way, do you know the answer? If you do not know the answer and if you want to know the answer, what will you do? You Google, right? Of course, why not? But you have to pick appropriate keywords like Motos lost and Symphony to search. Machine basically does the same. Then this Wikipedia page will be ranked top. Then machine reads the page. No, unfortunately, none of the modern AI's, including what Sun, Siri and Todai Robert is able to read. But they are very good at searching and optimizing. It will recognize that the keywords Motos lost and Symphony are appearing heavily around here. So if it can find a word which is a planet and which is co-occurring with these keywords, that must be the answer. This is ha, but some find the answer Jupiter in this case.

Our Todai Robert works similarly, but a bit smarter in answering history yes/no questions. Like, Selmey repelled the measures. Is this sentence true or false? Our Robert starts producing a factoid question like Selmey repelled this person type by itself. Then, Ava's but not Majors is ranked top. This sentence is likely to be false. Our Robert does not read, does not understand. But it is statistically correct in many cases. For the second stage written test, it is required to write 600-word essay like this one. And as I have shown earlier, our Robert took the sources from the textbooks and Wikipedia, combined them together and optimized it to produce an essay without understanding a thing. But it surprisingly wrote a better essay than most of the students. How about mathematics? Fully automatic maths solving machine has been a dream since the birth of the word artificial intelligence. But it had stayed at the level of arithmetic for a long, long time. Last year, we finally succeeded in developing a system which solved the pre-university level problems from end to end, like this one. This is original problem written in Japanese. And we have to teach it 2000 mathematical axioms in 8,000 Japanese words to make it accept the problems written in natural language.

And it is now translating the original problems into machine readable formulas. Weird, but it is now ready to solve it, I think. Go and solve it. Yes, it is now executing symbolic computation. Even more weird, but probably this is the most fun part for the machine. Robert now outputs the perfect answer. Though its proof is impossible to read even for mathematicians. Anyway, last year, Robert was a man, top 1%, in the second stage written exam in mathematics. Thank you. So, did it enter Todai? No, not as I expected. Why? Because it doesn't understand any meaning. Let me show you a typical error it made in English test. Two people are talking. For us, who can understand the situation, it is obvious number 4 is the correct answer. But Todai Robert chose number 2. Even after learning 15 billion English sentences using deep learning technologies. Okay, so now you might understand what I said. Modern AI do not read, do not understand. They only disguise as if they do.

This is the distribution graph of half million students who took the same exam as Todai Robert. Now, our Todai Robert is a man top 20%. And it was capable to pass more than 60% of the universities in Japan, but not Todai. But see how it is beyond the volume zone of two big white color workers. You might think I was delighted. After all, Todai Robert was surpassing students everywhere. Instead, I was alarmed. How on earth could this intelligent machine outperform students? Our children, right? I decided to investigate what was going on in the human world. I took hundreds of sentences from the high school textbooks and made easy multiple choice quizzes and asked thousands of high school students to answer. Here is an example. Of course, the original problems are written in Japanese, their mother tongue. Obviously, Christianity is the answer, isn't it? It's written. And Todai Robert chose to crack down the two. But one third of junior high school students failed to answer this question.

Do you think it is only the case in Japan? I do not think so. Because Japan is always ranked among the top in OECD Pisa tests measuring 15-year-old students performance in mathematics, science and reading every three years. We have been believing that everybody can learn and learn well as long as we provide good learning materials free on the web so that they can access through the internet. But such wonderful materials may benefit only those who can read well, and the percentage of those who can read well may be much less than we expected. How we humans will exist with AI is something we have to think about carefully based on solid evidence. At the same time, we have to think in a hurry because time is running out. Thank you. Thank you. In your talk, you so beautifully give us a sense of how AI thinks, what they can do amazingly and what they can't do. But do I read you right that we really need quite an urgent revolution in education to help kids do the things that humans can do better than AI? Yes, yes. Because we humans can't understand the meaning. That is something which is very, very lacking in AI. But most of the students just pack the knowledge without understanding the meaning of the knowledge. So that is not knowledge, that is just memorizing and AI can do the same thing. So we have to think about new type of education. A shift from knowledge, rote knowledge to meaning. Well, there's a challenge for the educators here. Thank you very much. Thank you.