Thank you very much everybody for comingto what will hopefully be the first of many human-computer shared tasks atNAACL and affiliated workshops. We’re going to have a demonstration of aquestion answering system on a particular kind of question answeringtask called quiz bowl. Quiz bowl is a common trivia game playedamong academic institutions in English-speaking countries. Unlike traditional batch QA systemswhere you get the entire question the point of quiz bowl is to be able tointerrupt the question before it is finished and the questions arestructured in such a way that the earlier you answer the question shows your depth of knowledge. So ifyou can answer a question early, you are presumably smarter than youropponent. We ran a shared task with submissions of 11 systems (our system, QANTA, did not win). The winner was another system that we’ll be hearing about that in a second. So we had 11 systems computer systemsface off and we’re lucky enough to have the winner face off against one of the bestacademic team’s assembled in the United States and we’ll be introducing themshortly but before we introduce our human contestants, we’ll first have thewinner of the shared task system come up and speak a little bit about hiswork. My name is Ikuya Yamada and I present OUSIA:a question answering system for Quiz Bowl. so first time let me start withbackground. We are a Tokyo-based startup company working on NLP and we currently working on entity linking and a question answering. We participated in severalrecent conditions including this NAACL shared task. So maybe I can skip this. We use two models the answer the question. The first one is to be QB ranker which selectsthe most relevant answer from the answer candidates in the shared data set. So from there we narrow down the number of answer candidates. Thesecond one is Wikipedia ranker which selects an answer from popular Wikipediaentries using word matching features Because most of the questions areanswered using QB ranker, we only explain our QB ranker today so here’s the overview of our approachand in summary we combine convolutional neural net and traditional information retrieval-based measures such as BM 25 random forest. First we generatecandidates answers using top 10 results of a search engine that indexesof wikipedia articles of the answer candidates. So here are we are generated a lot ofcandidates to maintain the recall. Then we generate machine learningfeatures using these two modules and we use random forests to learn the ranking function so we assigna pointwise-relevance score to the each of candidates answers. The modelbuzzes with an answer if it passes a relevance threshold. This is our neural network model I i think this is very a simple and straighforward model. First we map words and the candidate answers into the samecontinuous vector space so embedding using skip-gram trained on wikipediatext and anchors and the question text to be encoded into a vector Qusing basic sentence CNN with max pooling and dropout. Then givin a question vector q and an answer vector a sothe probability of the answer being correct is computed using the softmax.The model is trained on dataset using ADAM optimizer As I explained earlier, we mainly use and information retrieval based features totrain the random forest for IR-based features we use on the BM25 and tf-idf basedmatching scores between the question
we use on the BM25 and tf-idf basedmatching scores between the question text and various texts such as a pastquestions and candidate answers’ Wikipedia page, paragraphs andsentences and paragraphs and sentences that contain links to the candidateanswer Wikipedia page. We also use some features and such asbinary value representing if the question text contained the candidate answer, thenumber of words, number of sentences and so on. Our system success wesaw with 64 questions and obtained first place on the result. Thank you! So now for the main event. So the way this is going to work is wewill have 30 questions In the quiz bowl community these arecalled toss-up questions and these can be interrupted at any time by a player.The human team that the winner of our shared task will face off against isthe Californian NASAT team. Each state in theUnited States gets to choose a collection of players that representsthe best academic performers from their state and we were very fortunate thatCalifornia’s team came entirely from the San Diego region so they were able to behere today. Let’s invite them up! First we haveVincent Doehr from La Jolla high school who is a graduating senior who will be going toGeorgetown next year. Next up we have Rahul Keyal a seniorat Westview High School who will be majoring in computer science orengineering. Next up we have Kion Yu from RanchoBernardo who is interested in English and political science. Finallywe have James Malouf a junior from La Jolla. Mohit will be reading thequestions and what I will do is I will advance the questions one word at a timeby pressing a button on this laptop when the system decides to answer it willanswer and the humans can do the same thing. Before we start just wantedto go over the scoring really quick. If either the humans or the computer geteach question right before a certain point in the question they get 15 points.If they get it after this mark they get 10 points and if they interruptme while I’m reading the question and they give a wrong answer they get minusfive point. Alright so with that we’ll start question 1 the Speaker of this poemwishes that he had a beaker full of the warm south so that he could leave theworld unseen. This poem compares a melody to that heard by Ruth who cried amistthe alien OUSIA: ode to a nightingale yeah that’s correct Number Two. Jean Guitton was amilitary commander of these people who were the target of a relief effort thatoccupied the Ile de Re was led by the Duke of Buckingham. The Affair of the Placards was put on by these people, who were the subject of the January Edict for 15 points Though Great Britain owed this country millions of pounds for grain exports it made during World War Two, it was able to withhold payments via the terms of the Roca-Runciman Treaty. During the Great Depression, this country experienced the Infamous Decade. The Casa (*) (*) Rosada is the traditional home of Humans: ArgentinaThat is correct for 10 points, go humans! At its premiere, the librettist of this opera portrayed a character who asks for a glass of wine with his dying wish. That character in this opera is instructed to ring some bells to summon his love. At its beginning, a man who claims to have killed a serpent has a padlock put on his mouth because of his lying. The plot of this opera Humans: Turandot?That is incorrect – 5 concerns a series of tests that Tamino must undergo to rescue Tamina from Sorastro. For 10 points, name this Wolfgang Mozart opera titled for an enchanted woodwind instrument. OUSIA: The Magic Flute That is correct Number five
That is correct Number five initiative a high-speed rail line calledthe Metroliner was built between new As part of this initiative, a high speed rail line called the Metroliner was built between New York and Washington by the newly-formed Department of Transportation. This initiative led to the creation of Community Action Agencies in low-income areas, and it OUSIA: great society that is correct for 15 points all right a man who was fermentingcommenting rebellion against this A man who was fermenting rebellion against this monarch was turned over to this king by Lupus II of Gascony. This king solidified his possessions by creating the Spanish March. This ruler perpetrated the Massacre of Verden, thus encouraging Widukind and the (*) OUSIA: Charlemagne yep that is also correct for 10 points The right side of a painting by this artist depicts the late former wife, the current wife, and the daughter of the burgermeister of Basel. That painting by this man shows Mary and the infant Jesus in front of a scallop shell. This painter of the (*) Darmstadt Madonna placed a celestial and a terrestrial globe on the top shelf of a stand OUSIA: Hans Holbein the Younger yes that is correct for 15 points Tossup 8 Primary sclerosing cholangitis primarily causes damage to this organ. An alanine aminotransferase test assesses the functionality of this organ. The Canals of Hering transport substances from this organ, and it is the site of the breakdown of (*) OUSIA: Liver yep that is 15 points as well Heinrich von Furstenberg died in combat against troops from this country, who won the Battle of Dornach to defeat the Swabian League. The Three Leagues were active in this country, and it was where the Kappel Wars were fought. Humans: Switzerlandyep that is 15 points for the humans In a scene from this novel, a drunken man drops his squalling and kicking child over the banister, but is caught by a man who could not show a blanker countenance than a miser who had lost five thousand pounds. A character in this novel overhears the woman he loves saying that it would degrade her to marry him, running away before she goes on to relate how much she loves him. Its frame story is presented to Mr. (*) Lockwood while he stays atHumans: Wuthering Heights yep that is correctThis artist used only the colors of black, yellow, and white in his painting The Deep, and those colors appear alongside shades of red and orange in his painting ironically titled Blue Poles. Many photographs of this artist at work were taken by Hans Namuth, and Humans: Jackson Pollock yep that is 15 points good job A militia in this state continually beat drums so that a royal agent could not be heard over the ruckus. A colony in this state was established by John Davenport and Theophilus Eaton. The Saybrook Colony was founded here, Humans: New Hampshirethat is incorrect to which a group of dissenters traveled under the guidance of Thomas Hooker. The (*) (*) Fundamental Orders was a governing document outlined in this state, and a copy of that charter was supposedly hidden from Edmund Andros in an oak tree. For 10 points, name this state where colonies were established at New Haven and Hartford. OUSIA: Connecticut yep that is 10 points no computer fans here John Cox, Stephen Ross and Mark Rubinstein developed a model to evaluate the cost of these entities that is based on the binomial theorem. Robert Merton won a Nobel Prize for his work on a theorem about these entities. They can be classified as (*) (*) American or European, depending on whether they can be fulfilled at any time. The Black-Scholes formula prices them. For 10 points, name these financial instruments that give the holder the choice to exercise a contract at a later date and that come in put and call varieties.Humans: Option yep that is 10 points In a story by this author, Charley is saved from a shipwreck by a Polynesian man who
match the questions yep that is 10 points In a story by this author, Charley is saved from a shipwreck by a Polynesian man who
yep that is 10 points In a story by this author, Charley is saved from a shipwreck by a Polynesian man who helps him become a captain and a plantation owner. This author of “The Heathen” wrote about a cheechako, or newcomer, who ignores the warnings of the locals by traveling to (*) Henderson Creek Humans: Joseph Conrad in weather that is seventy-five degrees below zero. This writer used the Alaskan gold rush as the setting of “To Build a Fire” and as the backdrop for a novel about the sled dog Buck. For 10 points, name this author of The Call of the Wild. OUSIA: Jack London yep that is correct This monarch agreed to give his vote for Holy Roman Emperor to Francis, the Duke of Lorraine, via the terms of the Treaty of Dresden. He ordered the construction of the Sanssouci Palace, where he often separated himself from his wife, (*) Elisabeth Christine. This king won the Battle of Mollwitz, after which he was able to seize Silesia Humans: Frederick the GreatSo we’re at the halfway point Let’s check the score.We have humans at 70 computer at 100 anybody’s game any substitutions? all right we’ll continue then … number 16 The protagonist of a novel by this author is evicted from the Bridge Inn and is talked into becoming a school janitor by a character whose role is often translated as the Council Chairman. A character created by this writer is surprised to discover that he no longer (*) (*) likes the taste of milk, but enjoys eating Humans: Kafka yep that is 10 pointsnumber 17 The Ne’ilah prayer is only said during the conclusion of this holiday. In a service Humans: Rosh Hashanahno that’s incorrect -5; right religion though during this holiday, a priest randomly selects one of two goats to be of the lord, while the author is given a red ribbon for Azazel. The evening before this holiday, a prayer called (*) (*) “All Vows,” or the Kol Nidre, is read. It occurs on the tenth day of Tishrei, and its veneration of the number five is seen in that there are five prayer services rather than the traditional three. For 10 points, name this Sabbath of Sabbaths, the Jewish Day of Atonement. OUSIA: Yom Kippur A character in this poem is told that he’s in the “animal soup of time.” Aside from a single question mark, the only punctuation marks in its second section are exclamation points. A reading from a draft of this poem was the highlight of the Six Gallery reading. Humans: Howlthat is 15 points This novel relates the story of a tortoise that convinced the birds to give it feathers so it could fly to the heavens and steal the birds’ best food and wine. A Humans: Things Fall Apartyeah that is 15 points good job all right number 20 and 11 questionsleft A condition caused by an example of this entity can be diagnosed by the presence of 14-3-3 protein in the cerebrospinal fluid. The replication of these entities can be Humans: PrionYes, and the humans take the lead ok The subject of a poem by this author had never heard of a world “where promises were kept” or where one could “weep because another wept.” This poet described an object that depicts “An artificial wilderness / And a sky like lead” in his poem “The (*) (*) Shield of AchillesHumans: W.H. Auden yeah that’s correct 10 points Along with Austin Garretson, this president averted a railroad strike by negotiating for an eight-hour workday in what would become the Adamson Act. He appointed his advisor, Louis Brandeis, to the Supreme Court, OUSIA: Woodrow Wilson that is 15 pointsA character in this Biblical book is appalled to see an altar written with the words “To the Unknown God,” and another character has a vision of a great sheet of all manner of fourfooted beasts of the earth, wild beasts, creeping things, and fowls of the air. The couple Aquila and Priscilla are first mentioned
couple Aquila and Priscilla are first mentioned Humans: Acts yep that is 15 points For these devices, Haitz’s Law was designed as a specific version of Moore’s Law. The single-avalanche photon type of these devices can be used to measure noise in hardware devices. Edison’s Law was used in the development of the first of these devices, which was constructed using a (*) (*) vacuum tube. Humans: LaserIncorrect, minus five. Today, their most common type consists of a semiconductor connected to a p-n junction. For 10 points, name these devices which only allow current to flow primarily in one direction, and that come in “light-emitting” varieties abbreviated LEDs.OUSIA: Diode yep that is 10 points all right This man sometimes names the residue theorem. A distribution usually named for this man has no defined mean or variance. The conditions for a complex function to be holomorphic are given by Humans: Cauchy yep that’s 15 points all right five questions left a closegame This man is featured in the Thomas Nast cartoon “Taking a Rest,” which puns on a marching song in its subtitle, “The Watch on” this person. He is depicted departing a ship in a political cartoon by John Tenniel titled “Dropping the Pilot.” He claimed that “the great questions of the time will not be resolved by speeches and majority decisions”, Humans: Bismark that is 15 Yeah, blood and iron yeah all right number 27 In an Orphic tradition, this god fathered a figure whose heart was consumed by the Titans before the birth of humankind, Zagreus. He disguised himself as Pluto to sleep with Persephone, leading to the birth of Melinoe, and an oracle to this deity was found at Humans: PoseidonNope, that’s minus five Dodona. Some stories claim that he was raised by a (*) (*) goat named Amalthea, after his mother swapped him for a rock so that he would not be eaten by his father, Cronus. For 10 points, name this counterpart of the Roman Jupiter, the chief god of the Greek pantheon. OUSIA: Zeus yep that’s 10 points This scientist names a constant that is equal to Loschmidt’s Constant times “RT over P” and is equal to the Faraday constant over the elementary charge. A project named for Humans: Avogadro Yeah that is 15 points. With that they havesealed the game. there’s two questions left.A story set in this country ends with the narrator Pichon being released from prison and discovering he has a son who resembles him. The stories in the collection The Burning country that imagines a Humans: Mexico yep that is also 15 points and the lastquestion In a book based on this novel, Arabella vows to wed cousin Glanville after throwing herself into the Thames; mirroring this novel’s four judges of river who force the protagonists to declare an oath. Charlotte Lennox wrote a “female” version of this novel. An attempt to recreate this novel through a line-by-line immersion Humans: Ulyssesthat is incorrect is undertaken by (*)Pierre Menard in a Jorge Luis Borge short story. The title character of this novel is the subject of pranks by the Duke and the Duchess, and imagines a peasant girl to be the legendary Dulcinea. For 10 points, name this novel by Miguel OUSIA: Don QuixoteYep that is correct and that’s the end ofthe game the job, you guys! Would the developer of thesister at system come up and shake hands with the human team? yeah yeah yeah good game Thank you very much to ourhuman team! If you think that your system could dobetter please submit to our next shared taskand hopefully we’ll do something like this again at either ACL or ICML nextyear. So this concludes our human computer question answering workshop.Thank you all very much for your participation your your great researchand your enthusiasm for the shared task. And i hope to see you again at our nextiteration where perhaps we will have seen today the last time that humansbeat a computer question answering system.Ray: Let’s thank the organizers! Thanks to He He, Mohit, and Hal Daume!