MICDS welcomed one of its own back to campus for the 2023 Harbison Lecture this week: Dr. Bill Swartout ’70. JK-12 Science Department Chair Paul Zahller introduced Dr. Swartout to the Upper School assembly:
I am honored to introduce the 2023-24 academic year’s Harbison Lecture. Mr. and Mrs. Harbison established the Harbison Lecture Fund in 1994 to commemorate Mrs. Harbison’s 50th Class Reunion. I had the opportunity to meet this year’s lecturer in advance of today’s event which stimulated my own thinking about the intersection of science, technology, and humanity. Albert Einstein, arguably one of the most influential scientists and thought leaders of all time, is credited with the quote: “It has become appallingly obvious that our technology has exceeded our humanity.” This is a valid perspective, yet I’m not certain that this has to be true. Our school’s mission compels us to look beyond the present to secure a more compassionate and stable future for all people. It is my hope that today’s lecture will inspire you to embrace that charge.
I want to welcome Dr. Bill Swartout and his guests today. Dr. Swartout, class of 70, is the director of technology at the USC Institute for Creative Technologies. His particular research interests include virtual humans, explanation and text generation, knowledge acquisition, knowledge representation, intelligent computer-based education, and the development of new AI architectures.
Dr. Swartout took the stage and jumped right in, letting students know that even though he, too, learned computer science at St. Louis Country Day School, it was a little different back then. “Don Webb, the Dean of Students, organized a club for computer science, and that’s how I got started,” he said. “Back then we had to use punch cards, which is antiquated now. It inspired me to get involved with computers and find interesting and amazing things you can do with them.”
He then gave an overview of the Institute for Creative Technologies at University of Southern California, where he is the director of technology. From virtual reality to computer graphics, AI and the learning sciences, the institute pairs computer scientists with people from the entertainment industry. “It’s technology plus the people who can create content, who create fun and interesting things.” The institute also works on the next generation of training and education using simulations.
The project he demonstrated for students was his work to preserve the ability to converse with Holocaust survivors. The institute partners with the Shoah Foundation, whose charter is to interview survivors so their stories can be shared across generations. “A highlight for many people visiting a museum is the ability to talk with a survivor,” Dr. Swartout explained. “They’re all getting a lot older, and we’re the last generation who can do that.” Is there some way to preserve that ability to have a conversation into the future? “Having a conversation is more engaging than watching an interview. You get a lot more involved,” he said. Plus, a conversation often involves stories. “Stories are one of the earliest ways of learning. Stories are highly effective in improving learning and recall.”
Dr. Swartout’s team set to answering the question, “Could we preserve the ability to have a conversation with a Holocaust survivor?” They recorded Holocaust survivors in 3D and high resolution, then used AI programming to play back answers to verbal questions.
“We worked with museums to develop a list of questions that people typically ask survivors,” he explained. “Then we interviewed 12 survivors, and recorded on video. We have a database of 1600 question-and-answer pairs, or 19 hours of footage. Then we figured out other ways of asking questions, like ‘What is your name?’ and ‘Who are you?'” They fed the question-answer pairs to a machine-learning system called NPCEditor, which creates a statistical model of what words are likely to be in the answer based on the words that are in the question.
Dr. Swartout then gave a live demonstration of an interaction with Holocaust Survivor Pinchas Gutter. He laughed as he loaded the demo, admitting, “It always feels a little bit like working without a net.” Mr. Gutter was soon on the screen, and Dr. Swartout asked him a series of questions.
Watch the entire presentation!
He then opened the forum up for students to ask their own questions. The AI system didn’t pause; Mr. Gutter’s answers were immediate and appropriate.
This system was tested in a Holocaust museum in Los Angeles, and Dr. Swartout explained that, after an interactive conversation with a young woman, she felt compelled to apologize to the character, even while knowing that there wasn’t a real person there. “We wouldn’t have gotten that if she had just watched an interview,” he said. “But because she was able to interact, she felt moved.”
Dr. Swartout stressed that the recordings haven’t been edited at all. The system delivers exactly what each survivor actually said. “If you modify, you open yourself up to Holocaust deniers,” he said.
He also explained the difference between this work and ChatGPT. “It’s a precursor to some of the work that went on with ChatGPT. The way Chat works is it takes lots of information all over the internet and constructs a statistical model that is predictive. It uses the Engram model, where one word is followed by another followed by another…if these three words appear, then the fourth word is likely this. The things it generates are maybe neat in the sense that they are not necessarily things part of the input, but the problem with using that for this kind of thing is that it makes things up that are maybe approximately what they said but not exactly what they said. Then you get back to Holocaust deniers who say this is all made up.”
What are the next steps for this technology? Dr. Swartout’s team is exploring other uses, such as recording famous people like astronauts, and having them share their experiences. It can also be used for training. A victim of sexual assault in the Army volunteered to record his experience, which is now used by the Army to train counselors of people who are victims. It gives them actual experience, and it’s scalable. Another example is to create virtual mentors, allowing young people who may not otherwise have access to “talk with” seasoned professionals in various careers.
The audience was captivated by Dr. Swartout’s work, his demonstration and his passion for using technology to make the world a better place. Thank you, Dr. Bill Swartout, for spending time with our students this week!