188宝金博页面版

Towards Sensing the Inf l uence ofVisual Narratives on Human AffectMihai Burzo ∗Mechanical and Energy EngineeringUniversity of North TexasMihai.Burzo@unt.eduDaniel McDuffMedia LabMassachusetts Institute of Technologydjmcduff@mit.eduRada MihalceaComputer Science and EngineeringUniversity of North Texasrada@cs.unt.eduLouis-Philippe MorencyInstitute for Creative TechnologiesUniversity of Southern Californiamorency@ict.usc.eduAlexis NarvaezMechanical and Energy EngineeringUniversity of North TexasAlexisNarvaez@my.unt.eduVerónica Pérez-RosasComputer Science and EngineeringUniversity of North Texasveronica.perezrosas@gmail.comABSTRACTIn this paper, we explore a multimodal approach to sensingaf f ective state during exposure to visual narratives. Usingfour dif f erent modalities, consisting of visual facial behav-iors, thermal imaging, heart rate measurements, and verbaldescriptions, we show that we can ef f ectively predict changesin human af f ect. Our experiments show that these modal-ities complement each other, and illustrate the role playedby each of the four modalities in detecting human af f ect.Categories and Subject DescriptorsI.2.7 [Artif i cial Intelligence]: Natural Language Process-ing—DiscourseGeneral TermsAlgorithms, ExperimentationKeywordsMultimodal signal processing, Multimodal sensing, Af f ectiveBehavior1. INTRODUCTIONNarratives are a constant presence in our everyday lives,and can have signif i cant inf l uence on one’s thoughts and ac-tions. Narratives are often designed to explicitly appeal to∗ The order of the authors is alphabetical, as all the authorshave equally contributed to this work.Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for prof i t or commercial advantage and that copiesbear this notice and the full citation on the f i rst page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specif i cpermission and/or a fee.ICMI’12, October 22–26, 2012, Santa Monica, California, USA.Copyright 2012 ACM 978-1-4503-1467-1/12/10 ...$15.00.the emotions of the reader or listener, and act as an “emo-tional prime” [11, 16]. Once an af f ective state has beeninduced, it can also lead to changes in cognition and action,in agreement with the large body of previous research onemotions [15].A specif i c type of narrative that is becoming extremelypopular with the Internet age is the visual narrative. Withmore than 10,000 new videos posted online every day, socialwebsites such as YouTube and Facebook are an almost in-f i nite source of visual narrative. People are posting videosto express their opinion and sentiment about dif f erent top-ics, products and events. To better understand how theseonline videos are inf l uencing individuals and eventually thesociety at large, it is imperative that we develop automatictechniques to analyze human reactions to visual narrative.In this paper, we propose a non-invasive multimodal ap-proach to sense and interpret human reaction while watch-ing online videos. This is a f i rst important milestone to-ward a deeper understanding of visual narrative inf l uenceon human af f ective states. Our approach senses changes inhuman af f ect through four dif f erent modalities: visual fa-cial behaviors, physiological measurements, thermal imag-ing, and verbal descriptions. The f i rst three modalities arerecorded live during the narrative interaction while the ver-bal descriptions are acquired during a post-study interview.We evaluate our multimodal approach on a new corpus of70 narrative interactions. Figure 1 shows the overall f l ow ofour approach.The following section summarizes related work in visual,physiological, and linguistic analysis of human af f ective state.Section 3 presents our experimental methodology to createthis new visual narrative corpus. Section 4 presents a de-tailed description of the multimodal features automaticallyextracted. Section 5 presents experimental results compar-ing the performance of our multimodal predictive models.Section 6 discusses our results and shows an analysis of themultimodal features and their ef f ectiveness to predict hu-man af f ective state. Section 7 presents our conclusions andfuture directions.153