Exploring Machine Learning Methods to AutomaticallyIdentify Students in Need of AssistanceAlireza Ahadi and Raymond ListerUniversity of Technology, SydneyAustraliaalireza.ahadi@uts.edu.auraymond.lister@uts.edu.auHeikki Haapala and Arto VihavainenDepartment of Computer ScienceUniversity of HelsinkiFinlandheikki.haapala@cs.helsinki.f iarto.vihavainen@cs.helsinki.f iABSTRACTMethods for automatically identifying students in need ofassistance have been studied for decades. Initially, the workwas based on somewhat static factors such as students’ edu-cational background and results from various questionnaires,while more recently, constantly accumulating data such asprogress with course assignments and behavior in lectures hasgained attention. We contribute to this work with results onearly detection of students in need of assistance, and providea starting point for using machine learning techniques onnaturally accumulating programming process data.When combining source code snapshot data that is recordedfrom students’ programming process with machine learningmethods, we are able to detect high- and low-performingstudents with high accuracy already after the very f i rst weekof an introductory programming course. Comparison of ourresults to the prominent methods for predicting students’performance using source code snapshot data is also provided.This early information on students’ performance is benef i -cial from multiple viewpoints. Instructors can target theirguidance to struggling students early on, and provide morechallenging assignments for high-performing students. More-over, students that perform poorly in the introductory pro-gramming course, but who nevertheless pass, can be moni-tored more closely in their future studies.Categories and Subject DescriptorsK.3.2 [ Computer and Information Science Education ]:Computer science education; H.2.8 [ Database Applica-tions]: Data miningKeywordsintroductory programming; source code snapshot analysis;programming behavior; educational data mining; learninganalytics; novice programmers; detecting students in need ofassistancePermission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor prof i t or commercial advantage and that copies bear this notice and the full cita-tion on the f i rst page. Copyrights for components of this work owned by others thanACM must be honored. Abstracting with credit is permitted. To copy otherwise, or re-publish, to post on servers or to redistribute to lists, requires prior specif i c permissionand/or a fee. Request permissions from Permissions@acm.org.ICER ’15 August 9 – 13, 2015, Omaha, Nebraska, USACopyright 2015 ACM 978-1-4503-3630-7/15/08 ...$15.00.DOI: http://dx.doi.org/10.1145/2787622.2787717.1. INTRODUCTIONEvery year, tens of thousands of students fail introductoryprogramming courses world-wide, and numerous studentspass their courses with substandard knowledge. As a con-sequence, studies are retaken and postponed, careers arereconsidered, and substantial capital is invested into studentcounseling and support. World-wide, on average one thirdof students fail their introductory programming course [4,40]. Even when looking at statistics describing pass ratesafter teaching interventions, as many as one quarter of thestudents still fail the courses [38].One of the challenges in organizing teaching interventionsis that any change is likely to also af f ect students for whomthe prevalent situation is more suitable. For example, ifa student is already at a stage where she could work onmore challenging projects on her own, mandatory excessivelystructured learning activities that everyone needs to followmay even be counterproductive for her [16, 31]. To provideanother example, while collaborative learning practices suchas pair programming [45] have been highlighted as ef f i cientteaching approaches for introductory programming [23, 38],there are contexts in which students mostly work from adistance and rarely attend an institution.This diversity of institutions, students, and teaching ap-proaches is the setting upon which our work builds. Webelieve that the appropriate next step in teaching interven-tions is the transition towards interventions that addressonly those students that are in need of guidance, and worktowards that goal by analyzing methods for detecting suchstudents as early as possible. More specif i cally, in this work,we explore methods for detecting high- and low-performingstudents in an introductory programming course alreadybased on the performance during the very f i rst week of thecourse. Variants of the topic have been investigated previ-ously, for example, by Jadud, who proposed an approach toquantify students’ ability to solve errors using source codesnapshots [15], Ahadi et al., who measured students’ knowl-edge using tests [1, 2], and Porter et al., who used in-classclicker data as a lens into students’ performance [24, 25].This work is organized as follows. First, in Section 2, weprovide an overview of the evolution of the f i eld of under-standing factors that contribute to students’ performancein introductory programming. Then, in Sections 3 and 4we outline our research questions and data in more detail,as well as explain the methodology and outline the results.The results are discussed in Section 5, and f i nally, Section 6concludes the work and outlines future research questions.121
