Article: A First Look at Novice Compilation Behaviour Using Blue
“Novice compilation behavior” is, simply put, what novice programmers do while writing and compiling programs. As the compiler first checks to see if a program is syntactically correct, novices often find themselves editing and recompiling a program in an attempt to author a program that can make it past the compiler. Sadly, it is a rare compiler with error messages that are understandable by a novice. As a result, it is not uncommon for the novice programmer to edit and compile their program many, many times before it is syntactically correct.
This rapid edit-compile cycle is at the root of an assumption held by many programming educators”>”students who compile their program often are trying to get the computer to do their work for them.” The truth is far more complex, and our research begins to explore what students are actually doing from one compilation to the next. Using this knowledge, we hope we might improve our approach to programming education through a combination of improved tools and methodologies.
To understand the behavior of students compiling and recompiling their programs, we instrumented BlueJ, a widely used pedagogic programming environment for Java. Every time a student compiled their program, a copy of their program’s source was shipped off to a database, along with any messages generated by the compiler. Over the course of two years, this generated tens of thousands of programs from 120 students who consented to having their programming practice observed in this way. Truly, this was a mountain of data.
Collecting our data was easy; analyzing our data was a challenge. Our initial pilot study indicated that there were interesting aggregate quantities to be observed (distribution of syntax error types, average time between compilations), and the program sources were potentially rich sources for detailed document analysis. We believe our single largest mistake in our analysis was focusing too early for too long on quantitative, rather than qualitative, analyses of our data. Our understanding of our students’ behavior blossomed when we simply sat down and started reading through the sequences of small changes made in programs written by students in our study. Our in-depth qualitative work followed on from the research represented here.
Taking a grounded theoretic approach to our document analysis, we attempted to uncover editing patterns in our students’ behavior. Our intent was to evolve a protocol that other researchers might use to study their own students’ programs. Instead, we codified our protocol in the form of two executable tools. In this way, it is possible for other researchers to directly apply our results to similar studies, independent of the student population and language being used.
The first tool we developed visualizes an entire programming session in a concise, tabular view, showing us the kinds of syntax errors students encountered, and where in their program they encountered them. The second tool takes a programming session and assigns an ‘error quotient’ to the session. An error quotient of zero means that the student, upon encountering a syntax error, immediately fixed it—there was no repetition of error. An error quotient of one implies the student edited and compiled their program repeatedly, and every compilation ended with the exact same syntax error.
These two tools allow us, as educators, to quickly evaluate how productive a student was in a given session. The error quotient quickly tells us whether they spent most of their time wrestling with errors or not. The visualization lets us see exactly how they spent their time, and where in their program their efforts were focused. Did they spend most of their time wrestling with one syntax error in one part of their program? Or, perhaps they wrestled with many errors all over their program?
Given the data currently available, it is clear that the error quotient, averaged over many sessions, correlates strongly with a student’s performance on end of year examinations. While this is interesting, we are more excited about the prospect of future work regarding the effectiveness of our visualizations for supporting educators in the classroom. Being able to quickly evaluate the programming behavior of 10, 100, or 1000 students, and then recommend additional exercises to students focusing on perceived weaknesses, or perhaps one-on-one tutoring, strikes us as an excellent response to behaviors perceived to be unproductive. Ultimately, our goal is to provide timely and appropriate instruction to all of our students learning to program.
Author 1: Matthew C Jadud [email protected]