Article: Teaching the Tacit Knowledge of Programming to Novices with Natural Language Tutoring
For beginning programmers, inadequate problem solving and planning skills are among the most salient of their weaknesses. Novice programmers routinely struggle with the most basic of programming problems and much of it can be attributed to a failure to adequately plan. This paper presents ProPL (Pro-PELL), a dialogue-based intelligent tutoring system that is intended to introduce programming problems, elicit goal decompositions and program plans from students, and help them understand some of the nuances involved in planning a solution. The tutoring strategies used by ProPL are all derived from human-to-human tutoring dialogues that shared the same goals.
Many of these tutoring tactics seek to leverage students’ intuitive understandings of the problem, how it might be solved, and the underlying concepts of programming. For example, many problems are easy to perform by hand or on paper, but are subtle and complex (for a novice) to solve in a programming language. The classic game of Rock, Paper, Scissors (RPS) is a perfect example. ProPL will ask students to demonstrate the skill and reflect on the decisions being made in an effort to call out the subtleties. For RPS, this means helping them realize that although we (as humans) can instantly tell who wins when one person picks rock and another paper, we must explicitly encode all possible combinations in our program. In one sense, ProPL helps students decompile their intuitive knowledge and uses that to support their expression of it algorithmically.
The tutoring tactics walk students through three questions repeatedly: What goal should be pursued? What general approach is best? And how can you build a plan to achieve the goal? The tutor applies this 3-step pattern repeatedly until all programming goals have been discussed. As the dialogue proceeds, a pseudocode solution grows to cover new goals, ultimately achieving all goals and solving the problem. This is called staged approach and is a known tactic novices use to deal with complex programming problems.
To test ProPL and the efficacy of natural language tutoring to teach and scaffold acquisition program planning skills, we conducted a small-scale experiment comparing students who used ProPL with a control group who read the same content. Students were recruited from 3 different introductory programming courses and given the same assignments and pretests. Copies of all programs submitted to the compiler were collected during the semester (with approval) and these logs were inspected for assessment. We used intention-based scoring to judge the quality of their first attempts at achieving programming goals – this metric ignores syntax errors and focuses on logical correctness (it is performed by human graders). We also gave a written post-test and a follow-up survey. Our primary finding is that students who received tutoring from ProPL, when compared to those in the control group, demonstrated an ability to work with “chunks” of code rather than the more common, line-by-line approach taken by beginners. In addition, ProPL students reported understanding the material less than the control group – thus, the control group suffered from the “illusion of knowing” and it seems dialogue-based tutoring mitigated this effect.
Author 1: H. Chad Lane [email protected]
Author 2: Kurt VanLehn [email protected]