Article: A Paradigm for Assessing Conceptual and Procedural Knowledge in Engineering Students
Taraban, R., DeFinis, A., Brown, A., Anderson, E. E., & Sharma, M. P., “A Paradigm for Assessing Conceptual and Procedural Knowledge in Engineering Students,” Journal of Engineering Education, Vol. 96, No. 4, 2007, pp. 335-345.
This research is part of a series of case studies that we have carried out [1-3] in order to better grasp how undergraduate engineering students learn. This work is based on the premise that developing reliable models of students’ cognitions and behaviors is a crucial step in designing effective teaching methods and materials. From a theoretical perspective, cognitive psychology has identified several distinct kinds of knowledge (and associated neural networks and pathways) that are responsible for expertise. In general, some of this knowledge consists of KNOWING THAT, and some consists of KNOWING HOW. To the credit of the engineering education community, these distinctions have been implemented into ABET criteria. According to the ABET criteria for accrediting engineering programs [4], a major competency that undergraduate programs are expected to develop in engineering students relates to knowledge in specific disciplines. These objectives are expressed in the ABET criteria through verbal forms like knowledge of, an understanding of, and occur with specific arguments like knowledge of environmental issues, and understanding of concepts of professional practice. These objectives can be distinguished from another class of ABET objectives that focus on what students should be able to do, which use verbal forms like an ability to apply, and an ability to use, in specific expressions like ability to use mathematical and computational techniques, and ability to conduct laboratory experiments. These differences are summarized in the ABET definition proposed for 2008-2009 evaluations of program outcomes as describing what students are expected to know and be able to do by the time of graduation.
The ABET criteria formulate learning using educational objectives, which is a standard pedagogical practice since the groundbreaking work of educator Ralph Tyler [5]. Our work, from a cognitive perspective, has attempted to articulate these objectives in terms of cognitive processes and problem-solving behaviors. In a study just prior to this one [3], we developed a coding table and assessment rubric for reliably coding students’ cognitions and behaviors. That research involved undergraduates who had not taken introductory thermodynamics; the present research involved students who were enrolled in introductory thermodynamics (Thermo I) and students who were enrolled in the course sequel (Thermo II). The materials were computer-based instructional supplements authored by E. E. Anderson for the textbook Thermodynamics”>An Engineering Approach (4th Edition) [6], and were quite rich in content and visual appeal. Computer screens presented students with text content, tables, figures, and graphs, and also included active-learning screens with interactive exercises, graphical modeling, physical world simulations, exploration, and quiz screens. The main purpose of both studies was to validate an assessment paradigm capable of quantifying engineering students’ conceptual (knowing that) and problem-solving (knowing how) knowledge.
Data were collected one-on-one in a lab setting. Students provided verbal protocol (“think-aloud”) data as they used the instructional software. The verbal protocol data were generated by asking students to say what they were thinking as they worked through the computer screens. The method is well-established in the learning sciences [7]. The results showed more cognitive activity on computer screens requiring overt user interaction compared to text-based screens. The data also indicated that higher- versus lower-performing students, based on course grades in Thermo I and Thermo II, engaged in more higher-order cognitive processing. There was no evidence that students gained deeper cognitive processing as they advanced from Thermo I to Thermo II. The results indicated that students gain more from instructional software when they are involved in active learning, rather than simply reading text and figures on a computer screen. The finding that students, as a whole, did not engage in higher-level cognitive processes as they advanced from Thermo I to Thermo II is consistent with a hypothesis that many students succeed in their coursework, but not through the same means. A strong version of this hypothesis is that the most successful students, regardless of level (freshman to senior) apply higher-level cognitive processing. That hypothesis stands as the most important to confirm in future research.
References
[1] Taraban, R., M.W. Hayes, E.E. Anderson, and M.P. Sharma, “Giving Students Time for the Academic Resources That Work,” Journal of Engineering Education, Vol. 93, No. 3, 2004, pp. 205-210.
[2] Taraban, R., E.E. Anderson, M.W. Hayes, and M.P. Sharma, “Developing On-Line Homework for Introductory Thermodynamics,” Journal of Engineering Education, Vol. 94, No. 3, 2005, pp. 339-342.
[3] Taraban, R., Anderson, E. E., DeFinis, A., Brown, A., Weigold, A., & Sharma, M. P., “First Steps In Understanding Engineering Students’ Growth of Conceptual and Procedural Knowledge In An Interactive Learning Context,” Journal of Engineering Education, Vol. 96, No. 1, 2007, pp. 57-68.
[4] ABET (Engineering Accreditation Commission), “Criteria for Accrediting Engineering Programs,” Accessed on 02/01/07 at www.abet.org/forms.shtml.
[5] Tyler, R. W., Basic Principles of Curriculum and Instruction, Chicago, IL”>University of Chicago Press, 1949.
[6] Cengel, Y.A. and M.A. Boles, Thermodynamics: An Engineering Approach, 4th Edition, Boston, MA: McGraw-Hill, 2001.
[7] Ericsson, K.A., and H.A. Simon, Protocol Analysis”>Verbal Reports as Data, Cambridge, MA: MIT Press, 1984.
Author:
Article Link: