Article: A longitudinal evaluative study of student difficulties with engineering graphics
A LONGITUDINAL EVALUATIVE STUDY OF STUDENT DIFFICULTIES WITH ENGINEERING GRAPHICS
STRUCTURED SUMMARY
By Charles Potter, Errol van der Merwe, Wendy Kaufman and Julie Delacour
University of the Witwatersand, Johannesburg
1. Overview
Our recent paper in the European Journal of Engineering Education presents the results of a long-term evaluative study undertaken at a South African university (Potter, Van der Merwe and Kemp, 1984; 1987; Potter, 1991; Potter and Van der Merwe, 1994; 2000; 2001; 2003; Kaufman, 2001; 2003; Delacour, 2004). In the paper we also focus on evaluation methodology, as well as the theoretical basis and implications of our research.
2. Focuses of our Paper
Our paper reports the results of a longitudinal evaluation focused on an intervention designed to change existing teaching practices in engineering graphics, which is a course area in our university associated with high failure rates. It has had the aim of improving pass rates in the course, as well as throughput in terms of increased numbers of students entering subsequent years of study, and ultimately graduating as engineers.
Our intervention has been conducted over a period of rapidly changing socio-political and educational context in South Africa both prior to and subsequent to the country’s first democratic elections in 1994. There have been major socio-political changes, major demographic changes, as well as broader educational and curricular changes affecting first year university teaching in South Africa. Our evaluation has thus been longitudinal, long-term and ongoing, with the aim of establishing the course’s modus operandi in a context of change. We have used cross-validation through a series of non-experimental and quasi-experimental studies as a strategy for building inference, as well as for judging the continuing value of the intervention to students.
3. Results of our research
3.1 Initial studies
The results of our predictive studies established a firm link between spatial ability (and in particular three dimensional spatial perception) and academic performance (Potter, 1991a; 991b; Potter & Van der Merwe, 1993; 1994), but did not establish whether spatial ability was the only influence on academic performance which was operating. The results of pre- and post-testing of samples of the first year Engineering students over a two year period indicated that gains in spatial ability had taken place in both years (Sample 1″>t = 15,00 (262), p < ,001; Sample 2: t = 9,23 (137), p < ,001). A comparison was also made in the second year of the study with the pre- and post-test scores of a sample of first year Science students, using the test of three dimensional spatial perception which had correlated most highly with first year academic performance in the previous predictive analyses. Both the Engineering and the Science students were matched by courses taken (N’s at beginning of academic year = 175; 165 respectively), and were registered for different variants of the BSc degree. The results indicated that gains in spatial ability took place in both samples after elapse of six months (Engineering sample t = 9,23 (137), p < ,001; Science sample t = 3,90 (125), p < ,001). The Engineering students gained more that the Science students (F = 1,90 (137, 108), p < ,001) over a comparative period of six months between pre-and post-testing.
The Engineering students had commenced the year with significantly greater scores in spatial ability than the Science students (t = 6,53 (337), p < , 001). Analysis of variance and co-variance was thus conducted, holding initial level of spatial ability (as measured by the test of three dimensional spatial perception) constant. The results of the within-groups comparisons indicated that spatial ability changed significantly in both Science and Engineering students over the year, while the between-groups comparisons indicated that Engineering students gained more than Science students (F = 4,79 (1, 245), p < ,05). In addition, when initial level of spatial ability was held constant, it was apparent that level of spatial ability at entry point to the university was an important influence on subsequent academic performance (F = 38,43 (3, 243), p < ,001). Overall, the results of the analyses indicated that initial level of spatial ability at time of intake to the university had been a significant influence on the results of both groups, and had been a greater influence on academic performance than type of degree taken while at university.
Viewed in relation to the previous predictive research undertaken with the first year engineering student body, it was thus possible to conclude not only that spatial ability was an important predictor of subsequent academic performance in engineering graphics, but that instruction in engineering graphics improved spatial ability. This conclusion was reinforced by the findings of our previous predictive studies, indicated clearly that students with low levels of spatial ability (and in particular low levels of three-dimensional spatial perception) at time of intake to the university were at risk, and would be unlikely to pass the first year Engineering Graphics course without remedial intervention.
3.2 Results of our cross-validations
3.2.1 Longitudinal trends in the data
To cross-validate our findings, we undertook a number of studies using the instrument (the H test) which had the highest predictive relationship with academic performance in the engineering graphics course. These were supported by analyses of pass rates and qualitative data from questionnaires, interviews and focus groups.
To establish whether levels of three dimensional spatial perception were consistent in the first year engineering student body over time, we analysed longitudinal trends using pretest data from the H Test in 1982, 1983, 1992, 2000 and 2001 (Delacour, 2004). This analysis indicated that the three dimensional spatial perception of first year engineering students at time of intake remained relatively constant over the total evaluation period, with the exception of 2000, where there was evidence of a decline. However, this was apparently an isolated phenomenon, as there was a consistent trend across the different years, other than for the year 2000. There was also no difference in the 2001 data as compared to previous years (1982; 1983; 1992). Longitudinal trends in post-test data were also analysed from 1982, 1983, 1992, 2000 and 2001 (Delacour, 2004). Analysis using Tukey’s studentised range (HSD) test revealed a clear distinction between different groups of post-test scores on the H test, as follows:
· The initial two measures (1982, 1983) were not significantly different from one another and neither were the last two years (2000, 2001).
· However there were statistically significant differences between the 1980’s and the latter three years (1992, 2000, 2001).
· There were also statistically significant differences between the 1992 data and the last two years (2000, 2001).
Overall, these analyses indicated that there had been fluctuations from year to year in the amount of gains in three dimensional perception made by the engineering students. The implication was that there had been a decline in gains in spatial ability, despite continuing high pass rates for the course.
3.2.3 Comparison of Engineering and Science Students
Similar trends were found in analyses based on pre and post-testing of samples of engineering and science students on the H test in both 2000 and 2001 (Kaufman, 2000; 2003). In the data for both these years, testing for equality of means on the pre-test revealed a significant difference between the two independent samples (2000 t = 9.111 (237), p < ,0001; 2001 F = 55,79 (290), p < ,0001). The differences between the means on the posttest, also revealed a significant difference between the samples (2000 t = 5.502 (159), p < ,0001; 2001 F = 29,38 (290), p < ,0001).
Results of paired t-tests also indicated that both the engineering samples (2000 t = 5.05 (83), p < ,0001; 2001 t = 7.17 (149), p < ,0001) and the science contrast groups groups (2000 t = 9.48 (77), p < ,001; 2001 t = 9.65 (148), p < ,0001) improved in terms of three dimensional spatial perception from pre-test to post-test (Kaufman, 2000; 2003). This corroborated previous findings indicating that spatial ability was not fixed, but changed in response to instruction (Potter, 1991a; 1991b).
In terms of programme effects, these analyses indicated that three dimensional spatial perception continued to develop in the engineering students taught using the activities and materials developed as part of the intervention. At the same time, there were indications that the amount of change had decreased in the years subsequent to the country’s first democratic election in 1994.
The stability of the relationship between three dimensional spatial perception and the intervention was further tested by examining changes within the data yielded by the H Test at the beginning of the academic year and after six month’s instruction in the engineering graphics course, over the twenty year period from 1982 to 2001. Multivariate analysis was conducted using MANOVA. (Delacour, 2004). This analysis indicated both a significant main effect of the intervention and a significant interaction between intervention and yeargroup (ie scores on the H Test in different years). Both main effect and interaction were significant beyond the .0001 level, with the main effect exerting the major proportion of influence within the data.
The significant interaction indicated that the relationship between the variables was a complex one. Inspection of the matrix indicated that there were fluctuations in the scores of the engineering students on the H Test over the twenty year period spanned by the evaluation, which were likely to have contributed to this result.
In relation to this interaction, the significant main effect indicated that, over and above significant differences observed between level of three dimensional spatial perception in students studying in different years, the ‘intervention’ effect of the engineering graphics course exerted a powerful and significant influence in the data matrix. There was also evidence of a firm and ongoing relationship between level of three dimensional spatial perception at time of intake to the university and examination results (with scores on the H Test in the year 2001 accounting for between 38% and 40% of the variance in first year results in both engineering graphics as well as other first year engineering subjects).
Overall, the analyses indicated that the amount of improvement in three dimensional spatial perception between pre and post-testing had decreased over the twenty years between the early 1980’s and the year 2000. Despite these fluctuations, the H Test remained a stable predictor of academic performance in engineering graphics, despite evidence of third variables (eg changes in demographics) affecting the results obtained by students. It was also a consistently reliable instrument (reliability coefficients of between ,86 and ,90). The evidence also indicated that there was an ongoing need for the intervention, as there was an ongoing predictive relationship between three dimensional spatial perception and academic performance in the course, as well as academic performance in all first year engineering subjects.
4. Conclusions
Based on the results of our analyses, three main inferences concerning programme effects can be drawn, as follows:
· For all the samples of engineering and science students we have studied in our university over a twenty five year period, clear indications emerge that three dimensional spatial perception is an important influence on academic performance at university, forming part of a broader spatial ability factor affecting performance in engineering graphics in particular.
· Clear indications also emerge from the various studies conducted concerning the trainability of three dimensional spatial perception in engineering students.
· There are clear indications from our data that a teaching methodology based on Piaget’s theories of perception and mental imagery is likely to improve the spatial ability both of students studying engineering graphics both at pre-university level, as well as students studying engineering graphics at first year university level.
The results of our analyses have also indicated that the type of instruction we have provided in engineering graphics remains effective in producing change in the spatial ability of the engineering student samples we have tested, and with whom we have worked in our university. The overall conclusion is that both three dimensional spatial perception and academic performance improve in South African engineering students, in response to instructional techniques designed to increase the ability to model, copy, sketch, visualise and represent objects in three dimensions. Level of three dimensional spatial perception at time of intake to our university has been an important influence on the academic performance of students taking engineering graphics courses, and also appears to influence the development of spatial ability in comparison groups of science students. Given the multicultural composition of our first year student intake, these results may also have applicability in other contexts.
Author 1: Charles POTTER [email protected]
Author 2: Errol VAN DER MERWE
Author 3: Wendy KAUFMAN
Author 4: Julie DELACOUR
Article Link: https://www.tandfonline.com/doi/full/10.1080/03043790600567894