A Multimodal Examination of Visual Problem Solving

Introduction. Stoichiometry problems can be presented and represented multiple ways, challenging students to apply their conceptual understandings. These concepts can be understood on an empirical level by the composition of the system, or on a model level, using formulas and stoichiometric numbers.¹ Student difficulty in solving these problems has been linked to the student’s representational focus,² and problem solving approach.³ Explanations sometimes reveal inconsistent usage of correct concepts.^4,5 By comparing oral, symbolic, algorithmic, and gaze-patterns of each participant solving visual chemistry problems with multiple representations, this study sought to gain insight into the problem solving approaches used. Chemistry understanding was investigated using multiple representations, building off the triplet nature of chemical knowledge.⁶ Following Kress,⁷ we viewed data from a single assessment or modality as incomplete, an orchestration of meaning was sought using multiple data sources.

Setting, Participants, and Data Collection. Forty-one undergraduate chemistry students, enrolled at an urban Northeastern US university, volunteered to participate in this study. IRB approval was obtained before participants were recruited and data collected. Each participant completed both a pre- and post-assessment. These assessments consisted of an online Chemistry Self Efficacy Questionnaire (CSE) (modified from a published questionnaire),⁸ a 15 question multiple-choice assessment of the participant’s knowledge of the Particulate Nature of Matter (PNM). Participants then played an adapted version of the PhET chemistry problem solving game called Reactants, Products, and Leftovers⁹ that scored for the amount of successful responses. The PhET game requires students to determine the number of reactant molecules when given product molecules or the number of product molecules when given the reactant molecules.

The PNM assessment is a combination of three assessments^10,11,12 focusing on definitions and concepts related to chemical change. Following an existing protocol for validation,¹³ the correct answers for all assessments were determined by consensus from three general chemistry professors. After the PNM Assessment, students were asked to reflect on their performance during the assessment and to evaluate three dimensions of their performance: 1) their level of confidence in understanding the questions being asked, 2) their success solving problems, and 3) their confidence in solving problems of a similar type.  Each question asks the participant to rate their confidence on five-point Likert-type scale, where 1 is ‘not at all confident’ and 5 is ‘completely confident’.  Overall the confidence assessment has a total score range of 3 to 15. The Representational Competency Assessment (RCA) is an adaption and extension of an assessment used previously to investigate students’ abilities to draw and interpret diagrams of atoms and molecules.¹⁴ The version used here consisted of eight questions with oral, symbolic, and algorithmic answers. After completing the assessment, a retrospectively ‘talk aloud’ protocol provided an opportunity for each participant to describe their problem solving approach.

Eye-tracking. As each participant played the PhET game,⁹ eye-tracking data was collected using the T-60 Tobii Eye-Tracker. This allows the researcher to detect where a participant is looking, and we assume that this correlates to where their visual attention is directed.¹⁵ Eye-tracking alone cannot indicate why a participant looks at a particular area, though deeper analysis is possible by combining multiple modalities of analysis.

Data Analysis.  Each question attempt lasting longer than 2 seconds was clustered by the percentage of fixations in each of six identified areas of interest (the equation reactants, equation products, submicroscopic reactants, submicroscopic products, reactant numbers, and reactant products). We used principal component analysis to characterize and further investigate each cluster,¹⁶ and these viewing pattern clusters were compared to the themes arising from qualitative analysis. Misconceptions related to the concept of conservation before and after a chemical reaction were used as a lens¹⁷ to guide both the quantitative and qualitative analysis for this study.  Answers from two specific questions in the PNM assessment (Figure 1) were used to group participants for further analysis. PhET scores for each of the two groups (those who correctly answered both questions in the pre-assessment and groups who incorrectly answered one or both) were compared and a grounded theory, discourse analysis approach was used to qualitatively investigate the talk aloud protocols for participants who answered one or both of the questions incorrectly.

Quantitative Findings. A significant difference was observed (t = 2.975, p = 0.005) in the PhET scores for participants who correctly answered both of Q 5 and 6 of the pre-PNM (n = 20), when compared to the group of participants who missed one or both of the questions (n = 22). A significant difference (t = 1.7, p = 0.04) between the two groups was also observed when comparing the total post-assessment scores (the sum of the CSE, PNM, CA, and RCA that were administered after playing the PhET game). During the session, the only feedback participants received was their PhET score. The pre- and post-assessments were scored after the session ended. The two participants (Table 1) with the lowest PhET game scores were selected for further qualitative analysis.

Table 1: The assessment scores for the two participants selected for deeper qualitative analysis, both participants correctly answered question 5 but incorrectly answered question 6 (selecting answer option E).  Neither changed their answers between the pre- and post- assessments.

Qualitative Findings. Qualitative analysis of the talk aloud protocols for the 22 students who incorrectly answered one or both of the two selected pre-assessment questions, revealed three key themes: using the concepts of moles or molecules interchangeably, discussing concepts related to conservation, and, how each participant reflected on the difficulty of questions when they found a mistake during the talk aloud protocol.

Profiles of Selected Students. Both Ryan and Heather demonstrated a literal translation between representations but their frustrations (and initial CA) were strikingly different. Both students selected answer option e on the PNM, Q6, which represents a translation of coefficients into molecular composition. Ryan was a first-year engineering student in the first semester of the intensive General Chemistry Lecture course. He commented that he “used to feel very confident on chemistry, but since college, I feel that I do not get to allocate enough time to chemistry which has resulted in a drop of my performance.” Heather was a post-baccalaureate, pre-medical student in the first semester of the regular General Chemistry Lecture course. When asked if she felt chemistry was relevant to her daily life, this time she selected ‘not at all’ and stated, “It's really like a foreign language to me that I feel I can rarely use.” Her visible frustration when playing the PhET game was consistent with her earlier comments about the language of chemistry being difficult for her to work with. When interpreting problems with multiple representations she sometimes reported the equation as depicted in the submicro image resulting in coefficients that corresponded to the number of each molecule or atom (Figure 2), including leftover excess reactants if they were present in the diagram. When reporting the limiting reactant she selected the reactant with the least number of molecules, disregarding the mole ratio required. This direct translation of the submicro drawing into an equation suggests her use of equations is developing, but she is not yet using the concepts behind the equation notation.  Although she struggled with the chemistry, she continued to try to solve each problem and when she finished the session she was positive and smiling.

Figure 2: Heather’s answer to question 8 from the post-RCA. Heather’s literal translation of the submicroscopic images included both the products and leftover excess reactants in the balanced chemical equation. She reported: “And I could see that from the drawing.  So, then I knew exactly what the limiting reactant was and how many moles of the excess reactant would be there.".

PhET Eye-tracking. Ryan’s eye-tracking gaze patterns show almost exclusive use of the numbers section of the game, only occasionally viewing the submicro representations or the equations (Figure 3). This horizontal shifting within the bottom (numbers) section of the simulation can be contrasted with other participants who shifted between different representations of a single species (resulting in a more vertical viewing pattern).

Figure 3: Ryan’s gaze patterns for two attempts on Level 1 Question 5, answered correctly and Level 3 Question 3 (answered incorrectly) respectively. Both gaze plots show Ryan’s heavy use of the numbers portion of the simulation with no use of the equations section. Early in the game Ryan did view the submicro section occasionally but shifted his viewing pattern by the end of the game to exclusive use of the numbers.

Ryan’s scrap paper adds an additional element to this analysis since his problem solving relied heavily on accounting done on those pages. Although he did not make notations for every question, most of the questions are worked out on his scrap paper (Figure 4). Heather used a counting approach to determining the number of atoms rather than the mole ratios, shifting her gaze within and between the submicro representations (not using the coefficients). She sometimes ‘searches’ between the representations, with her fixations shifting between each of the viewing areas in a circular or ‘back and forth’ motion.

Figure 4: Ryan’s scrap paper for Level 3 Question 3, similar to his answers from the RCA he writes the equation

Clustered Viewing Patterns. Viewing patterns for each question attempt from each of the 41 participants (410 separate question attempts) were clustered and four distinct groups emerged.¹⁴ Participants often shifted between viewing patterns, seven of the 41 participants had viewing patterns in all four clustered groups with only three participants employing the same viewing pattern for all question attempts. Ryan’s viewing segments were clustered with other participants who heavily relied on the numbers section of the simulation. Heather’s viewing patterns shifted between three of the viewing pattern clusters, making use of different representations for different questions. Her visible frustration during the simulation may be related to her shifting visual strategy, as reflected in her circular ‘searching’ viewing patterns and answers that were changed, reconsidered, then returned to their original value before submitting her answer.

Discussion and Implications. The three themes that emerged from the discourse analysis (conservation, reflection, and using moles versus molecules) provided insight into the PNM assessment responses and consistent with conceptual understandings supported by evidence from the multimodal individual student profile. These themes of conservation, reflection, and the use of moles versus molecules have deep implications for the teaching and learning of solving stoichiometry problems. An examination of all the incorrect answers entered in the PhET game by both selected students shows a persistent pattern. Namely they do not conserve atoms, as was observed in their responses to RCA and PNM assessments. This suggests they may hold a misconception about the conservation of mass (not conserving the number of atoms before and after the reaction). Moreover they appear to directly link the balanced equation to the submicro representation.

            A multimodal student assessment offers opportunities for teachers to identify student misconceptions or conflicting conceptual understandings then support the student in rethinking how they approached solving these problems. Posner et al.¹⁸ highlighted the importance of Socratic teaching in supporting students in shifting towards fully engaging with new, scientifically correct concepts. The evidence reported here of the student’s opportunity to catch mistakes and rethink problem-solving approaches (during the retrospective talk aloud) supports the inclusion of discourse in chemistry learning environments, providing students with multiple opportunities to verbalize and reflect on how they interact with visual chemistry representations. The use of multimodal artifacts to support this reflection may be particularly important when student answers are inconsistent and viewing patterns represent a searching or limited use of multiple representations. Allowing these students to reflect on their viewing pattern and discuss their algorithmic answers may be particularly useful. One option is to present students with gaze plots that contrast with their own and ask the student to identify similarities and differences between the two.  

By asking students to talk through their problem solving process and reflect on how they solved a problem, this study provided opportunities for students to reconsider how they approach these chemistry problems. Students may exhibit aspects of misconceptions while correctly using correct conceptions; as a result, a deeper view of how students understand and use chemistry ideas is needed to increase understanding and learning. By listening to how students solve problems, discussing their problem solving approaches, and including gaze patterns with this discussion, educators may be better able to interact with and challenge misconceptions held by students. Specifically, designing instruction to provide students with opportunities to reflect on their problem solving approach using both their retrospective talk aloud protocol and a viewing of eye-tracking gaze patterns.

References.

1.      BouJaoude, S.; Barakat, H. 2000, School Sci. Rev., 81, 91-98.
2.      Arasasingham, R. D.; Taagepera, M.; Potter, F.; Longers, S. 2004, J. Chem. Educ., 81, 1517.
3.      Fach, M., de Boer, T.; Parchmann, I. 2006, Chem. Educ. Res. Pract., 8(1), 13-31.
4.      Posner, G. J.; Strike, K. A.; Hewson, P. W.; Gertzog, W. A. 1982, Sci. Educ., 66(2), 211-227.
5.      Gauchon, L.; Meheut, M. 2007, Chem. Educ. Res. Pract., 8(4), 362-275.
6.      Johnstone, A. H. 1991, J. Comp. Assisted Learning, 7(2), 75-83.
7.      Kress, G. 2001, Multimodal teaching and learning: Theories and the science classroom. Continuum: New York, NY
8.      Garcia, C. A. 2010, PhD Thesis, University of South Florida.   (Retrieved from http://scholarcommons.usf.edu/etd/1638)
9.      PhET. (2013). Retrieved from http://phet.colorado.edu.
10.  Frank, D. V. Focus on physical science; Pearson Education, Prentice Hall: Upper Saddle, NJ, 2001
11.  Mulford, D. (1996). Masters Thesis. Purdue University. (Retrieved from http://www.jce.divched.org/jcedlib/qbank/collection/CQandChP/CQs/Concept...)
12.  Modic, A. L. 2011, Masters Thesis. Montana State University, Bozeman, Montana. (Retrieved from http://scholarworks.montana.edu/xmlui/handle/1/1886)
13.  Yezierski, E. J.; Birk, J. P. 2006, J. Chem. Educ., 83(6), 954-960.
14.  Davidowitz, B.; Bhittleborough, G.; Murry, E. 2010, Chem. Educ. Res. Pract., 11, 154-164.
15.  Just, M.; Carpenter; P. Psychol. Rev. 1980, 87, 329–354. 

16.  Hansen, S. J. R. H.,  2014, PhD Thesis, Columbia University.
17.  Saldaña, J. 2013, An introduction to codes and coding. The Coding Manual for Qualitative Researchers (2 ed.) Sage Pubs. Ltd.:Thousand Oaks, CA. pp. 40
18.  Posner, G. J.; Strike, K. A.; Hewson, P. W.; Gertzog, W. A. 1982, Sci. Educ., 66(2), 211-227.