PQtutor, a quasi-intelligent tutoring system for quantitative problems in General Chemistry

PQtutor, a quasi-intelligent tutoring system for quantitative problems in General Chemistry

Karsten Theis, Ph.D., Dept. Chemical and Physical Chemistry, Westfield State University

ktheis@westfield.ma.edu

Abstract

The best tutors are humans who give a student the appropriate amount of guidance necessary for learning while helping the student stay confident, motivated and focused. So-called intelligent tutoring systems, trying to replicate the discipline-specific and the psychological dimensions of expert human tutoring, require enormous investments and are not accessible to the larger student population. In a six-month sabbatical effort, I created PQtutor, a free online tutor designed to help students work out homework problems closely related to worked examples they already studied. The software is built on top of an online calculator for science learners (J. Chem. Educ., 2015, 92, 1953–1955) and uses problems from an open textbook, making PQtutor accessible in terms of both technology and cost. PQtutor works by comparing student input to a model answer in order to generate prompts for finding a path to the solution and for correcting mistakes. The feedback is in the form of questions from a virtual study group suggesting problem-solving moves such as accessing content knowledge, reviewing the worked examples, or thinking about the meaning of the question and the answer. In cases where these moves have been exhausted but the problem remains unsolved, the tutoring system suggests seeking intelligent human help.

Introduction

Learning to solve quantitative problems in college chemistry courses is a challenge for many students. Students have to integrate general problem-solving skills, chemical insight, unit algebra and error propagation into their solution, and learn how to tackle multi-step problems on paper. Students who get frustrated or stuck while doing homework seek various forms of help (Fig. 1). While there is a growing market for paid online supplements such as intelligent tutoring software or online human tutoring, not all students can afford these. There is also a growing amount of free help available on the internet's question-and-answer sites, but these contain material that can be wrong or misleading. In contrast, high-quality open textbooks are becoming available at no cost to students (and paid for by donations and support from foundations and not-for-profit organizations), for example those published by OpenStax [external link]. Striving to provide a free but valuable tutoring system supplementing a free textbook to students of General Chemistry, I spent a sabbatical writing PQtutor, an online tool to help students practice solving quantitative problems.

Figure 1.  Different ways of getting help with homework

Purpose of PQtutor

One goal of PQtutor is to allow students to practice quantitative problem-solving without losing sight of the chemistry context. Students give names to all quantities in the calculation, and they can easily include chemical or mathematical notation, figures or links to chemical data in comments. Moreover, when students submit their solution, they are presented with a multiple-choice follow-up question designed to elicit reflection on the meaning of the answer. These "What does the answer mean?" questions are inspired by the final step in the COAST problem-solving recipe used in W.W. Norton's chemistry textbooks by Gilbert et al. [external link]. Finally, chemical data is available within PQtutor (e.g. molar masses, table of quantities and units) and via links to outside sources (e.g. periodic table, thermodynamic data). Integrating the scientific context into the calculator helps bridge the divide between algorithmic and conceptual aspects of learning chemistry.

A second goal of PQtutor is to remind students of sound problem-solving strategies while they are working alone outside of the classroom. At different stages of solving a problem, students can request help from software agents that come in the form of quasi-intelligent peers in a slightly odd study group. Asking for help triggers reminders of general problem-solving strategies first, followed by more specific hints that do not give away the answer. Hopefully, the way hints are delivered will keep frustration and boredom at bay while encouraging students to think on their own.

PQtutor is limited in that it can not give help with higher level cognitive tasks (i.e. it is not intelligent). Thus, this homework system helps students to develop low-level and mid-level skills in quantitative problem-solving, making time to work on high-level skill development (such as identifying problem type or solving novel problems) in the classroom and in the laboratory.

Solving problems with PQtutor – use as a calculator

PQtutor works through a browser, and input is plain text entered via the keyboard (with some typing shortcuts provided), while mathematical and chemical notation is typeset in the output (the details of using the calculator are described in an online manual). Each command in PQtutor starts with a name, describing the quantity one is defining or calculating, followed by a value or mathematical operations on quantities already defined. For example, to calculate the density of a sample with a volume of 54.3 mL and a mass of 81.4 g, you would type the following into the user interface:

   V_sample = 54.3 mL

   m_sample = 81.4 g

   ρ_sample = m_sample / V_sample

Typing "return" or pressing “go” results in the output shown in Fig. 2. Variable names and math are typeset in the results for better readability. For example, fractions are written with a fraction bar, and subscripts are used for chemical formulae and for variable names. The calculator takes care of the lower level mechanics such as number arithmetic, unit algebra and error propagation, and catches errors such as trying to add a mass to a volume, or taking the logarithm of a quantity that is not dimensionless. (While these are low-level errors, it does not mean textbooks are free of them, see supplemental information on "dimensional rigor" below). Thus, the calculator ensures that work done by the students is mathematically sound.

Figure 2: Output from a simple calculation in PQtutor

Homework with PQtutor

PQtutor lets students work on homework problems that are very similar to problems already discussed in the class or in the textbook. The problems currently available are all adapted from an open educational resource, the Chemistry textbook published by OpenStax [external link]. In the textbook, worked examples are paired with similar problems to which the numerical answer, but not the solution path is given. The latter problems are used as homework problems. Thus, students can refer back to the worked examples to learn how to solve this type of problem (they can access these directly in PQtutor, e.g. example 17 of chapter 3 [view in PQtutor]). The instructor would post a link to the homework problem, in this case the follow-up to example 17. When students click on the link given by the instructor, the calculator displays the question, and students can proceed to enter commands (Fig. 3 and [view in PQtutor]). If students are unsure of how to proceed with a problem, or they just want feedback on what they already entered, they can request help from a virtual study group by just pressing go without any input. Hints come in two flavors, feedback or pumps; feedback points out problems with the work so far, while pumps are leading questions suggesting what the student might do next. These two types of hints are discussed in detail below. Once a student submits an answer, PQtutor shows a “What does the answer mean?” question, allowing for reflection on the calculation and its meaning. A short video illustrates the mechanics of solving a problem, submitting it to PQtutor and sending a report to the instructor.

Figure 3: PQtutor presenting a homework problem.  The question is shown in maroon on the top. Commands to the calculator are typed into the empty box.

The virtual study group

There are four characters in the virtual study group named Q, B, C, and L (Fig. 4). Each one of them represents a different set of skills, strategies or tactics to tackle quantitative problems. Q has general quantitative problem-solving skills such as considering the relationship between knowns and unknowns, dimensional analysis, picking up on common patterns (e.g. two-state problems), algebra to solve for an unknown and some general troubleshooting strategies. B has a firm grasp of the big picture tools chemistry has to offer, such as the periodic table, symbolic notation (chemical formulae and equations), conventions for naming different types of quantities, and data tables available in the textbooks. C is the expert on the specific chapter, the concepts that are introduced, the common pitfalls associated with the chapter, and the chapter-specific tools information the textbook offers. L, finally, likes to look up how others have worked out similar problems, primarily by referring to the worked example (but will probably start consulting the primary literature before she graduates). The following two sections discuss how the study group responds to different student input, illustrated by live demonstrations that show these characters in action.

Figure 4: Members of the study group and their respective special power

Feedback from the study group

PQtutor checks student input by comparing it to a model solution (the model solution for example 3.17 is shown in the supplemental information below under the heading "authoring new questions"). Given quantities are checked for dimensions [demo 1], value [demo 2], number of significant figures [demo 3] and sensible name [demo 4]. Results of calculations are matched with those in the model solution. If there is a no good match, the study group will comment [demo 5]. If there is some unexpected input, there will be feedback as well [demo 6]. Feedback is given in general terms, but there will be some more specific information (often about which line of input might be incorrect) when hovering over the character's picture. Feedback never directly states that something is incorrect – it is up to the student to critically examine their work and correct it if necessary. The reason for this is twofold: On the one hand, the software’s assessment of the students’ work is not flawless (see supplemental information "alternative solutions" below), on the other hand, the students should develop a critical mindset and be responsible for their own conclusions.

Pumps from the study group

The best human tutors do not give away the answer to homework, but give just the right amount of scaffolding so that students can figure out solutions by themselves. This requires substantial knowledge about the psychology of learning in general and about the discipline and the student specifically, none of which PQtutor has. Instead, PQtutor turns the model solution into a graph of dependencies (Fig. 5), matches the student answer to the solution graph, and identifies reachable subgoals. Then, it proceeds to ask guiding questions about missing data from the question [demo 7], missing data from other sources [demo 8], relationships between knowns and unknowns, and subgoals and goals [demo 9]. Like for the corrective feedback described in the previous section, the pumps are ordered from general to specific. Students who take pride in figuring out a solution on their own will not receive more help than they need (i.e. as soon as one of the prompts gives them an idea on how to proceed, they can try on their own). Students who are trying to game the system and max out the hints will receive plenty of hints, but the solution is never revealed directly. Because there might be students who are unable to find a solution even with the help of PQtutor, it is good practice to discuss the problem or at least show the solution at the beginning of the next class.

Figure 5: Solution graph for the example shown in Fig. 3

Chemistry resources in PQtutor

Sometimes, students are not aware of resources in their textbook that might help with quantitative problem-solving. PQtutor integrates these tools into the hints given by the virtual study group (there is also a study tools page [PQtutor page]). The B character shows links to the periodic table, the appendices, and information about formulae introduced up to a given chapter. The Q character shows links to a table of quantities and units along with their definitions up to a given chapter. As mentioned above, the L character always seeks information from the matched worked example. Finally, the C character shows links to the chapter summaries and the explanation of keywords in the given chapter. If students adopt the strategies modeled by the virtual peer group, they will learn to make better use of these resources in the textbook (and discipline-specific resources like databases and primary literature later on).

“What does the answer mean?” questions

When students are ready to submit their solution to a homework question, they have to answer one final multiple-choice question before moving on. These questions are called "What does the answer mean?" because they have the purpose of connecting the numerical answer to the chemistry the student is learning in a given chapter. The questions come in three flavors. Some ask whether the magnitude (or sign) of the result could have been anticipated from an estimate, for example:

  How does the density compare to that of water (water has a density of approx. 1 g/mL)?

   a) It is lower because it sank to the bottom

   b) It is higher because it sank to the bottom

   c) It is lower because it floats at the surface

   d) It is higher because it floats at the surface

Others ask for an interpretation of the calculated result, for example:

  Based on the pressure you just calculated, the airbag ...

   a) will collapse because the pressure is much lower than atmospheric pressure

   b*) will feel soft like a birthday balloon because the pressure is similar to atmospheric pressure

   c) will feel semi-hard like a basketball, which is inflated to 1.5 times atmospheric pressure

   d) will feel hard like a bicycle tire, which is inflated to 7 times atmospheric pressure

. Finally, some go after common misconceptions, for example:

  What happened?

   a) The volume increased because the distance between the particles increased

   b) The volume increased because the size of the particles increased

   c*) The volume increased because the number of particles increased

   d) The volume increased because the balloon got stretchier over time

In all cases, the questions ask the student to make a connection between the result of their calculation and concepts that they are familiar with or have just learned. The idea is to nudge students to think about what their result means, and to check whether they might want to re-examine their calculation to troubleshoot it.

Field testing and further development

Teaching General Chemistry I and II this year, I will give students homework through PQtutor to find out how the tutoring system works in practice. For each homework submitted, PQtutor reports the student answer as well as the hints, if any, given by the virtual study group. In aggregate, these data will show how much students rely on the hints, at which stage in problem-solving they request hints, and how many hints they view before proceeding with the calculation. The submitted homework will include any comments students entered, giving those students who do not succeed at solving a problem an opportunity to formulate where they got stuck, and what additional information they might have needed to continue. By talking to my students about their experiences using PQtutor, I can also gather anecdotal data about their academic emotions, in particular whether they experienced boredom or frustration while solving interacting with the online tool. Based on this data and feedback, I will be able to fix any problems with individual homework questions, fine-tune how the virtual study group interacts with the students (clarity and tone of the hints), and address any annoyances with the mechanics of entering commands. I will also be able to decide which of the many ideas for additional features would be worthwhile to implement (e.g. suggestions for more challenging questions after successful problem-solving or for foundational problems after failed problem-solving attempts, interpreting chemical equations, algebra or free-form responses entered by students, measuring student response time or asking them about how they are doing to decide on what type of hints to offer, giving students detailed feedback after they have studied a model answer upon submitting their work). Finally, this first field test with students will provide important pointers on how one might rigorously test the impact of PQtutor on the development of quantitative problem-solving skills in students of general chemistry.

Acknowledgements

I would like to acknowledge Westfield State University and my home department Chemical and Physical Sciences for granting the sabbatical that allowed me to work on this project. I would also like to thank the Biochemistry department of the Max-Planck-Institute of Chemical Ecology in Jena for their gracious support in form of office space, and Estonia-based artists Vladimir and Maksim Loginov of handmadefonts.com for allowing me to use their characters for the virtual study group. Finally, I would like to thank past and present students for their patience in testing and using the PQcalc and PQtutor software.

Supplemental information

Support for chemical notation and algebra

While PQtutor does not know any algebra (and will not help solving a mathematical formula for an unknown), it does support students in doing algebra by allowing them to write arbitrary math in comments. While PQtutors knowledge of chemistry is very limited (it does have a database of the fifty or so compounds most commonly mentioned in the textbook), it allows students to write chemical formulae and chemical equations in comments. In a live demonstration (i.e. a link to PQtutor with pre-filled input), you can experience how to more fully document a solution using comments integrated into the calculations [demo 10]. You can also click on different parts of the chemical equations to see how you can easily paste stoichiometric coefficients into the actual calculation.

Alternate solutions

Even though PQtutor is an example tracer working off a single model answer, it is able to adapt to many variations in student answers gracefully. The order of steps is flexible as long steps are independent of one another. Combining two steps into a single one is tolerated, as is using an intermediate variable to break a single calculation into two. Naming variables is flexible, as long as the root of the name matches the dimensions, and its value is correct or close to correct. However, some homework problems have two correct solution pathways that are sufficiently different to get PQtutor into trouble. For example, consider calculating the pOH from a given hydronium ion concentration (in aqueous solution at room temperature). The model answer might first calculate the pH, and then subtract it from 14 to get the answer. The student answer, in contrast, first calculates the hydroxide concentration via Kw and then takes the negative logarithm to get the answer. In this case, the relationship between the two solutions is not clear to PQtutors algorithm (one solution depends on knowing pKw while the other depends on knowing Kw). This could be solved in two ways. Either the question is changed to explicitly give either Kw or pKw, and asks to use this quantity in the calculation, or (which would require a modification of the software), two model solutions are allowed and the one that matches the student work better is used.

Dimensional rigor

There are at least three instances where typical general chemistry textbooks are less rigorous in their treatment of dimensions, scales, and units than PQtutor requires. The first concerns the dimensions of concentration in the context of equilibrium thermodynamics (and the logarithm function). In relationships such as pH = -log([H+]) or ΔG = -RT ln(K), the argument of the logarithm has to be dimensionless. However, introductory textbooks usually lack separate notations for concentration (units mol/L) and concentration divided by the standard state (unitless, equal to activity in the limit of infinite dilution). In PQcalc, the convention is that c_solute refers to a concentration, and [solute] to the dimensionless quantity. To convert between the two, multiply or divide by the standard state (1 mol/L for solutes). The second concerns dimensions of thermodynamic quantities ΔH, ΔG, and ΔS. Depending on the textbook and the chapter, these are either extensive quantities (e.g. units of J or J/K) or molar quantities (e.g. units of J/mol and J/(K mol)), but often the same symbols are used (upper-level textbooks sometimes use the convention that bold-faced quantities are molar to distinguish the two possibilities). In some cases, the formula used imply molar quantities (e.g. ΔG = RT ln K) or extensive quantities (ΔS = k ln (W1/W2), ΔS = q_rev / T). In PQtutor, the convention is that thermodynamic quantities H, G, and S are molar. If necessary, homework questions are changed to reflect that (for example, instead of calculating the enthalpy of burning a sample of a substance, the question will ask for the molar enthalpy of the reaction as well as the heat produced given how much reacts). The third concerns temperature scales, often treated less than rigorously in introductory textbooks. Internally, PQcalc always uses the Kelvin scale for temperature. To be able to enter and display quantities using the Celsius scale, PQcalc allows separate units for absolute temperature and temperature differences using the Celsius scale. Absolute temperatures are entered with units °aC, temperature differences with units °ΔC. This is relevant for the conversion into the Kelvin scale. For example, 25 °aC converts into 298 K, while 25 °ΔC converts to 25 K. If you subtract 0 °aC from 25 °aC, the answer is 25 °ΔC, if you try to add 0 °aC to 25 °aC, you get an error message. While ugly, this seems to work reasonably well.

Authoring new questions

It is straightforward to author new questions, and it requires minimal work. First, you need a description of the problem in plain text. It will be formatted just like user input in PQtutor. Second, you need a solution that PQtutor understands, i.e. input for a correct solution. Third, you need to write a multiple choice follow-up question, with the correct answer marked by an asterisk. For example, here is the information for the example 3.17 used throughout this paper:

    <h3>Example 3.17b: Determining the Mass of Solute in a Given Volume of Solution</h3>

    How many grams of CaCl2 are contained in 250.0 mL of a 0.200 M solution of calcium chloride?

    <h4>Answer:</h4>

    V_solution = 250.0 mL

    c[CaCl2] = 0.200 M

    m[CaCl2] = ?

    n[CaCl2]  = c[CaCl2] V_solution

    m[CaCl2] =n[CaCl2] M[CaCl2]

    Think about it: If this is an aqueous solution, we can estimate the density as about 1 g/mL.

    What is the mass of the solution, and what is its main component?

    a) 250g, CaCl2

    b*) 250g, water

    c) 4 kg, CaCl2

    d) 4 kg, water

PQtutor has no other knowledge about this question, and generates the feedback and prompts from the model solution and from the logic of the underlying mathematics. It does have some general knowledge about dimensions, units, and commonly used symbols for certain quantities covering the concepts covered in General Chemistry I and II.