ChemPrep : Self-Paced OWL Preparation for First-Semester General and Organic Chemistry

ChemPrep Demo Site 
 

Introduction 

General chemistry is often one of the first really challenging college courses faced by students.  We have found that students frequently overestimate their "readiness" to do well in this course.  Organic chemistry, on the other hand, is a course that many students approach with trepidation, either because of its reputation for being “tough” or because of the need for a very good grade.  To help students with a diverse range of needs and backgrounds succeed in general and organic chemistry we have designed a pair of online short courses¹.

ChemPrep/General Chemistry was developed as a self-paced preparatory course to help students review concepts, fill in content gaps, build confidence, and strengthen mathematical skills. ChemPrep/Organic Chemistry was designed to help students review the basic concepts of general chemistry that are used extensively in organic chemistry.  The curriculum for each course is listed in Table 1.

  Table 1: ChemPrep Curricula 
 

Each topic contains short information pages to introduce a concept or activity, interactive tutors and simulations to help develop the ideas more concretely, and homework questions that provide the opportunity to test understanding and see detailed, well-formulated solutions for each problem.

The ChemPrep courses are delivered over the internet using the OWL (Online Web-based Learning) system developed at the University of Massachusetts^2-9.  They are offered before the beginning of each semester (in January and July) and may be accessed from any location, 24 – 7, with a web browser.  Each course is designed to be complete, and no textbook is required or recommended.  Students work independently at their own pace, but are encouraged to interact with the course instructors via the OWL message system.  The average student takes approximately 20 hours to complete the general chemistry prep course and 10 hours to complete the organic prep course.  No formal grades are assigned and there is no penalty for non-participation or partial completion of the work.

The OWL System

The OWL electronic learning system has been used in the Chemistry Department at the University of Massachusetts for over eight years¹⁰.  Extensive databases of homework problems and interactive simulations, exercises, and tutors, have been developed by our faculty for general and organic chemistry.  These materials have been correlated to a number of textbooks and are licensed by Thomson Learning¹¹. 

The flexible authoring and course management tools in OWL allow for the creation and delivery of instructional materials via the web.  One of the most important pedagogical features within OWL is the support for mastery learning.  Students may work until they master a concept, and each time they attempt a problem a similar, but not identical, question is given.  This is accomplished in OWL by the powerful feature that allows authors to parameterize questions both chemically and numerically. 

Once a student submits an answer for evaluation OWL gives a detailed solution specifically parameterized to the problem.  Students may try the question again (similar but not identical) or move on to new material.  Contrast this to hand-graded homework where a student may turn in incomplete or even incorrect work, and, by the time they receive feedback, they are no longer engaged with the question.  Contrast this again to traditional end-of-chapter homework questions where it is difficult for students to test their understanding using a new instance of the question, and it is easy for them to memorize solutions to particular problems without seeing the general idea behind each. 

Instant, answer-based feedback has been shown to help students build confidence with course material in a non-threatening (and non-testing) environment¹².  The combination of parameterized questions with detailed, immediate feedback, and the requirement that a student correctly complete a question before credit is given, makes OWL a very powerful learning system. 

Additional features in OWL help to enhance the interaction between instructors and students.  The message system allows students to send questions directly to instructors about specific problems.  A detailed log of student activity makes it possible to see the particular instance of each question that a student has been assigned, and the responses that were entered.  Instructors can retrace a student's steps and help students diagnose where they are having difficulty.  Finally, instructors can collect data about completion of each assignment and use this to guide further in-class activities.

For ChemPrep students are particularly encouraged to use the OWL message system to ask questions or submit comments as they proceed through the material because they do not have direct contact with an instructor.  These messages are answered by an instructor, usually within 24 hours of submission.  Our students report that the combination of detailed content pages, interactive simulations, tutors and exercises, parameterized homework where they can retry problems until they master a concept, and the ability to send messages to the instructor, permits them to work at their own pace without the need for a textbook or lecture.

ChemPrep Study

The Spring 2004, off-sequence sections of first semester general chemistry and first semester organic chemistry were chosen to be our pilot groups.  The performance of participating students was compared to that of non-participants in the subsequent courses.  Based upon the encouraging results obtained with these groups, ChemPrep has been offered to all students enrolled in any of our introductory general and organic chemistry courses since that time.  These courses are referred to as: Nursing, General, General Honors, Organic and Organic Honors in the discussion that follows.  The ChemPrep curricula were identical for all three general chemistry courses and similarly for both of the organic courses¹³.  

Students pre-registered in one of these courses were invited via email to work on ChemPrep over the January break or during the summer.  They had until the first day of classes to complete the work (about 5 weeks) and their progress was recorded in the OWL system.   Online surveys were given at the end of ChemPrep and at the end of the subsequent course. 

Details of the pilot study for general chemistry have been presented at the Spring 2005 Meeting of the American Chemical Society¹⁴ and can be found in the preprint of a paper that has been accepted for publication in the Journal of Chemical Education¹⁵.  Details of the pilot study for organic chemistry are being prepared for submission. 

Completion Rate

The participation rate of students in each of the courses is summarized in Table 2, where "users" are defined as those students who completed more than 50% of the ChemPrep modules, and "non-users" as those who did not sign up for ChemPrep.  Students who signed up but completed less than half of the work in the prep courses were not included in the analysis.  Note the higher participation rate overall by organic students and by honors students.    

Table 2: Completion Rate

Women were more likely than men to participate in ChemPrep.  Data corresponding to the general chemistry courses are reported in Table 3.   

Table 3:  Percent Female Students by course 

Success Rates

We defined success in a course as a grade of "C-" or better.  The success rates for students in the Fall 04 courses are summarized below.  In all cases users out-performed non-users. 

Average Grades

Average grades based on a four-point scale were higher for users, as shown below.  

Discussion

Because the prep courses were entirely voluntary we sought to determine if the higher grades were due to ChemPrep or if they were simply a case of the better students choosing to take advantage of this resource to help them excel. 

We used math and verbal SAT I scores, and the algebra and trigonometry scores on the University’s Math Placement exam, as proxy measures for the students’ prior achievement and ability. We then determined the effect size for each of these measures as a standardized method of comparison.  Effect size (ES) is defined as the difference between the average score of user and non-user groups, divided by the standard deviation of the total class. 

      Effect Size =   (Avg Score User - Avg Score Non-user)

                                   (Standard Deviation Class)

            ES = 0, No difference between groups

            ES < 0, Users weaker than Non-users

            ES > 0, Users stronger than Non-users

Effect sizes with absolute values around 0.2 are considered small, 0.5 moderate, and 0.8 large by social science researchers^16,17.  Small effect sizes indicate that there is little difference in aptitude between user and non-user groups. 

Effect Size - General Chemistry

For every course, the effect size for the grade was higher than those of the prior achievement/ability measures.  While the students in GenChem and Nursing who elected to do ChemPreptended to be stronger than their peers (ES > 0 for SAT and Math Placement), they also appeared to benefit beyond what would be predicted by their entry skills.  As shown in the figure below, the effect sizes of all of the grades are positive and they range from moderate to large, indicating that there are meaningful grade differences between users and non-users in all of the courses.

In the Honors course, there does not appear to be a significant difference between users and non-users based on predictors of prior achievement/ability, but there is a large difference in grade performance.  Here, there seems to be a substantial positive relationship between ChemPrep usage and course grades.  This was a surprising result because the Honors group was stronger overall and we had expected there to be, at best, a small increase in course grades for users.  We were also surprised by the fact that a much larger percentage of Honors students completedChemPrep  (23% vs. 9% for GenChem and Nursing).  If we assume that this group has a larger proportion of motivated students then this result is reasonable. 

Effect Size - Organic Chemistry

In the standard organic chemistry courses it appears that ChemPrep had a greater effect than would be expected from the prior achievement/ability differences between groups.  However, in the honors course the grade difference is what one would expect from these measures.

Questions to Address

• How can we get students who really need a preparatory course to complete it ?

• What is best way to reach high school, transfer, and returning students ?

• Will we see similar results at other schools ?

• Does the curriculum meet the needs of potential users ?

Summary

On average, students who completed ChemPrep  had higher grades in the subsequent general and organic chemistry courses, with a greater percentage achieving a grade of C- or higher. Participation in ChemPrep was voluntary, and more women than men responded.  Students in the Honors and Organic courses enrolled in ChemPrep in higher percentages than students in GenChem and Nursing.  SAT and math placement exam scores were used as proxy measures of prior achievement/ability. Based on these, users in Honors GenChem were on par with their peers but performed better in the course than non-users.  Honors OChem users performed as would be expected from the predictor variables.  In GenChem and Nursing, ChemPrep helped students of high prior achievement/ability perform better than their achievement scores would predict. Weaker/less motivated students did not respond to the voluntary offerings of ChemPrepin the same numbers as stronger/more motivated students, and we are seeking alternate ways to reach this population. 

References and Notes

1.  Funding for this project was provided by the Professional Development Grant Program of the University of Massachusetts Information Technology Council.

2.  Stewart, Barbara N.; Vining, William J..  Analysis of student learning using online tutorials. In 229th ACS National Meeting, San Diego, CA, March 13-17, 2005. AN 2005:188556.

3.  Dufresne, R.; Mestre, J.; Hart, D.; Rath, K.  The Effect of Web-based homework on  Test Performance in Large Introductory Physics Classes.  Journal of Computers in Mathematics and Science Teaching. 2002 , 21(3), 229-251. 

4.  Vining, W. J.; Botch, B.; Day, R. O.; Hart, D.; Woolf, B. OWL web-based homework system: Improved student learning.  In 221st ACS National Meeting, San Diego, CA, April 1-5, 2001. AN 2001:199085.

5.  Vining, W. J.; Day, R. O.; Botch, B.; Woolf, B. Intelligent tutoring systems for general chemistry.    In 219th ACS National Meeting, San Francisco, CA, March 26-30, 2000. AN 2000:329052. 

6.  Woolf, B.; Hart, D.; Day, R.; Botch, B.; Vining, W. Improving Instruction and Reducing Costs with a Web-Based Learning Environment. Proceedings of the International Conference on Mathematics/Science Education & Technology, M/SET 2000. Pp. 410-415. 

7.  Hart, D.; Woolf, B.; Day, R.; Botch, B.; Vining, W. OWL: an Integrated Web-Based Learning Environment. Proceedings of the International Conference on Math/Science Education & Technology, San Antonio, TX, March, 1999. M/SET 99 Pp. 106–112. 

8.  Botch, B.; Day, R.; Vining, W.OWL: Online web-based learning.    In 216th ACS National Meeting, Boston, August 23-27, 1998. AN 1998:525480. 

9.  Day, R. O.; Botch, B. H.; Vining, W. J.; Woolf, B. P.; Hart, David M. OWL: Web-based interactive multimedia discovery and intelligent tutoring environments for general chemistry.  In215th ACS National Meeting, Dallas, March 29-April 2 1998. AN 1998:141481.

10.  OWL is currently used by over 20 departments on the UMass campus in disciplines ranging from physics and chemistry to resource economics and art history.  Examples of the OWL courses delivered at UMass Amherst can be found at http://owl.oit.umass.edu (accessed July, 2005). 

11.    OWL: General Chemistry and OWL: Organic Chemistry are licensed and distributed by Thomson Learning - Brooks/Cole Publishing and are used on over 150 college and university campuses nationwide.  A demonstration of the OWL system delivered through Thomson Learning can be found at http://owl.thomsonlearning.com/demo (accessed July, 2005).

12. Cole, Renée S.; Todd, John B. J. Chem. Educ. 2003 80 1338-1343.

13.  We were actually a bit surprised that the same curriculum worked for such diverse groups of students.  We attribute the general usefulness of these materials to the mastery learning feature in OWL.  Students who are comfortable with a topic can move through it quickly; those who are less familiar can cycle through as many questions as needed to solidify their understanding.

14.  Botch, Beatrice; Day, Roberta O.; Vining, William J.; Hixson, Steven; Samal, Peter; Hart, David; Peterfreund, Alan; Rath, Kenneth A.  ChemPrep: Self-paced OWL preparation for General Chemistry and Organic Chemistry.  In 229th ACS National Meeting, San Diego, CA, March 13-17, 2005. AN 2005:187164.

15.  Botch, Beatrice; Day, Roberta O.; Vining, William J.; Stewart, Barbara; Hart, David; Peterfreund, Alan; Rath, Kenneth A. “ChemPrep: A Self-Paced, Online Preparatory Course for General Chemistry” J. Chem. Educ, accepted.

16.  Cohen, J. Statistical Power Analysis for the Behavioral Sciences (Revised Edition).  New York: Academic Press. 1977. 

17.  Slavin, R. E.  Educational Researcher 1990 19(3); 30-34,44.