You are here

BestChoice, an interactive learning system: Supporting learning in large classes

Author(s): 

Sheila Woodgate and David Titheridge
The University of Auckland, Auckland, New Zealand

03/12/07 to 03/18/07
Abstract: 

BestChoice is an open-access interactive web site (www.bestchoice.che.auckland.ac.nz) that was developed initially to support learning in large first year university Chemistry classes at The University of Auckland. The BestChoice project has focused on exploring use of the web as a means of delivering scaffolded learning activities. More recently BestChoice has been expanded to provide learning opportunities for a wider ranger of users.

The model underpinning BestChoice learning activities is simulation of the interchange of a student with an experienced teacher. Thus student responses on BestChoice question pages generate instant assessment and instructive feedback. BestChoice is innovative in its emphasis on teaching both concepts and problem-solving strategies by guiding students in ways that promote their understanding.

Paper: 

BestChoice is an open-access interactive web site (www.bestchoice.che.auckland.ac.nz) that was developed initially to support learning in large first year university Chemistry classes at The University of Auckland. The BestChoice project has focused on exploring use of the web as a means of delivering scaffolded learning activities. More recently BestChoice has been expanded to provide learning opportunities for a wider ranger of users.

The model underpinning BestChoice learning activities is simulation of the interchange of a student with an experienced teacher. Thus student responses on BestChoice question pages generate instant assessment and instructive feedback. BestChoice is innovative in its emphasis on teaching both concepts and problem-solving strategies by guiding students in ways that promote their understanding.
 

View a selection of BestChoice Question Pages, by clicking on Live for 2007
For more background information on BestChoice and archived Question Pages click on Background


The journey to develop BestChoice as a learning tool and to establish high usage by students in our first year classes began in 2002 when the first web-based BestChoice modules became available. The extent of usage during 2002 indicated that, even though students who used BestChoice liked it, simply making modules available and advising students to use them was not sufficient to establish high usage. It was apparent that use of BestChoice had to be integrated into the fabric of our courses so that students viewed use of BestChoice as central to their success in the course as attending lectures or laboratory sessions. 

A selection of BestChoice modules was first made compulsory in two first year Chemistry courses in 2003. This began an on-going program of evaluation and consequent modification of the modules and the way in which they were used in our courses. Data supporting this study has been, for the most part, collected on-line and viewed in the stand-alone application BestChoice Reports. The studies were carried out with a view to achieving three objectives. 

Objective 1: To construct user-friendly activities that students like to use and which they perceive help them to learn
Objective 2: To develop methods for tracking how students use BestChoice in order to detect how better to use this method of supporting learning
Objective 3: To develop methods for integration of use of BestChoice into a variety of types of chemistry courses


The discussion of use in BestChoice in five first year university courses at The University of Auckland considers:

how BestChoice is introduced to the students in each course 
how BestChoice modules can be used to support course work 
how various courses use BestChoice as a required course component 
how BestChoice data can be used to find 
the workload implications of compulsory use of BestChoice
the extent of usage of compulsoryBestChoice modules
the extent of usage of non-compulsoryBestChoice modules
the student response to BestChoice
the effect increasing the BestChoice compulsory component

how progress has been made towards achieving the three objectives
how the results suggest changes in one course 
how the results provide background for our future work

Course coordinators and lecturers introduce their students to BestChoice

Although one of the authors has had extensive experience teaching first year Chemistry, neither of us is currently lecturing in any of the first year Chemistry courses. Therefore, achieving extensive use of BestChoice has required the support of the academic staff who coordinate and lecture in the first year Chemistry papers at The University of Auckland. These academic staff

  • identify BestChoice modules that are relevant to their course objectives. These are arranged in a BestChoice course.
  • introduce their class to BestChoice, providing details of the registration process in course materials.
  • promote BestChoice activities as learning experiences.
  • encourage students to use BestChoice activities to support their learning during the entire course

The encouragement provided has been, for the most part, making completion of BestChoice activities part of the course requirements. Compulsory BestChoice modules with marks assigned to them both introduce students to BestChoice and make them aware of an extensive selection of non-compulsory modules that support their course. A schedule of due dates for the compulsory modules is provided with the course material. 

None of the courses in the study are "on-line" courses. They are taught in a conventional way with some support from our in-house course management system (like Blackboard). Typically four or five lecturers are involved in each course.

BestChoice is not a significant feature of lectures in any of the courses that have compulsory modules. Students in one of the courses are reminded of these due dates by posting a weekly schedule both in lectures and on our course management system. One of the ten lecturers involved has included reference toBestChoice activities in lecture handouts.

BestChoice modules have been used as assignments or pre-labs

Assignments

BestChoice assignment modules are linked to course material. The scheduled completion date for each assignment is soon after that topic has been covered in lectures. Assignments had not previously been part of course requirements for any of the advancing courses in which BestChoice is used. However, practice problems and model answers had been distributed, and this practice has continued. 

BestChoice assignments are seen as additional support for problem-solving because users can solve problems step-wise with instant marking of each response. The feedback associated with each response has been designed to give students insight into problem-solving strategies of subject experts and to help them identify the source of their difficulties. Completing a BestChoice assignment is intended to be a bridge between watching the lecturer solve a problem in class and doing the problem without support. 
 

Pre-labs

BestChoice pre-lab modules have an obvious connection to laboratory work and include questions on experimental procedures and equipment. A completed and printed BestChoice page may assist students with experimental work. For example, on the last page of one of the BestChoice pre-labs, users construct a scheme designed to assist in the identification of an anion unknown. 

In all of the laboratory courses using BestChoice, students use fill-in lab report sheets that are submitted to be marked at the end of the period. Inevitably, this puts pressure on weaker students at the end of the laboratory period. Thus the printed last page of BestChoice pre-labs commonly give students a head-start on calculations that would have to be completed during the laboratory period. For example, in the pre-lab for an initial rates experiment, the initial concentrations of reacting substances for all runs are calculated. In the pre-lab of a synthesis experiment, the theoretical yield is calculated as well as the molar mass of the product so that once the mass of product is available, calculation of the actual yield is straightforward.

BestChoice pre-lab modules have replaced paper pre-labs that were marked at the beginning of the laboratory period. The on-line marking of BestChoice prelabs reduces workload for supervisors at the beginning of the period so that they can focus on helping students with practical work. Furthermore, the supportive nature of the BestChoice system through the feedback and instant marking means that students arrive at the laboratory session with a marked and corrected page. They know whether their answers are correct before they come to the laboratory, and more importantly, have some indication of the source of their errors.
 

Marking of the compulsory BestChoice modules

The assignment modules are marked according to the number of correct answers entered. BestChoice is learner-friendly in that wrong answers may be corrected without leaving the page, and users are given credit for the corrected answers. These modules are intended to be learning activities, not tests. Students who persevere can, without too much difficulty, gain a score of 100%. Through completion of the BestChoice assignments students can gain a maximum of 5% of the marks for the course.

Pre-lab modules are not given an explicit mark. Students get credit for the module by bringing a printed version of the completed last page to their laboratory session. This printout includes a header inserted by the system. This displays the student's user name and the date on which the page was completed. 

BestChoice has been used in a variety of courses

How large are they? 
What is the background and aspirations of the students?

In 2006 use of BestChoice was compulsory in five first year Chemistry courses, one of which is offered in both semesters. As shown in Table 1, four of these are large classes. The number of students quoted is the number who completed all course requirements. The total of these is 1906. Some students are enrolled in more than one course; however at least 1300 individuals are involved. 

Students in these courses have a wide range of abilities and backgrounds in chemistry. As shown in Table 1 students advancing in chemistry are part of the cohort for two of the courses. CHEM 110 in the first semester has the highest proportion of well-prepared students because this course is compulsory for competitive entry to Biomedical programmes. Based on grade point averages from first semester courses, the second semester CHEM 110 and CHEM 120 cohorts are similar. 
 

TABLE 1
Class details
Students
Cohort
Course content
Assignment 
modules
Prelab 
modules
Courses taken by students advancing in Chemistry 
(referred to as advancing courses)
CHEM 110
first semester
920
Chemistry major
Biomedical Science major
Organic Chemistry
Kinetics
Acid-base Equilibria

0.5 marks each
5
second semester
246
Chemistry major 
Science major
CHEM 120
357
Chemistry major 
Science major
Inorganic Chemistry
Gas and Solubility Equilibria
Electrochemistry
Thermodynamics
10
0.5 marks each
6
Courses taken by students with weak background in Chemistry
(referred to as remedial courses)
CHEM 150
233
Catch-up Chem Chemistry fundamentals
10
0.5 marks each
0
CHEM 91F
78
Foundation Chem
3
1 mark each
0
CHEM 92F
72
Foundation Chem
4
1 mark each
0


The students in CHEM 150 are the most able of those in the remedial courses. They are first year university students who do not have a strong enough background in Chemistry to take advancing first year papers. CHEM 91F and CHEM 92F students are enrolled in a pre-university programme.
 

Are there any differences between the courses in the way the modules are used or configured?

As shown in Table 1, the advancing classes use both pre-lab and assignment modules. The remedial classes use only assignment modules. 

Two of the classes with large BestChoice requirements are taken by students in their first semester at university (CHEM 110 and CHEM 150). CHEM 110 students have no pre-lab modules for the first laboratory session; however students are asked by their laboratory superivsor whether they have logged into BestChoice, and if they have not, they are shown how to do so. The first compulsory module for CHEM 110 is in week 3. The first compulsory module for CHEM 150 is due in week 2, and there is no association of BestChoice with the laboratories in this course. 

CHEM 120 is taught in the second semester. A significant proportion of CHEM 120 students will have taken or are taking a second chemistry course that requires use of BestChoice (CHEM 110 or CHEM 150). The first compulsory module for CHEM 120 is a pre-lab in week 2.

CHEM 110 required modules differ from the rest in that the Show Next/Give Up option is turned off. The Show Next/Give Up link is a feature of BestChoice that enables students to display either the next correct answer or give up on the entire page. In principle, students could use this option to find out the correct answer and then return to the page to enter it and gain the mark. 

BestChoice data can be used to give evidence for the success of the requiring BestChoice usage

Introducing a new assessed course component for a group of large classes is significant stepand it is appropriate to have in place means to evaluate both the effectiveness of this component in achieving the purpose for which it was intended and the success of integration of the component into the class requirement. Our evaluation that has focused on the four areas shown below. The questions posed are answered below by analysis of BestChoice usage data and student-entered responses.

Workload implications

How long does it take students in the various classes on average to complete the BestChoice requirement?


% Compulsory modules completions 

Are there significant differences between completion of the assignment and the pre-lab modules for the same class? for the different classes? What is the typical mark distribution for the entire compulsory component?


% Non-compulsory modules completions 

How does the extent of non-compulsory and compulsory usage compare? Is this the same for the different classes? Is there a correlation with the students abilities and backgrounds in chemistry?


Student perception of BestChoice

Is there a difference between the various classes? Is there a correlation with the students abilities and backgrounds in chemistry?


The consequences of an increase in the required BestChoice component 

How does this affect usage of the compulsory modules? the noncompulsory modules? student response?

The time taken to answer BestChoice question pages can be measured

Table 2 shows the total number of answers in the required BestChoice component for four of the classes is large. Most BestChoice screen pages require the user to enter multiple answers. Pages may have up to 35 answers, but on average, in these BestChoice courses there are 5 answers required per screen page. 

The BestChoice system records the time taken to answer each question page for each student, and the average over the entire course is displayed in BestChoice Reports. These times have been summed for the compulsory modules for each course. Table 2 shows that, for the large classes, on average completion of the required BestChoice question pages takes 25 minutes per week over the time period during which compulsory modules are due. Thus a large BestChoice requirement does not translate to a significant increase in student workload. 

The CHEM 91F and CHEM 92F classes are small, and their BestChoice requirement is also small, but it is interesting that where compulsory modules are the same as in CHEM 150, these cohorts take about the same time to complete these.
 

TABLE 2
Time taken
Students cohort
total number of 
answers avaiable
total time taken
time taken per week
(10 week semester)
Courses taken by students advancing in Chemistry 
(referred to as advancing courses)
CHEM 110
first semester
920
Chem major
Biomedical Science
538
209 minutes total
21 minutes per week
second semester
246
Chem major 
Science major
CHEM 120
357
Chem major 
Science major
769
262 minutes total 
26 minutes per week
Courses taken by students with weak background in Chemistry
(referred to as remedial courses)
CHEM 150
233
Catch-up Chem
571
228 minutes total 
23 minutes per week
CHEM 91F
78
Foundation Chem
108
86 minutes total
CHEM 92F
72
Foundation Chem
223
71 minutes total

The extent of completion of the pre-lab and assignment modules can be measured

Is there a difference in completion percentages for the pre-lab and assigned modules?
Is there any correlation between completion percentages and the background and ability of the student cohorts?

Data used in the analyses
The analyses below use student numbers as well as data that can be viewed in BestChoice Reports. This includes the number of students that attempt each module and the average percentage of the total marks gained by this set of students for a particular module. Each required response in BestChoicemodules has been assigned 1 mark.

Advancing courses 
The BestChoice requirement for CHEM 110 and CHEM 120 includes both assignment and pre-lab modules. As described above, the marking of these modules is different, with pre-labs receiving no explicit mark. On the other hand, the pre-lab modules are associated with a regularly scheduled event. Students know that they will be asked by their supervisor for the printed version of the last page of the module. Is this sufficient incentive for them to complete the entire module? 

Table 3 shows the results of analysis of usage of the two types of modules during 2006 for the advancing courses. The % attempted and % completed has been averaged over all compulsory modules for each courses. Users who complete have gained 100% of the marks available. 
 

TABLE 3
Usage in advancing courses
cohort
Students
for an assignment module
for a prelab module
% attempted
average % completed
% attempted
average % completed
CHEM 110
first semester
Chem major
Biomedical Science
920
88
86
99
81
second semester Chem major 
Science major
246
89
86
99
84
CHEM 120 Chem major 
Science major
357
82
80
93
79


These data show for students in the same course the difference in % completed for the prelab and assignment modules is 1-5%. Thus it appears that most students complete the pre-lab module despite not being assessed on anything but the last page.

% Attempted is somewhat higher for the pre-lab modules, and the % completed is somewhat lower. Does this mean that significant numbers of students are doing only the last page in the pre-lab modules? The answer is no. BestChoice also collects data showing the number of users who attempt any page of a module. Characteristically in the pre-lab modules the last page has been attempted by fewer students than the earlier pages in the module.

These average % completed columns show that there are not significant differences between the courses despite the CHEM 110 first semester cohort being more adequately prepared and probably more motivated than either of the second semester cohorts. 


Remedial courses 
The usage of the assignment modules in the remedial courses is shown inTable 4. An average of the corresponding data for the advancing courses is included for reference. The small remedial cohorts in CHEM 91F and CHEM 92F have comparable or even better average % completions, but CHEM 150 has the lowest average % completion of any of the courses.
 
TABLE 4
Usage in remedial courses
cohort
enrolled in paper
assignments
% attempted
average % completed
Advancing courses
 
 
86
84
CHEM 150
Catch-up Chem
233
77
74
CHEM 91F
Foundation Chem
78
96
93
CHEM 92F
Foundation Chem
63
88
85

 

What is the distribution of marks for the entire BestChoice component?

The overall mark that students receive for the BestChoice requirement is cumulative over up to 16 modules. Typically 50-60% of students get full marks for the BestChoice component, and 75% get a pass mark (more than 50% of the total). This is consistent with the pass rate in other required sections of these courses.

The usage of non-compulsory modules can be measured

BestChoice is a large resource. Table 5 shows that the number of marks that can be gained through answers in the compulsory modules is less than 25% of the total available for all of the courses. All non-compulsory modules are also relevant to course content. 

To what extent has the introduction of students to BestChoice through compulsory modules achieved noncompulsory usage? Table 5 also shows the number of marks gained on average from non-compulsory usage by students in each of the courses.
 

TABLE 5
Noncompulsory usage
Cohort

Average marks 
obtained/available
Average total marks 
obtained/available
Course
Compulsory
Non-compulsory
Courses taken by students advancing in Chemistry 
(referred to as advancing courses)
CHEM 110 
semester 1
Chemistry major
Biomedical Science
489  /  538
364  /  2431
853  /  2969
semester 2 Chemistry major 
Science major
515  /  538
324  /  2431
839  /  2969
CHEM 120 Chemistry major 
Science major
610  /  769
298  /  3073
908  /  3718
Courses taken by students with weak backgrounds in Chemistry 
(referred to as remedial courses)
CHEM 150 Catch-up Chemistry
411  /   571
90  /  1461
501  /  2032
CHEM 91F Foundation Chem
204  /  223
54  /  675
258  /  898
CHEM 92F Foundation Chem
144  /  171
132  /  973
276  /  1144


The extent of non-compulsory usage is significant in the advancing classes. These students complete 25-28% of the available BestChoice answers. Of the remedial classes, CHEM 92F has the most significant non-compulsory activity. This analysis has highlighted that of the large classes using BestChoice,CHEM 150 is the only one where the average marks gained is less than the total of the compulsory marks.

Evidence of student response to BestChoice can be collected

What data is collected?
How is it collected?

Each BestChoice module has a survey on the last page.  This has a free-text response box into which comments can be typed. In addition, the users can rate the module as to how helpful they found it by clicking on one of the radio buttons. Thus survey entries are made immediately after the BestChoicemodule is completed so that the BestChoice experience is still fresh in users' minds. 
 

Analysis of comments

Large numbers of comments have been received. In order to facilitate analysis, as they are entered, the comments are assigned by the users to the categories shown. A check on the validity of the assignments has shown that most assignments are appropriate. Three of the comment categories (compliment, suggestion and criticism) are relevant to teaching and learning. 75% of comments entered have been assigned to one of these. 25% of comments entered either do not pertain to BestChoice or highlight typographical errors and bugs. 

929 comments relevant to teaching and learning were entered in 2006 by the students in the courses discussed above.  Table 7 shows the response rate for each class and the distribution of the comments in the compliment, suggestion and criticism categories. Many of the comments relevant to teaching and learning have multiple threads. Any comment that includes a compliment is assigned to Compliment. Critical comments that include a suggestion are assigned to Suggestion.
 

TABLE 7
Comments
Cohort
Enrolment
Comments entered
Response
rate*
%
Compliments
%
Criticisms
%
Suggestions
Class
Courses taken by students advancing in Chemistry 
(referred to as advancing courses)
CHEM 110
semester 1
Chem major
Biomedical Science
1023
401
3%
71
10
19
CHEM 110 
semester 2
Chem major 
Science major
304
106
3%
75
6
20
CHEM 120 Chem major 
Science major
386
154
2.3%
80
13
7
Courses taken by students with weak background in Chemistry
(referred to as remedial courses)
CHEM 150 Catch-up Chem
262
186
9%
72
9
19
CHEM 91F Foundation Chem
78
39
8%
72
10
18
CHEM 92F Foundation Chem
72
43
12%
72
13
18
*based on total module completions


Table 7 shows that response rates in the remedial classes are significantly higher than in the advancing classes. The response rate for CHEM 150 is consistent with the other remedial courses. 

The distribution of comments between the various categories is similar for all classes.  The message of student satisfaction is strong with % compliments being over 70% for all cohorts and % criticisms being less than 15%. The extent and quality of the student feedback has been most pleasing. Consider the set of comments below. Most of these were entered over a 12 hour period by students enrolled in second semester classes, both advancing and remedial. 
 

18 Aug 06        Periodic Table
i thoroughly enjoy doing this it helps me understand things better.
19 Aug 06        Pushing Arrows
This was a very great module with clear explanations and excellent notes. Thank you!!!
19 Aug 06        Structure
enjoyed best choice by doing questions it helped me to understand the topic abit better..
19 Aug 06        Rates
The exersizes were really helpful. My understanding was a bit below par, but i believe with more practice i can only get better. My only improvment would be to ask for more exersizes...other than that it was fine
19 Aug 06        Pushing Arrows
I found this compulsory module very, very helpful. It helped reinforce what I had learnt and helped me understand the concept of curly arrows even better than before. The review sections helped clear some of the doubts and uncertainty I had about this topic. Now I have a much clearer understanding about the concept of curly arrows thanks to this compulsory module.
19 Aug 06        Quantum Numbers
I was so confused about all those quantum numbers and shells, subshells, couldnt visualise atomic structure beofre doing this module. Its finally clear
19 Aug 06        Pushing Arrows
I found this topic great, it really helped to consolidate my learning
19 Aug 06        Pushing Arrows
I think this bestchoice website is awesome. It gives me a chance to pratice using the concepts that are taught in the lectures. Keep it up!
20 Aug 06        NMR
This compulsory module was extremely helpful. It cleared a few of the doubts I had about this topic. I found this module very useful in revising for this topic.

The criticisms and suggestions that students enter are also instructive. It is noteworthy that CHEM 150 has the lowest percentage of criticisms. Again it would appear that CHEM 150 students who use BestChoice find it helpful. 

Analysis of end-of-module ratings:

Students in these classes entered 7127 ratings in 2006. Table 8 shows an analysis of the ratings using a scale of 1 (worst) to 6 (best).  
 

 

Table 8 
Module ratings
cohort
responses
average/6
response
rate*
% positives 
(4, 5, 6)
%
6's
Courses taken by students advancing in Chemistry 
(referred to as advancing courses)
CHEM 110 
first semester
Chem major
Health Science
3378
4.9
25%
87%
38%
second semester Chem major 
Science major
872
4.75
20%
83%
37%
CHEM 120 Chem major 
Science major
1438
4.7
20%
82%
34%
Courses taken by studentswith weak backgrounds in Chemistry 
(referred to as remedial courses)
CHEM 150 Catch-up Chem
977
4.6
47%
82%
31%
CHEM 91F Foundation Chem
188
4.9
38%
89%
34%
CHEM 92F Foundation Chem
274
4.8
54%
84%
38%



The response rate is again higher for the remedial classes. CHEM 150 has the highest response rate for courses with large enrollments again indicating that those CHEM 150 students who used BestChoice found it helpful. 

The distribution of ratings again indicate high student satisfaction.  The highest rating was the most common response, greater than 30% of the total responses for all classes.

The effect of increasing the BestChoice requirement can be determined

Effect on module completions and noncompulsory usage

Two of the courses described above had the BestChoice requirement increased significantly in 2006. Assignments were added to the CHEM 110 requirement, and pre-labs were added to the CHEM 120 requirement. The change in the number of required responses/marks is shown in Table 6 along with data that compares the usage of the 2005 group with the 2006 group. 

Because separation of data into the different classes was only possible after mid-year 2005, the CHEM 110 comparison uses only semester 2, 2005 data. This should not alter the validity of the comparison because of the similarities observed above in the CHEM 110 semester 1 and semester 2 courses for 2006.

 

TABLE 6
students 
compulsory marks 
average %marks
gained
average marks
gained
 
 
percentage of class
CHEM 120 assignments prelabs  
total
noncompulsory >300 pages >500 pages
2005 
(2006)
346
(357)
363 
(769)
80% 
(80%)
none 
(79%)
 
584
(908)
202
(298)
 
15%
(25%)
4%
(7%)
CHEM 110 semester 2
 
 
         
 
       
2005 
(2006)
246
(260)
329 
(538)
none
(86%)
67%
(84%)
 
437
(839)
296
(324)
 
1%
(22%)
0
(2%)


Table 6 shows that in CHEM 120 the increase in the BestChoice requirement had no effect on the extent to which the assignment modules were completed and a positive effect on the extent of non-compulsory usage. The pages in the table are screen pages. These are another way of keeping track ofBestChoice usage. 

In CHEM 110 the average %marks on the pre-lab modules increased when assignment modules were introduced, and there is also a positive effective on non-compulsory usage measured either in terms of marks or pages.
 

Effect on student response

Tables 8 and 9 show a comparison of the user-entered feedback for the second semester CHEM 110 courses in 2005 and 2006 and for the CHEM 120 courses in the same years. The percentage increase in the numbers of comments and survey ratings entered by the CHEM 110 students is similar to the percentage increase in the required component. In contrast, the number of comments and survey ratings entered by the CHEM 120 students was similar in 2006 to what it was in 2005 despite the large increase in the BestChoice component. 
 

Table 8 
Module ratings
Cohort
Responses
Average/6
% Positives
(4, 5, 6)
% 6
CHEM 110 2005 
(2006) 
second semester
Chem major
Health Science
487
(872)
4.81
(4.75)
85%
(83%)
38%
(34%)
CHEM 120 2005
(2006)
Chem major 
Science major
1301
(1438)
4.6
(4.65)
82%
(82%)
30%
(34%)


 

TABLE 9 
Survey comments
Cohort
Comments entered
%
Compliments
%
Criticisms
%
Suggestions
CHEM 110 2005 
(2006) 
semester 2
Chem major 
Science major
59
(106)
76
(75)
10
(6)
14
(20)
CHEM 120 2005
(2006)
Chem major 
Science major
169
(154)
84
(80)
14
(13)
2
(7)



Although response rates for both survey entries are somewhat lower for the CHEM 120 course having the increased requirement, the distribution of module ratings has not changed significantly. Similarly neither the distribution of comments or the content of the critical comments indicate lower student satisfaction with BestChoice.

Significant progress has been made towards achieving our overall objectives

Objective 1: To construct user-friendly activities that students like to use and which they perceive help them to learn

Analysis of the overall usage of BestChoice modules and of the student-entered response has been on-going for four years. These analyses have been used to prioritize enhancements to the BestChoice system and upgrades of content in BestChoice modules. The change in the module ratings and the distribution of comments during these four years points to the fact that we are moving steadily towards achieving Objective 1.
 

Module ratings using a 6 point scale
Year   Total responses Average ratio of 6's to 5's % 6
2006 9698 4.74 1.22 35
2005 6866 4.64 1.08 32
2004 6442 4.71 1.03 32
2003 2010 4.53 1.01 29

.

Comments entered in on-line survey by 
University of Auckland students
Year Total responses Compliment Suggestion Criticism
2006 1173 72% 17% 11%
2005 716 72% 11% 16%
2004 667 71% 13% 16%
2003 235 58% 12% 30%


Further evidence for student satisfaction is the overall increase in average noncompulsory usage. Unlike the survey data, this measure of student approval includes all students in the class. The increase in noncompulsory marks gained by CHEM 110 and CHEM 120 students over 2005 and 2006 is particularly pleasing in light of the large increase in the BestChoice compulsory requirement over this time interval. 
 

Objective 2: To develop methods for tracking how students use BestChoice in order to detect how better to use this method of supporting learning.

During the period 2003-2005 it become increasingly apparent that that data collected when BestChoice is used could help us to improve both the BestChoice modules and their usage. The application BestChoice Reports was first available in 2006. Data viewed in BestChoice Reports has informed this study of methods for integrating BestChoice into coursework, and has as well as provided direction for enhancements to the system and its modules. 

BestChoiceVarious methods have been trialled between 2003 to 2006 for integration of use of BestChoice into chemistry course work. The most successful of these has been assignment of compulsory modules. Completion percentages for all but one of the courses in the study are above 80% despite the wide variation of the various cohorts with respect to their abilities and backgrounds in chemistry . We have found that either of two methods of checking for completion (viewng on-line scores or a print-out of the completed last page) are equally successful. 

Our results suggest a change in the strategy used in CHEM 150

According to our survey responses, BestChoice was popular with CHEM 150 studentshowever surveys are voluntary. Our % completion data showed that unlike the other classes, 5% of CHEM 150 students did not become active users of BestChoice. This would have contributed to the fact that the % completion of compulsory modules was 5% lower than for any of the other classes. Getting weaker students to use learning resources is a well-recognized challenge. Better integration of BestChoice into CHEM 150 course activities may achieve better usage. 
 

Should references to BestChoice be included in CHEM 150 lecture notes?

This has been tried for one section of the course without the desired effect. Indeed CHEM 150 was the only course where students had reference to BestChoice activities in lecture notes. In our experience, the fact that including reference to BestChoice in CHEM 150 print materials has not been successful in achieving high usage has nothing to do with the weaker backgrounds of the CHEM 150 students. We have had a similar result for a cohort of very able high school students. 
 

Have the critical comments entered by CHEM 150 students given us any clues?

Three of the 36 critical comments entered for CHEM 150 this year mentioned that due dates for compulsory activities should be more widely advertised. These comments and the results achieved when pre-labs were added to CHEM 120 have encouraged us to introduce pre-lab activities for CHEM 150 in 2007. The alignment of BestChoice activities with a scheduled compulsory event may lead more students to attempt both compulsory and the noncompulsory modules. 

Conclusion

The BestChoice project began as an exercise in creating web-based activities based on scaffolded learning. Through a desire to obtain student feedback on these activities and the system that delivers them, a survey was included. Because limited numbers of students complete surveys, we also collected a variety of usage data. 

Through analysis of these data it has become apparent that web-based learning activities can be evaluated more extensively and more easily than other course materials. These evaluations afford us an opportunity to find out how students use the activities and what they do and do not like about them. This is very important for both the developers and the academics involved in the courses that use BestChoice. This study is one example of how the data can be used. 

It is gratifying that the data has endorsed our methods for integrating use of BestChoice into most of our courses. For the one class where the integration had not been as successful, they have suggested ways to modify our strategies. Some of the results were surprising. For example, we expected a greater dependence on the nature of the class cohort. The fact that we did not provides some evidence for the success of a scaffolded learning approach for all students, regardless of their abilities and backgrounds in chemistry. 

As the number of BestChoice users increases and the data set grows, the analyses that can be carried out become increasingly fine-grained. We have already progressed from considering the entire cohort of BestChoice users to comparing different cohorts of BestChoice users. The next step is to analyse user responses to questions in individual BestChoice modules. We are collecting data that enables us to do this, and have some evidence that analysis of this data gives insights into student difficulties. These insights can in turn be used to guide us to make BestChoice modules even more learner-friendly. It is exciting to have created a learning resource, the development of which is being continually informed by its users.


The authors would like to acknowledge the support of the course coordinators: Drs Judy Brittain, David Salter and Malini Arewgoda. We would also like to acknowledge financial support from The University of Auckland, The Ministry of Education in New Zealand and The Royal Society of Chemistry in the United Kingdom.

CONFCHEM on-line conferences are organized by the ACS Division of Chemical Education's Committee on Computers in Chemical Education ( CCCE ). Send additions or corrections for this page to John H. Penn at John.Penn@mail.wvu.edu .