You are here

BestChoice, a Model for Interactive Web-Based Teaching

Author(s): 

Sheila Woodgate and David Titheridge
The University of Auckland, Auckland, New Zealand

05/22/06 to 05/26/06
Abstract: 

The BestChoice online tutorial system designed by the authors has been used since 2002 in New Zealand to support both first-year university students and high school students in their learning of chemistry (4000 active users in 2005). 80 modules with content appropriate to the above cohorts have been written. In total there are 2500 screen views and 6000 possibilities for interaction generating feedback available. A selection of these can be viewed at www.bestchoice.che.auckland.ac.nz by clicking on the DEMO MODE link.

The primary focus of BestChoice modules is interactive teaching using a programmed learning approach. We have made major strides in developing methods to simulate on screen pen-and-paper solutions to multistep problems. The paper will describe
factors taken into account in the design and implementation of BestChoice
our experience incorporating use of BestChoice into first year university and high school courses
how evaluation by over 8000 users has provided validation of the approach taken in BestChoice modules as well as useful insights for future work

Paper: 

BestChoice is an open-access interactive web site (www.bestchoice.che.auckland.ac.nz) that was developed initially to support learning in large first year Chemistry classes at The University of Auckland. More recently BestChoice has been expanded to provide learning opportunities for a wider ranger of users. The model underpinning BestChoice learning activities is simulation of the interchange of a student with an experienced teacher. Thus student responses on BestChoice question pages generate instant assessment and feedback. BestChoice is innovative in its emphasis on teaching both concepts and problem-solving strategies by guiding students in ways that promote their understanding.
 

View a selection of BestChoice Question Pages, by clicking on Live (during 2006). 
View screen shots of BestChoice Question Pages by clicking on For Archive (after 2006).


The discussion below considers the 

 

Design considerations

Facilitating both teaching and learning using web-based activities

Teaching Chemistry to large classes at first year university is largely transmission of information to students who adopt a passive role. However, for quality learning to occur, the student must assume an active role and engage with the content through problem-solving. Often the subject expert is not readily accessible to offer guidance to the student during the learning process. This may cause

  • uncertain learners to become discouraged and to conclude that chemistry is "too hard".
  • confident learners to solve problems algorithmically without developing full understanding.

BestChoice activities have been designed to support the learning process by acting as a bridge between "hearing about it in lectures" and "doing it yourself". The authors' aim was

  • to construct activities that students like to use and which help them to learn
  • to develop methods for tracking how students use the activities in order to detect how better to support learning

The process by which we have achieved these objectives is described below.
 

Establishing a connection between web-based learning activities and other study resources

Traditionally, students study using print materials. They solve problems using pen and paper. Many students prefer to have model answers readily accessible.

If BestChoice activities are to be valued as learning tools, students should be able to relate these activities to other study tools. The authors have established this connection by

  • constructing BestChoice pages that are visually appealing and use conventional symbolism in display of mathematical and chemical concepts
  • developing activities that help students learn how to solve problems that appear on written assessments
  • giving users ready access to correct answers and to background material required.

Overcoming some of the limitations of teaching over the web

Reading from the screen is more difficult than reading from a printed page.

BestChoice has been designed to present information in a way that prevents cognitive overload. The emphasis is on requiring users to answer questions. Content is then exposed in small quantities in the feedback generated by the user's response.


Waiting for downloads or for the computer to respond is frustrating.

BestChoice pages are text-basedGraphics and animations are used only where necessary. This allows BestChoice to be used on a dial-up connection.


Common web browser answer-input devices (text-boxes, dropdown lists) do not support subscripts and superscripts. 

This limitation has been overcome by developing answer input devices having support for both images and formatted text.

 

Constructing BestChoice modules to simulate a one-on-one tutorial

An experienced teacher would, in a one-on-one tutorial, mix discussion of concepts with questions to probe the student's level of understanding. Concepts are presented as Review Pages in BestChoice. These are typically brief and highlight only major principles.
 

Some examples of Review Pages can be viewed by clicking Live (during 2006). 
Screen shots of Review Pages are available on clicking For Archive (after 2006).


Most modules begin with a Review Page. Concepts are then developed systematically through the

  • questions on Question Pages that follow each Review Page.
  • feedback displayed when users answer questions on Question Pages.The feedback either relates to the previous Review page by reiterating principles or presages the next Review Page by applying and extending the principles.

Thus, during completion of a BestChoice module, learning is scaffolded by the constant feedback and guidance provided by the system. Some students struggle to understand what the question requires or may not be able to enter the correct answer. These BestChoice users have the option for the system to show them the correct answer so that they can overcome this barrier and move on to complete the problem.
 

Connecting the BestChoice activities with the student's course of study

Teachers choose course resources to help students study. In order for BestChoice learning activities to be accepted as a course resource, we constructed groupings of modules for target groups both in first year Chemistry courses at The University of Auckland and in Chemistry classes at New Zealand high schools. We also constructed a General course (aimed at General Chemistry courses) and an Organic Chemistry course for users outside the target groups. 
 

Implementation of BestChoice

BestChoice for the web 2002

The BestChoice web site went live in 2002, with a total of 1900 pages in 60 modules available. The content was taken from a DOS version of BestChoice that had been developed by one of us. 90% of the pages were Question Pages; thus the emphasis was, as intended, on users entering responses that generate feedback. 

The use of BestChoice was recommended to students in two first-year courses at The University of Auckland in Semester 2, 2002. An end-of-semester survey in one of these courses endorsed the approach taken in BestChoice modules. The survey asked "What feature of BestChoice did you like best?". The two most common responses to the question were "combination of questions and review pages" and "feedback".

While we had put in place the structure described above and had some validation of the BestChoice approach to supporting learning, BestChoice 2002 was limited to one answer per question page; thus multistep problems extended over several pages. Furthermore, the lack of control over the positioning of the answer fields and their associated feedback was a severe hindrance in authoring. 
 

BestChoice for the web 2003-2006

Both the BestChoice content and the BestChoice system have been upgraded and expanded on a continual basis during the last three years. BestChoice currently has 2500 different pages in more than 100 modules with 6000 possibilities for interaction that result in instructive feedback. A variety of new features have been developed so that the author can

  • construct Question Pages having any number of answer fields.
  • control the positioning of the answer fields and their associated feedback on Question Pages.
  • use a variety of types of answer fields on the same Question Page.
  • use correct answers to one part of a Question Page to trigger the appearance of a second part.

The BestChoice Question Pages accessed using the links below show that we have made significant progress toward mimicking stepwise problem-solving for a variety of problem types.
 

To see actual BestChoice Question pages, click on Live (during 2006) 
To access screen shots of BestChoice Question pages, click on For Archive (after 2006).

 

Evaluation of BestChoice using an on-line survey

Establishing mechanisms for on-line evaluation of BestChoice by students

We wanted our student users to indicate what they liked about BestChoice and how BestChoice could be improved. Therefore we incorporated on-line evaluation mechanisms into theBestChoice system, in order to get a continual feedback from our users. 

In BestChoice 2002 students could enter feedback using a text form accessed by clicking a Comments button that appears at the top of every page. Very few comments were received. As a consequence, for 2003 we added the survey below on the last page of each module. This allows the user to enter comments and to rate the modules on a six-point scale. Each user could enter one rating and one comment per module. 
 

image

 

The comments entered in the survey indicate the student perception of BestChoice

This on-line survey has been a very rich source of student comment, most of which is pertinent to teaching and learning. 2100 complimentary comments were entered during 2003-2005. A few unedited examples are given below. 
 

User 1: Wonderful! Sooooo helpful! Best feature is that you can read the theory, then do the quiz! (3 Nov 03)
User 2:This was fun and helped me understand much easier. I was not placed under pressure at all, because I was allowed to make mistakes. I really enjoyed this. (22 Mar 04)
User 3: This has made so much that I didnt understand much clearer. The little amounts of information followed by heaps of questions makes it really easy to absorb!(04 Aug 03) 
User 4: Man, you have no idea how helpful Bestchoice is....the questions on them are relavent to what we are learning and to make it even more helpful....the hint and give up function help alot....thank you so much! (15 Jun 03) 
User 5: This site is AWESOME!!! im really glad theres something like this up and running to help students who are willing to study at home. it gives a break from just reading and doing examples from books and really helps!!! im sure my marks are going to improve!!!! THANKS!!! (30 Apr 03) 
User 6: i learnt alot, & it went smoothly, gradually getting harder & making me learn much more efficiently. it is very helpful thank you (9 May 03)


The comments entered during 2003-2005 have been assigned to categories. Three of these categories (compliment, suggestion and criticism) are relevant to teaching and learning. 78% of comments have been assigned to one of these. 22% of comments either do not pertain to BestChoice or highlight typographical errors and bugs that have been fixed. 

Many of the comments relevant to teaching and learning have multiple threads. Any comment that includes a compliment is assigned to compliment. Critical comments that include a suggestion are assigned to suggestion. The table shows the distribution of comments in these three categories. The trend is toward a lower percentage in the criticism category and higher percentages in the compliment and suggestion categories as a result of enhancement both of the system and the content.
 

Comments entered in on-line survey
Year Total Compliment image Suggestion image Criticism image
2005 1173 68% 15% 17%
2004 1134 67% 14% 19%
2003 401 64% 12% 24%


These data show that the learners perceive that BestChoice is beneficial to them and helps them to learn. Furthermore users suggest ways in which BestChoice can be improved (even through critical comments). One advantage of a web-based system is that problems identified can be rectified and the revised version made available within a short time frame.

The module ratings (out of 6) indicate that most students find BestChoice modules helpful 
Students indicate how helpful they have found a module by choosing one of the six radio buttons on the end-of-module survey form. These range from not at all (1) to fantastic (6). The average response rate to this part of the survey over all modules and all cohorts of users is 30%. As shown below, the radio buttons corresponding to 4,5 and 6 ratings are much more commonly chosen than those corresponding to 1,2 and 3 ratings. The most frequent choice in each year is fantastic (6).
 

Module ratings using a 6 point scale
Year Total responses Average 1 2 3 4 5 6
2005 6866 4.64 253 275 717 1383 2034 2204
2004 6442 4.71 186 266 570 1309 2019 2092
2003 2010 4.53 83 106 223 437 578 583

 

Using BestChoice as part of course materials

BestChoice for Chemistry students at The University of Auckland

In 2003, use of BestChoice was compulsory in two first year Chemistry courses at The University of Auckland. In 2006 use of BestChoice is compulsory in five first year Chemistry courses. In total this involves approximately 2000 users, some of whom are enrolled in more than one course. 

Although one of the authors has had extensive experience teaching first year Chemistry, neither of us is currently lecturing in any of the first year Chemistry courses. Therefore, support of course coordinators and teachers is crucial. These academic staff

  • introduce their class to BestChoice, providing details of the registration process.
  • promote BestChoice activities as learning experiences.
  • encourage students to use BestChoice to support their learning during the entire course. One lecturer has recently referenced relevant BestChoice modules in the course notes.

What form does the compulsory usage take?
BestChoice modules are currently used as assignments in four courses. 

Cohort: Students in one of these courses are science majors. Students in the other three courses are studying Chemistry at pre-university level. 

Contribution to overall assessment: All courses have a maximum of 100 marks. Ten modules at 0.5 marks each are compulsory in the course for science majors and in one of the pre-university courses. Five modules at 1 mark each are compulsory for the other two pre-university courses.

 

One of these courses was the first to use BestChoice as an assessment component. Based on that experience, course coordinators specify the modules to be completed and set completion deadlines. This ensures that students begin to use BestChoice early in the course.


BestChoice modules are used in one course as both pre-lab activities and assignments

Cohort: In semester 1 these students are intending Health Science students and science majors (including Chemistry). In Semester 2 they are science majors.

Assessment of prelabs: Evidence for completion of the module is submission of a printout of the last page of the module with the lab script. For some experiments, the last page is part of the background information/data required for the experiment. 

Contribution of assignments to overall assessment: The course maximum is 100 marks. Four modules at 0.5 marks each are compulsory.

 

The BestChoice pre-labs replace written pre-labs that were marked by supervisors at the beginning of the lab session. Because BestChoice pages are marked as they are completed, the supervisors now have more time to help students with their practical work at the beginning of the lab. The course coordinator also believes that there is less "blind copying" and more opportunity for learning when the students complete pre-lab exercises on line

This course coordinator has decided, for both the pre-lab and assignment modules, to disable the Give Up and Show Next options that allow BestChoice users to view the correct answer. Despite this, these modules have high student ratings, and students do not comment on the absence of these features in the on-line survey.


BestChoice sessions in a computer lab were also a feature of our four day catch-up course in 2006. 

Each day 80 students used BestChoice modules that reinforced lecture material.

 

BestChoice for New Zealand high school students

New Zealand high school students use BestChoice on a voluntary basis. High schools in New Zealand have computer labs, and students are often introduced to BestChoice through their teacher booking a period in the school computer lab. Teachers are made aware of BestChoice nationally through an electronic newsletter circulated by a high school teacher. 
 

Evaluating BestChoice through usage data

Usage of BestChoice has grown steadily

BestChoice collects a variety of data for each user. This includes demographic details (user-entered), registration date, number of logins, and a count of pages on which answers have been entered as well as how many attempts were required to get the correct answer and the time interval over which the question was answered. 

The table below shows that there has been a steady increase in the number of registrations per year over 2003-2005. The increase in 2005 is largely due to New Zealand high school students.

Year Number of registrations Number of Active Users Logins
2005 7 560 4 000 50 190
2004 5 983 3 200 37 916
2003 3 380 1 400 15 581


There has also been growth in the number of active users. These users have answered questions on more than 20 pages. In 2003 an active user entered at least 20 answers. In 2004-2005 an active user will have have entered at least 100 answers. There is a correlation between the percentage increase in either logins or registrations over an entire year and the percentage increase in active usage.
 

Active usage data highlights the importance of the connection to the user's course of study

Users are asked on registration to identify themselves with one of the institutions shown in the table below. The table also shows the number of active users associated with each institution for 2005. 
 

Institution Active users in 2005
The University of Auckland 2230
NZ high schools 1670 (from 210 schools)
Other NZ Universities 88
Outside NZ 12


The largest groups of active users are those for whom specific courses have been created. The difference between the two large cohorts is that while use of BestChoice is compulsory for most university users, New Zealand high school students use BestChoice on a voluntary basis. 
 

The timing of logins gives evidence for how these two cohorts use BestChoice

The graph below shows the number of logins by students at The University of Auckland for each month during 2005. This indicates that usage of BestChoice during the semester (March-June and then July - November) is fairly consistent and that usage drops during the inter-semester period. Comparison with data from 2003 and 2004 shows that, due to the support of the course coordinators, students in each subsequent year use BestChoice both earlier in the semester and more consistently throughout the semester. 
 

image


The graph above for New Zealand high school students logins shows that their highest usage is at the end of the year. Thus this cohort regards BestChoice primarily as an exam preparation tool. This perception may, however, be changing because the percentage of logins to BestChoice during the period February through September is higher for 2005 (53%) than for 2004 (41%).
 

In conclusion

What have we found?

The BestChoice project has shown that web-based activities can facilitate learning by requiring the user to take an active role, with the system then responding to the user's input. The guidance offered by BestChoice makes successful completion of the problem more accessible for more learners who, over time, should gain the skills to answer the questions without guidance. Consider the unedited comments. 
 

User 7: The main thing I liked is that it gave me the opportunity to practice and learn from my mistakes. (31 Mar 06) 
User 8: liked this topic. Being able to see and work everything out myself really helped me understand this. Thanks (31 Mar 06) 
User 9: It helped very much. It was much easier to understand than my notes, textbook and lecturer! The reviews are so concise and great. Thanks! (31 Mar 06)


BestChoice activities also provide capable learners with an enhanced learning experience as insights are revealed during problem-solving and appear in the form of feedback just as the user has entered the answer that they thought was correct. 

User 10: Very good, now I understand alot more little things I wouldve never thought of asking. (19 Mar 06) 
User 11: it lead us to think critically and logically, and not just know facts that have already been provided for us in the course notes. (19 Mar 04)


The BestChoice project has also shown that users can inform the design of systems to support their learning. The extent and quality of feedback entered by users has been overwhelming.
 

User 12: BC helps me understand chemistry in a way that is both faster and more convenient than tedious text book exercises - almost like having a personal tutor watching over my shoulder. Thank you and keep up the good work! (2 Apr 06) 
User 13: Bestchoice is simply the best learning technique i ever came across. It walks me through the important steps to follow inorder to solve a question and makes the theory more simpler. (26 Mar 06)

 

How does BestChoice compare to other learning tools?

A paper survey asking "What feature of the course most helps you to learn?" has been circulated in some courses where use of BestChoice is compulsory.
 

(Science majors) 100 surveys, 62 responses to this question, BestChoice 41; Labs 39 
(Science majors) 170 surveys, 111 responses, BestChoice 37; Handouts 44; Labs 25; Lecturers 20 
(Summer pre-university) 80 surveys, 63 responses, BestChoice 25; Lectures/lecturers 28; Handouts 13


The student responses indicate that they view BestChoice as an effective learning tool that complements traditional learning tools. BestChoice is not, and was never intended to be, a replacement for these. 

It is a challenge for lecturers to accommodate the variety of learning styles and academic backgrounds in the large classes at first year university level. Web-based activities like those inBestChoice can be an addition to the suite of tools available to help meet this challenge. Unlike the most other learning tools, use of BestChoice by students generates data that can provide information on how to support learning more effectively and efficiently. 
 

Where to now?

The most frequent suggestion entered in our survey is "more questions". New Question Pages are currently being added on a continual basis. An important aspect of the writing of these pages is that new ways being explored to present content interactively, including working toward enabling the user to choose the extent of guidance offered on a BestChoice page.

The discussion above is a broad analysis of data pertinent to student perception and usage. It is now appropriate to extend these investigations by considering student response data relevant to individual modules and questions to determine whether areas of difficulty for students can be identified. Once barriers to learning have been identified, the focus would be on improving support for learning in these areas. 

In conclusion, it is evident that a large number of users believe that BestChoice is very effective in helping them learn Chemistry. Furthermore, once a system like BestChoice is in place, it thrives on student usage because usage creates data that highlights where the system needs improving. 

BestChoice (www.che.auckland.ac.nz/bestchoice) is open access. Use Demo mode (no registration, no marks stored) or register. In either case, we would appreciate your feedback.


Acknowledgement: We acknowledge Drs David Salter and Judy Brittain, and Ian Torrie. David and Judy coordinate Chemistry courses that use BestChoice. Ian advertises BestChoice to high school teachers. We thank the Chemistry Department and the Teaching Improvement Grants Committee of The University of Auckland for financial support.