MUST-Know Pilot—Math Preparation Study from Texas

Declining numeracy in the U.S. is real and gaining concern. The curiosity for this investigation piqued when a co-author from the Naval Academy noticed that U.S. students were "calculator dependent" and had not received appropriate number-sense training in their K-12 studies (Hartman & Nelson, 2016). These authors offered a link to a quiz (http://bit.ly/1HyamPc) that the last author of this paper named the MUST (Math-up Skills Test), and subsequently employed it as part of a pilot study named the MUST-Know project initiating a statewide investigation.

Introduction

Texas, we have a problem!

In general, college-ready Texas students are less prepared now than they have been over the last 30 years (Fig. 1).

Figure 1. SAT scores over 30 years (points 1-30) with demarcations indicating changes to state adopted curriculum standards. [Note when Science Director Comer resigned: point 21 (2007).]

Some of the justification of declining scores is attributed to the 2010-11 academic year (AY), when the Texas Education Association (TEA) funded free SAT exams. With lower-income students able to take the SAT, potentially additional weaker students may have contributed to the decline. In AY 2011-2012 and thereafter, some districts began offering SAT exams during the school day, thereby increasing the number of less motivated students (i.e., a get out of class free card!) that may have attracted a population who had not completed the suggested college-prep curriculum. Related to the graph in figure 1, separating Math SAT scores from SAT Reading + Math, a decline of 22 points occurred from 2010 to 2015 (504 to 482, respectively). However, on a positive note, there is a slight bump in AY 2012-2013, when the 4´4 curriculum was fully implemented. The Texas 4´4 program required all high school students to sit for four assessments in English, mathematics, science and social studies, and pass a minimum of three in each discipline in order to graduate. One can assume from this small upward movement that “when required” (i.e., when tested), students' understanding will improve. Since 2013, high-stakes testing is no longer required and SAT scores in Texas have plummeted.

Texas Curriculum Assessments
 

Texas has changed the state-adopted curriculum four times over the last 30 years. Each was accompanied by high-stakes assessments (estimated to cost about $1M each). TEA instituted a statewide testing program in 1979 for grades 3, 5 and 9. Prior to 1990, there was TABS (Texas Assessment of Basic Skills), and by 1986, TEA implemented TEAMS (Texas Educational Assessment of Minimum Skills) that when not passed students were not eligible to receive a high school diploma stemming from Governor White's "no pass, no play" policy. Curriculum was changed to TAAS (Texas Assessment of Academic Skills) in 1990 and then to TAKS (Texas Assessment of Knowledge and Skills) in 2003, with the latest version (2011-2012) becoming the STAAR (State of Texas Assessment of Academic Readiness) program that was dismissed by the current governor as a requirement for graduation. Now, from four science assessments being required for graduation there is only one required test in science (Biology STAAR) and poor performance no longer prevents a student from graduating. Another observation coinciding with the constant decline of SAT scores is the resignation of a dynamic TEA science director in 2007. Science Director Comer helped develop and promote the 4´4 curriculum as a strong advocate of advancing study in all sciences and recognizing the necessity of partnering with mathematics education.

Calculator Usage

Currently, Texas high school students only take one high-stakes science assessment and STAARs in Algebra I and II. The calculator policy states no calculators are permitted on STAARs in grades 3-7, but districts must ensure that each student has a graphing calculator to use on all STAARs starting with 8th-grade mathematics (both paper and online versions) and biology. For the biology assessment, there should be one calculator (four-function, scientific, or graphing) for every five students. Students may bring their own calculators with them to the assessments, but Internet capabilities must be disabled and calculation applications on smartphones are not allowed. [There was at one time a graduation proposal that student's score on end-of-course assessments would be 15% of their final grade for that course, but this was rejected almost as soon as it was suggested!] Beginning in May 2018, the grade 8 science STAAR will require students to have access to calculators with four-functions, scientific or graphing capability (TEA, 2017).

MUST [Mathematics: Underlying Skills and Thinking] Know Pilot Study

Demographics: Institutions

One strength brought to this investigation on what arithmetic-fluency levels are necessary to succeed in general chemistry lies in the team's differences. With variations in required institutional prerequisites, class sizes, instructors, textbooks, teaching methods, information and communication technology (ICT) tools, etc., the evaluations have produced similar results leading the team to a "value added" model that may contribute to curricular improvements.

Our research team consists of eight general chemistry instructors employed at six universities (three public research; two Hispanic Serving Institutions (HSIs); and one, four-year private) spread across 32,000 mi², about 12% of the state. All faculty team members have acquired IRB approval for this research at each institution.

Abilene Christian University (ACU) is a small private university in west Texas. The student body is ethnically diverse; there are ~4,500 full-time enrollees with 63% of students listing Caucasian, while 37% are from underrepresented minority groups. Female students comprise 59% of the student population. Texas residents make up 86% of the student body.

Texas A&M University–San Antonio (TSA) was the first Texas A&M University System institution to be established in a major urban center in 2009. Currently enrolled are approximately 5,500 students. Both undergraduate and graduate-level classes are offered. The Fall 2016 semester marked A&M-SA’s first cohort of freshman and sophomore students. Of these students, 74% are first generation, 60% female, and nearly 83% identify as Hispanic or Latino recognizing A&M-SA as a HSI. Nearly 1 in 6 students are military connected.

Texas State University (TSU) founded in 1899 is the fourth largest public university in the state of Texas and 34^th largest in nation with an enrollment of almost 40,000 students with over 34,000 classified as undergraduates. The university offers 98 bachelors degrees, 91 masters degrees, and 13 doctoral degrees, and is in the top 6 in graduation rates among the 38 public universities in Texas. The population includes 57.9% females, 10.7% African-American, 34.7% Hispanic, and 48.1% white with the remaining 6.6% being Native American, Asian/Pacific Islander, or Non-Resident Alien. This HSI ranks 14^th in the nation for total number of bachelors degrees awarded to Hispanic students. The reported six-year graduation rate stands at 54% and the retention rate of returning freshmen is 77.4%.

Texas A&M University (A&M) opened its doors in 1876 as the state's largest and first public institution of higher learning. TAMU is among nation’s five largest universities with an enrollment of over 66,000 students. TAMU is one of only a few universities in the country to be designated a land grant, sea grant and space grant university, and is reported by the U.S. News & World Report as ranking second in the nation in the "Best Value Schools" category among public universities. Enrollment includes 52% male, with 58% white, 20% Hispanic, and 22% other ethnic groups (black, Asian, international, Native American, etc.). The university has more than 130 undergraduate degree programs, 170 masters degree programs, 93 doctoral programs and 5 first-professional degrees as options for study. The reported six-year graduation rate for the undergrads stands at 79.5%.

The University of Texas at Austin (UTX) is a Tier One research institute, the flagship campus of The University of Texas System, and second largest in the state. Enrollment of 51,000 students (40,000 undergraduates) represents all 50 states alongside 118 countries. Student demographics include a population of 51.5% female, 43.3% white, 20.0% Hispanic, 17.8% Asian, 3.9% black, 10.1% foreign, and less than 5% other or combination of these. UT strives to improve upon several accolades, including Forbes’ 17^th Best Value School and Kiplinger’s #13 Best Value Public College. As one of the largest science colleges in the U.S., UT’s College of Natural Sciences includes over 13,000 undergraduates. Many of these students participate in groundbreaking, nationally recognized programs such as the Freshman Research Initiative (FRI) and Texas Interdisciplinary Program (TIP). UT currently reports a six-year graduation rate of 81.2% for undergraduates.

University of North Texas (UNT) established in 1890 is a four-year public R1 (Carnegie Classification) doctoral university with an enrollment over 38,000 students (fifth largest in the state), 31,000+ classified as undergraduates. For 21 years in a row, UNT has been named one of America's Best College BuysÒ with 16 programs (5 STEM areas) reported by the U.S. News & World Report as ranking in the Top 100. The reported ethnic makeup includes African-Americans (14.01%), Hispanics (22.12%), and white-non-Hispanics (48.41%) with the remaining 15.46% being Native American/Alaskan and Asian and Pacific Islanders or Non-Resident Alien. The reported six-year graduation rate for the 2008 UNT undergrads stands at 59.1%.

Demographics: Students

The research team investigated relationships between solving arithmetic problems appropriate for success in general chemistry and course grades of 2,127 students. The combined student population consists of 60.3% female and 85.4% freshmen and sophomores enrolled in general chemistry I and II (Chem I and Chem II) and engineering chemistry courses. Gathering data on the ethnicities within these classes proved problematic at different institutions given various IRB inclinations. However, we assume that the combined students' ethnicities mirror those of Texas given the wide geographic area involved.

Texas Student Profile 2015-2016 (2017 Texas Public Higher Education Almanac)

• Debt level of bachelor-degreed graduates: State average = $31,186
• Racial and ethnic distribution (majority minority state):
  42.5% white, 39.9% Hispanic, 11.4% African American, 6.1% other
• Higher education enrollment:
  36.4% white, 36.0% Hispanic, 13.2% African American, 14.4% other
• Students meeting college readiness benchmarks: 26%

MUST Instrument: Statistically valid and reliable

The instrument chosen to assess the arithmetic skills of general chemistry students in the pilot study was published in a report by Hartman and Nelson (2016). This instrument contains a total of 16 items, has two versions, and is named the MUST (Math-Up Skills Test). Both versions of the MUST were validated by two UNT mathematics professors. The MUSTs were statistically proven to be highly reliable (KR-21 = 0.821) and no statistical differences between versions were shown to exist. The two mathematics professors noted that the concepts covered by this instrument were not taught at the college level because they had been previously taught and assessed prior to post-secondary matriculation.

The MUST was given to students (n = 2,127) face-to-face during class without the use of a calculator (time limit of 12 min) followed by with use of a calculator (time limit of 12 min). Each correct answer earned 1.0 point and no points were awarded to an incorrect answer. Table 1 presents the grade level where the various topics on the MUST are introduced to Texas students and the means for each correctly answered question. The overall mean is X̄ = 7.36/16 = 46.0%.

As can be seen in Table 1, some of the topics are introduced as early as 4th grade and all have been presented to students prior to high school completion. The last two questions cover the topic of balancing equations, technically an exercise in counting, but not a required course for all high school graduates. Raising an integer to the zero power appears to be the most understood concept with base-10 logarithms being the least understood concept. A challenge to teaching general chemistry is presented when only 66% of the students assessed can multiply two, two-digits numbers (like, 87 ´ 69) correctly.

Results

Data without student identifiers from each institution were sent to the research team leader (last author) for compilation. The data analyses to date include descriptive statistics, measures of reliability, correlations, and t-tests. As the database grows and the study continues, more statistical evaluations are planned such as Spearman rho correlations and ANOVAs to compare relationships between groups.

Combined data (n = 2,127) from this pilot study were evaluated, then separated by courses, by institution and semester, and re-evaluated. Some of the team members presented the MUST with demographic information and IRB consent forms on different days, some gave the MUST without a calculator and with a calculator on different days, and some students did not answer all the required demographic information requested reducing the population with complete data sets to n = 1,415 or 66.5% of the whole. However, for the purpose of this report, the larger population will be acknowledged most of the time.

One of the first observations made was how the scores on the MUST followed the same pattern across multiple classes at various institutions (Fig. 2). It is not that students at the various universities scored the same, but the up and down flow of the means of each question regardless of class (Chem I, II, Engineering), institution, semester (fall, spring) all appear to illustrate the same trends. The majority of these students were educated in Texas secondary schools, so it appears that many have garnered similar understandings.

Figure 2. Pattern produced by MUST scores across multiple settings. Y-axis: point value of 1.0 per question. X-axis: MUST question numbers.

By Student Success

The percentage of successful (grades of ABC) Chem I students (n = 482) is 66.1%, Chem II (n = 901) is 79.9%, and Engineering Chem (n = 32) is 68.8%. However, some of the successful students in the courses did poorly on the MUST and vice versa. The percentage of successful Chem I students who have a MUST score below the mean (i.e., MUST scores = 0-4) is 153/319 = 48.0%. The percentage of unsuccessful (grades of DF) Chem I students who have a MUST average below the mean (i.e., MUST scores = 0-4) is 119/163 = 73.0%, highlighting that a higher percentage of Chem I students with a low MUST score are unsuccessful students in this course. The percentage of successful Chem II students who have a MUST average below the mean (i.e., MUST scores = 0-8) is 280/720 = 38.9%, and the percentage of unsuccessful Chem II students who have a MUST average below the mean (i.e., MUST scores = 0-8) is 145/181 = 80.1%. Yes, students can be successful with low MUST scores and with above average MUST scores one is not guaranteed success, but the odds are better for success if a student has adequate arithmetic skill. If you are in Chem II, lacking MUST skills is even more pronounced with over 80% not being successful in the course when MUST scores are below average.

By Course

As reported in Table 2, students without the aid of a calculator and complete data sets (n = 1,415) succeeded at less than 30%-correct level in Chem I (4.53/16) and slightly more than the 50%-correct level in Chem II on the MUST (8.38/16) with the engineering class' MUST score falling between (7.63/16). With the use of a calculator, students performed better in Chem I and II with approximately 70% and 80% correct, respectively. However, the correlation to course grades without a calculator was higher than with a calculator, r = 0.451(Fig. 3) and r = 0.402, respectively. Even though correlations are low, the MUST was shown to be a consistent predictor of success despite existing variations between the classes; the combined data relationship between MUST scores and course grades appears to be linear (Figs. 3 & 4).

Graphical representations of the data supporting Table 2 show the greater linear relationship of course grades to the MUST without the use of a calculator than to the MUST with a calculator especially in Chem I (Fig. 5). Students who used a calculator have a greater variance in success in Chem I as noted the percentage of students who did well on the MUST but not so well in the subsequent course. In Chem II (Fig. 6) both without and with the use of a calculator there appears to be less of a difference when compared to course grades.

By Institution and Semester

Table 3 separates data from the various institutions. The research universities in ranked order are UT Austin, A&M and UNT. Noting Chem II MUST scores of these universities in the spring semester, they are 11.41, 10.73 and 5.38, respectively. ACU is a private university in Abilene and performed very well on the MUST, and the two HSIs (TSU and TSA) reported the lowest MUST scores in both the fall and spring courses.

By Gender and Classification

When general chemistry data, separated by semesters, were evaluated (Table 4), males outperformed females on the MUST without the use of a calculator (p < .05) in the fall, but not in the spring. As to course grades, no statistical difference was evident in the fall course averages, but in the spring, females statistically outperformed males.
 

No statistical differences were discovered between the various classifications (Table 5) where of interest is that freshmen distinguished themselves by bringing the highest MUST and course averages while students identified as juniors had both the lowest MUST and course averages.
 

Successful and unsuccessful male MUST scores were statistically higher (p < .05) than those of the females (Table 6). Course averages presented no statistical differences for successful or unsuccessful students of either gender. A slightly greater percentage of females were successful in the classes on the average than males even though they entered with lower MUST scores. It is possible that this observation is due to the nature of general chemistry curriculum in that algorithmic assessments are paired with conceptual understanding assessments and females improve their overall grades because final grades are not solely based on mathematics fluency.

Conclusions

The average successful (course grades of ABC) general chemistry student based on pilot study data has the following profile: Chem I MUST score ≥ >32% correct and Chem II ≥ 58% correct. The team is in the process of building a more definable profile of what it takes to be a successful general chemistry student. In the fall 2017, an algebra-skills assessment is being added to the investigation, and the MUST (with calculator) is being eliminated. The demographic section has been expanded in order to give us more information about students' experiences so that those in danger of not succeeding may receive informed advising and hopefully avoid some of the noted attrition, and thereby grow the understanding of a successful general chemistry student.

References

2017 Texas Public Higher Education Almanac Chem13 News (September 2012). Why students fail in college.

De Vega, C. A.; McAnally-Salas, L. (2011). US-China Education Review, 1, 10.

Hartman, JudithAnn, and Nelson, Eric A. (2016). Automaticity in Computation and Student Success in Introductory Physical Science Courses. Cornell University Library. arXiv:1608.05006v2 [physics.ed-ph] Paper presented as part of Chemistry & Cognition: Support for Cognitive-Based First-Year Chemistry.

Technical Digest 2008-2009, Chapter 1: Historical Overview of Assessment in Texas (pp 1-8).

Texas Education Agency (TEA) Student Assessment Division. (2017). Updated calculator policy. http://tea.texas.gov/student.assessment/staar/

Texas Essential Knowledge and Skills (TEKS) Home Page. http://www.tea.state.tx.us/index2.aspx?id=6148 (accessed September 2017).

Texas CCRS. Website of the Texas Higher Education Coordinating Board, Texas College Readiness Standards. http://www.thecb.state.tx.us/collegereadiness/CRS.pdf (accessed Sep 2017).

Tai, R. H.; Ward, R. B.; Sadler, P. M. (2006). High School Chemistry Content Background of Introductory College Chemistry Students and Its Association with College Chemistry Grades. Journal of Chemical Education, 83, 1703–1711. DOI: 10.1021/ed083p1703

Zeegers, P. L. M. (2001). A Learning-to-learn Program in a First-year Chemistry Class. Higher Education Research Development, 20, 35–52. DOI: 10.1080/07924360120043630

Supplemental Information for Discussion

For the past 25 years, academic statistics on college readiness have remained relatively constant (Tai, Ward, & Sadler, 2006; Chem 13 News, 1986 and 2012). On average, students take six years to complete a four-year college degree, and 30-60% of these students will require remedial coursework upon entering college (Tai, Ward, & Sadler, 2006). A more disturbing statistic is that roughly 30% of incoming first-year students consider terminating their academic studies entirely (Zeegers, 2001). Students have many challenges as they progress from secondary to postsecondary education. The failure of many freshmen comes from their inability to become proficient at time management, planned study time, a heavier reading load, no reminders of tests and homework, balancing work and play, and having to seek out help on their own (De Vega & McAnally-Salas, 2011).

College-Ready Students

"Why students fail in college" was published in Chem 13 News (September 2012), but originally published in October/November 1986 in Chem 13 News, pages 10-11.

Which of the following remain true today?

1.  Unprepared (or underprepared) to assume responsibility for their own learning.

2.  Time management skills are lacking.

3.  Lack of self-discipline needed to study effectively.

4.  Do not understand whether or not they comprehend the material needed.

5.  Lack of skills to find needed information or how to separate misleading or irrelevant information.

6.  Difficulty in synthesizing information from several sources.

7.  Failure to complete (and sometimes even begin) assignments.

8.  Failure to interpret tables, diagrams, graphs, mathematical expressions, and specialized languages such as chemical equations.

9.  Poor communication skills especially when attempting to express their own ideas.

10. Lack of originality needed to synthesize subject matter and draw conclusions.

      11. Writing is often poorly organized, grammatically incorrect and riddled with contradictions.

      12. Inability to evaluate facts, directions, or other information.

13. Lack flexibility when faced with a poor instructional environment to acquire useful knowledge on own.

      14. Meaning of memorized words remains unclear.

      15. Failure to understand logic behind the algorithms or rule.

16. Proportional reasoning is lacking at a level required to understanding most chemistry concepts and computations.

17. The logic inherent in mathematical and chemical language, spatial reasoning, mental constructs and how to think about chemical changes are at best in an immature stage.

      18. Inability to retrieve information "taught" from long-term memory.

Transition from high school to college

Coursework in high school chemistry and in general chemistry is aligned (Texas College and Career Readiness Standards, 2009). Future employers expect diligence, persistence, reliability, problem-solving skills, and logical thinking that can be promoted in chemistry courses. What is not in place is how to help student adjust to a set of new expectations, but we can help here, too! Maybe when implementing your latest high impact practices (HIPs) consider the following: