Addressing Math Deficits With Cognitive Science

Pre-Test

Please jot brief answers to these.

1. When I teach a “first-level” chem course, I assume students have previously had about ____ year(s) of prior chem and ____  years of prior math (arithmetic, fractions, algebra, etc.).
2. Graphed below are scores for 9^th graders in a US state on a test that separately measured math reasoning, math computation, and a weighted composite (total math).  As a measure of preparation for general chemistry, in your view, are these results GOOD or BAD news, and briefly, why?

Figure 1:  Reasoning vs. Computation

 

Scope

This paper’s scope is limited to chemistry courses preparing students for science majors, including general, AP, GOB, engineering, “prep chem,” and college preparatory high school chemistry.

 

Trends in Student Science-Majors

Data indicate that our work as chemistry instructors to help students succeed in science majors has become increasingly difficult. Since 1966, the percentage of college students who graduate as chemistry majors is down by about 60% (Table 1).  More recently, between 1984 and 2012, the percentage graduating as chem majors was down over 30% (Figure 2) (NSF 2015).

Figure 2:  NSF Data – Chemistry and Physics

 

These data contain one item of very good news:  The percentage of chem majors who are female rose from 19% in 1966 to 49% in 2012 (Table 1).
 

Table 1:  NSF Data on Bachelor’s Degrees and Chemistry Majors, 1966-2012

Chem Majors:  1966-2012
US Bachelor’s Degrees Awarded	1966	1978	1990	2012
In All Fields	524,008	930,201	1,062,151	1,810,647
Chem Majors	9,735 (1.86% of BAs)	11,474 (1.23%)	8,289 (0.78%)	13,714 (0.76%)
Male	7934 (1.51% of BAs)	8593 (0.92%)	4965 (0.46%)	6984 (0.38%)
Female	1801 (19% of chem)	2881	3324	6730 (49% of chem)

However, between 1978 and 2012, while the annual number of U.S. students earning bachelor’s degrees nearly doubled, the annual number of male chemistry majors declined.  

Are students instead entering high-demand majors that include a strong chemistry foundation?  Yes and no.  NSF data show about 6 times more students complete engineering than chemistry majors.  However, from 1984 to 2012, the percentage of graduates receiving engineering bachelor’s degrees fell by 40% - vs. the 30% decline in chemistry (Figure 3). 

Figure 3:  NSF Data –  Engineering, Chemistry, Physics

Due to the variety of credentials in nursing/health programs, biology/health care majors are more difficult to track, but available NSF data generally show similar declines in those programs as well.

STEM graduates are in demand. In 2012, the US President’s Council of Advisors on Science and Technology (PCAST) cited “a need for approximately 1 million more STEM professionals than the U.S. will produce at the current rate over the next decade” (Olson and Riordan, 2012).

In career planning, students respond to this demand.  Between 1975 and 2008, students entering college stating their intention to major in the physical sciences and engineering remained relatively level: between 10%-13% each year (NSB 2002, 2012).  However, among the entrants stating those intentions in 2003, less than 43% had attained a STEM degree of any kind six years later (Olson and Riordan, 2012).

In general chemistry, a gateway to STEM majors, studies have reported “DFW” rates as high as 60% in two-semester sequences at selective state universities (Cavanaugh 2008) and higher than 80% at community colleges (NCEE 2013).

Most of these data are not good news, but thanks to recent progress in cognitive science, to a substantial extent, we know what went wrong (and it wasn’t the fault of chemistry instruction), and we know how to improve success for students seeking STEM careers.

 

Preparation for Science

Among students who complete general chemistry nationwide, for every one future graduate who is a chem-major, there are about 6 engineering majors, 7-8 life science majors, and one physics or earth science major (NSF 2015).   For all of these majors, chemistry is tasked with introducing a core component of science:  calculations based on measurements. 

In general chemistry, solutions to problems often include:

• Complex fractions that mix numbers and units;

Figure 4: A Fraction with Four Denominators

 

• Powers and roots of numbers, exponential terms, and units;

Figure 5: Fractions, Exponents, and Roots (From Zumdahl, Chemistry, 5th Ed.)

 

• And logarithms.

 

 

Figure 6:  Exponents, Powers, and Logs (From Zumdahl, 5^th edition)

 

In Figures 4 to 6, after data are substituted, solving requires arithmetic, fraction math, and algebra.  Most general chemistry textbooks assume this math is background knowledge that has been mastered in grades 1-12. 

Let us examine:  Is that assumption accurate for the current generation of students?

 

Calculator Use

Prior to 1970, most math learning relied upon applying more than 300 exact, memorized “math facts” for “addition and subtraction within 20” and “multiplication and division through 12’s.”

Handheld calculators became available in 1970 and were gradually introduced into K-12 instruction.  Daily calculator use was reported by 26% of U.S. high school math students in 1986 and 60% of 8th graders in 1996 (Waits and Demana 2000).  By 2005, over 30 states directed teachers to teach calculator use for third grade arithmetic (Klein et al. 2005, Hartman and Nelson 2016).  As calculator use increased, studies of U.S. college students between 1991 and 2001 found that many could not recall basic addition, subtraction, and multiplication facts (Geary et al. 2008).

 

“Decreased Attention” to Math in Memory

In addition to increased calculator use, between 1990 and 2002 nearly every U.S. state adopted “K-12 standards.”  In math, most states aligned their standards with recommendations of the National Council of Teachers of Mathematics (NCTM).  The “1989 NCTM standards” supported “increased attention” to “reasoning inductively and deductively” and “decreased attention” to “memorizing rules and algorithms,” “manipulating symbols,” “rote practice,” and “paper and pencil fraction computation” (NCTM 1989).  These standards hoped to minimize the drudgery of drill and practice.  Why “rote memorize” when you can use a calculator, reason, and from 1995 forward, “look up on the internet” whatever you need to know?

The 1989 answer was:  Why indeed?  But in key respects, in 2017, the answer from science is different. 

 

Science and Memory

Cognitive science is the study of how the brain works and how it learns. Since 2010, with help from new technologies, cognitive scientists have reached consensus on many aspects of how the brain solves problems.

In numbered points below are parts of the consensus cognitive science model for problem solving.  Additional detail and references are available (see Hartman and Nelson 2015, 2016).  If this content is unfamiliar, don’t expect to remember all points in a first reading, but if the topic sparks your interest, return when time permits.  At the end will be a short “summary to remember for now.”

As instructors, understanding what the brain can and cannot do will help us to design more effective instruction.

 

Cognitive Architecture

1.   Learning has an “atomic structure.”  The brain decomposes new knowledge into small elements that are stored in long-term memory (LTM). LTM consists of billions of “neurons:” cells that can selectively connect and communicate via electrical impulses (Anderson and Lebiere, 1998).

2.   In learning math and science, the goal is to solve problems.  In structures that together serve as working memory (WM), the brain solves problems by processing multiple elements during stepwise procedures.  WM can acquire elements from the senses (such as by reading or listening), or from a “middle-step” result of processing, or by recall of previously learned elements from LTM.

3.   Neurons can connect via “wires” that meet at synapses.  These linkages tend to form when elements stored in LTM neurons are processed at the same time during problem solving (Hebb 1949).  Repeated simultaneous processing strengthens connections.  Connected neurons form the physical substance of the brain’s “conceptual frameworks,” also called “schema.” 

4.   For most learning in math and science, storing elements and forming and maintaining neural connections requires repeated effort to recall new relationships, spaced over multiple days, revisited occasionally thereafter.  Elements studied intensely, but for only a few days, tend not to “stick” in recallable memory (Geary et al. 2008, Sweller 2009, Brown et al. 2014).

5.   Elements moved into WM can be divided into two types. Previously well-memorized elements, when entering WM, cause matching and related elements in LTM to “activate.”   Novel elements in WM are either not stored in LTM or not well linked to knowledge in the problem (Clark et al. 2012).

6.   Working memory can be divided into “novel WM” with slots to hold novel elements and “long-term working memory” slots for recallable elements.

7.   Novel WM is quite limited in both duration and capacity.  WM can typically hold and process only 3 to 5 novel elements at each step during problem solving (Cowan 2001, 2010).  Novel elements tend to fall out of WM in 3-30 seconds unless rehearsed (Peterson and Peterson 1959, Cowan 2010).

8.   In WM, space for well-memorized and activated LTM elements is essentially unlimited (Ericsson and Kintsch 1995). 

9.   If more than a few elements of data in a problem are novel, or if too many elements that must be held in WM during the steps of processing are novel,  novel WM tends to overload, needed novel elements may be dropped, and confusion tends to result (Geary et al. 2008, Clark et al. 2012).

Some models for problem solving under varying circumstances differ in terminology and detail from the description above, but all recent models include a working memory that is large for well-memorized but minimal for novel elements.

 

Cognitive Model Summary

Here’s the “summary to remember:”

Working memory (where the brain solves problems) has minimal space for information that has not previously been well-memorized.

 

Is This Reliable Science?

Can we be certain this new model for learning accurately predicts outcomes?

Between 2001 and 2010, the measured 3-5 element limit in novel WM was experimentally verified. Based on those results and others, dozens of experts on cognition writing since 2010 have accepted this model.  Equally important, few if any cognitive experts disagreed.  Science is what the experts in a sub-discipline agree it is (Kuhn 1962). 

Implications of Cognitive Architecture

In a U.S. Presidential Commission report in 2008, five of the nation’s leading cognitive experts wrote:

“[T]here are several ways to improve the functional capacity of working memory. The most central of these is the achievement of automaticity, that is, the fast, implicit, and automatic retrieval of a fact or a procedure from long-term memory….  [T]o obtain the maximal benefits of automaticity in support of complex problem solving, arithmetic facts and fundamental algorithms should be thoroughly mastered, and indeed, over-learned, rather than merely learned to a moderate degree of proficiency (Geary et al. 2008).

“Overlearning” means practicing recall beyond mastery: using “retrieval” strategies such as flashcards, mnemonics, or sequence recitation (methane, ethane …) to the point of mastery, and doing so repeatedly (Willingham 2004).

To learn math and science, why is such intense study required?  For many types of learning, summary (“gist”) memory is sufficient, and a species brain built for speech comprehension is good at summarizing meaning (Pinker 1994).  But the relationships of math and science often require exact (verbatim) memory (6 times 7 is not “about 40” and phosphate ion isn’t “PO something”), which the brain finds more difficult to store (Geary et al. 2008).

Initial memorization of fundamental relationships, though necessary, is not sufficient.  Neural links on a variety of characteristics must be constructed and weighted by solving problems in different contexts, so that elements can be fluently recalled when appropriate (Anderson et al. 2004, Willingham 2008).  Demonstrations, discussions, guided inquiry, and other forms of “active learning” can provide visual, spatial, auditory, and sequential associations for new knowledge. 

However, neural connections cannot form until after the elements being connected are stored in and recallable from LTM for processing.

To summarize:  according to science, to learn both math and science efficiently and effectively, students must begin each topic by “automating” (via “overlearning”) their recall of the fundamental relationships for the topic.

 

Should Calculator Use Be Restricted?

Let’s apply what science has recently learned to a practical question.  In chemistry, should we ask students to “refresh their memory” on the quick recall of the fundamental arithmetic facts, and then use those facts to solve calculations with simple numbers without a calculator?  Why not just use a calculator?  

Some reasons to encourage mental math in some cases may simply be common sense:

• During explanations of science based on simple whole-number ratios or proportional reasoning, students need to apply quick “mental math.” 
• Most of us would prefer that those building our bridges or administering our personal health care are able, by a mental math estimate, to check whether an answer from technology is reasonable.
• Students who need a calculator to balance an equation may find chemistry frustrating.  
• If students can’t solve calculations with simplified numbers using “pencil and paper” math, how will they remember the sequence to press on a calculator? 

Cognitive research offers additional arguments for “automated” recall of math facts.

• When a simple calculation like 56/8 must be done on a calculator, storing the answer mentally for transfer to paper or screen occupies a novel WM slot.  If novel WM is already full, this storage will bump out an element that may be needed during subsequent processing (Willingham 2009).
• For simple arithmetic, a calculator takes more time than automated recall, and during calculations, speed matters, because problem elements stored in novel WM begin to drop out after only a few seconds. 
• “Looking up” any data -- in a table, calculator, or computer -- tends to result in other novel elements being bumped out or “timed out” of WM, leading to confusion. 
• Automated recall helps to free slots in novel WM for “context cues” that distinguish different types of calculations.  Processing those cues concurrently is a key step in building conceptual frameworks (Willingham 2003, 2008, 2015).

To other operations, the same logic applies.  If students cannot solve log 1000 and ln(1) and ln(2.718) without a calculator, how well do they understand first-order rates, pH, free energy, or the Nernst equation?

Software is available that solves many types of standard general chemistry calculations.  Why shouldn’t students simply use this software rather than memorize procedures with problem-solving steps?  Willingham suggests the reason is: “Complex problems have simple problems embedded in them” (2009b).  If the logic of simple problems has not been automated, when students are presented with a problem they cannot find a computer program to solve, slots are not likely to be available in novel WM for managing the steps of problem-solving (Anderson et al. 2000, Clark 2010).

In calculations with complex numbers, calculators save time.  But during learning, if simple examples can be solved with recallable facts and procedures, more room is available in WM to note and process conceptual relationships.

 

Trends in Computation Skills

The cognitive science of 2017 predicts that students taught under 1989 NCTM-type standards will likely have substantial difficulty solving scientific calculations.  Do test scores support this prediction?

From 1990 to 2002, two states, Iowa and Virginia, reported statewide test scores for “math computation” separately from other math components (such as concept application, data interpretation, and reasoning).  During this period, in both states, computation scores plummeted.
 

Fig.7: Iowa 8^th grade ITBS Math Results, 1978-2001

 

Figure 8: Virginia 9^th Grade Math Results on the Stanford 9

 

For most states between 1990 and 2002, computation subtest scores were not reported, but the textbooks adopted in most U.S. states tended to be the same as in Iowa and Virginia (Hartman and Nelson 2016).

 

The Limits of Reasoning

In Figure 8, which line best measures preparation for scientific calculations?  Cognitive science says:  Computation.   

Cognitive studies have found that while reasoning skills in a discipline are important, they are nearly always limited in number and can be taught relatively quickly (Sweller, 2010).  The factor that is controlling is:  The knowledge you can reason with is limited almost entirely to what you can recall from LTM.  Initial retrieval practice, then linking neurons by varied practice, are the slow, and therefore rate determining, steps in learning. 

This new cognitive science explains why students have no choice but to be algorithmic problem-solvers.  Successful stepwise procedures (algorithms) limit the number of novel elements that must be retained in WM at any one time during processing (Willingham 2008, 2009).  Without applying algorithms, students attempting to reason must hold in WM the problem goal plus unique problem data while testing different reasoning steps.  Trying to do so tends to overload novel working memory (Clark et al. 2012). 

To “solve problems like a scientist,” you need the LTM of a scientist, a domain-specific conceptual framework of tens of thousands of stored and connected elements that requires years of study to construct (Willingham 2006). 

 

Since 2002

Since the 2002 passage of the No Child Left Behind Act, no state has reported student test scores in computation, so we cannot say for certain whether since 2002, computation skills are better, worse, or unchanged.  We do know that from 2002 to 2010, except for some New England states and Minnesota, most states maintained standards and textbooks aligned with the 1989 NCTM-standards (Hartman and Nelson 2016).

Between 2010 and 2015, most states gradually implemented standards similar to the 2010 “Common Core Math Standards” (CCMS).  Under the CCMS, some fundamentals, but not all, are to be automated (Nelson 2017).  The CCMS impact will not be clear until students are taught under CCMS-based curricula for most years of K-12 schooling.

For several years to come, most U.S. students will have spent most of their schooling under standards that substantially de-emphasized computation.

 

Skills Measured by International Standards

Though state data in computation are lacking, for college-age adults we have one recent test that measured numeracy skills.  In 2012, the Organization for Economic Co-operation and Development (OECD) tested proficiencies in problem-solving in 22 developed-world nations including the United States. In “numeracy” (solving problems with mathematical content), U. S. 16-24 year olds scored 22^nd out of 22:  dead last (OECD 2013, Goodman, Sands, Coley 2015).

Figure 9:  OECD PIAAC Test in Numeracy, 2012

Age 16-34 in Blue Bars, Age 16-24 as Red Dots

 

To Summarize

For millennia, students learned math and science by memorizing and applying fundamental facts and procedures.  With the advent of calculators and computers, theorists made the assumption that memorization could be “de-emphasized.”  In most U.S. states, K-12 math topics in computation were covered, but practice was reduced.  As a result, due to circumstances beyond the control of teachers and students, many current students entering higher education will lack the automaticity in math that science has recently verified is needed to reliably solve quantitative problems.

The brain’s requirement for initial memorization may be disappointing, but the good news is, we now know much more about how we can scientifically guide students in learning -- and guide students to “catch up” in math automaticity.

 

Our Experiments

In 2006, the author (retired from classroom teaching) began a collaboration with Dr. Donald J. Dahm at Rowan University.  For Engineering Chemistry, where two semesters of general chemistry was covered on the schedule of a one-semester course, we wrote tutorials that transferred parts of lecture and math review to homework. 

In the second year of these experiments, Don’s students scored at the 63^th percentile on the two semester ACS General Chemistry Exam at the end of the one semester course, with high student retention (Hartman et al. 2015b).  Don subsequently used portions of the tutorials in “mostly bio major” gen chem and “prep chem.”  Our assessment was: 

• Homework based on “overlearn then practice,” if combined with short but frequent quizzes that encouraged steady homework completion, can help most students fill in math gaps. 
• Better-math-prepared students can catch up in homework concurrent with gen chem. 
• Less-well-prepared need “prep chem” that contains a quite a bit of “prep math.”  
• Some will need a full 3 credits of “computation for the sciences” -- which is more math than our materials provide.

Our current “math then chem” homework tutorials are available as a 500 page paperback for use as a “prep chem” textbook or first-semester gen chem supplement, and a 1,250 page gen chem ebook or multi-volume paperback. 

What users say about our approach is in the reviews at Amazon:  www.amazon.com/Calculations-Chemistry-Introduction-Donald-Dahm/dp/0393912868/  .  Review copies of all versions are available from the publisher representative.

But below are some suggestions for cognitive experiments in chemistry that you can try without having students buy any materials from anyone.

 

Assessing Skills

Chemistry assumes 12 years of K-12 preparation in computation fundamentals.  Science says that to reliably solve calculations, students must automate recall of math fundamentals.  How can you determine if math automaticity is an issue in your student populations?

• One option is a quick “computation quiz” for a student sample using the multiple-choice assessment in Dr. Doreen Leopold’s paper for this conference, or for small sections, a one-page quiz available at http://bit.ly/1HyamPc .  A quiz now (after the semester starts), when memory has been refreshed, may be a good measure of skills they can call upon during the semester.  Correlations between automaticity, course grades, and retention may be of interest.
• Does your department employ a test to place students into general vs. preparatory chemistry?  Can a section of the test results report on “calculations without a calculator?”  A test emphasizing “reasoning” may not measure calculation proficiency (see Figure 8).   

 

Suggestions for Experiments

If automated math fundamentals are found to be a concern, experiments you might consider include:

1.   Help Students Automate “Mental Arithmetic”

At http://chemreview.net/blog/?p=409  is posted a 4-minute quiz on mental arithmetic. For students found to have gaps, activities are suggested to fill them.

2.   Quizzes without a Calculator

Consider assigning occasional “simplified number” problems to be solved without a calculator, accompanied by announced short quizzes to be completed without a calculator.  Or -- design multiple choice questions and quizzes where numeric choices differ by 50%, and have students choose correct answers based on mental math estimation.

3.   Try Homework to Fill Gaps and Open Time for In-Class Activities

At http://chemreview.net/blog/?p=254 , are 50 pages of free exponential and metric homework that can be assigned in whole or selectively at all first-year levels with editable quiz files included.

4.   Help Students Automate Chemistry Fundamentals

At http://chemreview.net/blog/?p=268 , homework assignments (and quiz suggestions) help students automate the knowledge relied upon most often in chemistry.

5.   Help your students learn how to study.  Consider assigning:

a.   The six short Strategies for Effective Learning videos from: 

www.learningscientists.org/videos   

b.   A more detailed “How to Study” post or two from the list at www.learningscientists.org/blog/2016/9/1-1?rq=Learn%20how%20to%20study%20using   

c.   Articles on the impact of “social media addiction” on learning -- at

http://www.learningscientists.org/blog/2017/8/27/weekly-digest-74

d.   The “Learning Chemistry” lesson at www.chemreview.Net/Lesson1.1.pdf   

6.   Explore how cognitive experts say instructors can help students learn.

a.   See “Brain 101” on pages 8 to 11 at:

http://www.aft.org/pdfs/americaneducator/spring2012/Clark.pdf

b.   Check out Make It Stick: The Science of Successful Learning (Harvard University Press, 2014) on the value of retrieval, distributed, and interleaved practice.

c.   Pick a topic of interest:  Critical thinking, spaced study, factual, procedural and conceptual knowledge, or helping students construct memory.  Then see short articles by Daniel Willingham at:

http://chemreview.net/blog/?p=321

7.   End the need for “prep chem-math.”

For the sake of your students, children, grandchildren, and nation, through your professional organizations, talk to your state elected officials about aligning their K-12 math standards with the findings of cognitive science on how students learn.

As instructors, if we experiment with putting into practice what science has learned about how the brain solves problems, will our students benefit?

#  #  #  #  #

References

Anderson, J. R., Bothell, D., Byrne, M. D., Douglass, S., Lebiere, C., Qin, Y.: An integrated theory of the mind. Psychological Review 111(4) 1036 (2004)

Anderson, J., Lebiere, C., (Eds.): The Atomic Components of Thought.  Erlbaum Publishers, Mahwah, NJ (1998)

Anderson, J., Reder, L., Simon, H.: Applications and misapplications of cognitive psychology to mathematics education. Tex. Educator Rev. 1(2) 29-49 (2000)

Brown, P., Roediger, H., McDaniel, M.: Make it Stick: The Science of Successful Learning. Harvard University Press, Cambridge, MA (2014)

Cavanaugh, S.:  College-K-12 projects aim to patch STEM pipeline.  Education Week, Sept. 17 (2008)

Clark, R., Sweller, J., Kirschner, P.:  Putting students on the path to learning:  The case for fully guided instruction.  Am. Educator, 36(1), 6-11 (2012)

Clark, R.: Cognitive and neuroscience research on learning and instruction: Recent insights. In: 11th International Conference on Education Research (ICER), Seoul, South Korea (2010) http://www.aect.org/publications/whitepapers/2010/ICER3.pdf    Accessed 12 May 2015

Common Core State Standards Initiative:  Common Core State Standards for Mathematics. National Governors Association and the Council of Chief State School Officers, Washington, DC (2010)

Cowan, N.: The magical mystery four:  How is working memory capacity limited, and why? Current Directions in Psychological Science 19(1), 51-57 (2010)

Cowan, N.: The magical number 4 in short-term memory: A reconsideration of mental storage capacity. Behav. and Brain Sciences 24, 87-185. (2001)

Ericsson, K., Kintsch, W.: Long-term working memory. Psychological Rev. 102, 211-245 (1995)

Geary, D., et al.:  Chapter 4 - Report of the Task Group on Learning Processes.  In: Foundations for Success: The Final Report of the National Mathematics Advisory Panel, pp. 4-2 to 4-11. US Dept. of Education, Washington, DC (2008)  https://www2.ed.gov/about/bdscomm/list/mathpanel/report/learning-processes.pdf  Accessed 27 August 2017

Goodman, M., Sands, A., Coley, R.:  America’s Skills Challenge: Millennials and the Future.  Educational Testing Service, Princeton, NJ (2015)

Hartman, J., Dahm, D, Nelson, E:  ConfChem conference on flipped classroom: Time-saving resources aligned with cognitive science. J. Chem. Ed. 92.9 (2015b): 1568-1569.

Hartman, J., Nelson E.: “Do we need to memorize that?” or cognitive science for chemists.  Foundations of Chemistry, 17(3), 263-274  (2015)

Hartman, J., Nelson E.: Automaticity in computation and student success in introductory physical science courses. http://arxiv.org/abs/1608.05006 (2016)

Hebb, D.: The Organization of Behavior. New York: Wiley (1949).

Klein, D., Braams, B., Parker, T., Quirk, W., Schmid, W., Wilson, W.: The State of the State Math Standards 2005. Fordham Foundation, Washington, DC (2005)

Koedinger, K., Corbett, A., Perfetti, C.: The Knowledge-Learning-Instruction (KLI) framework: Bridging the science-practice chasm to enhance robust student learning. Cognitive Science, 36 (5), 757-798 (2012)

Kuhn, T. S.: The Structure of Scientific Revolutions. University of Chicago Press: Chicago, (1962).

National Center on Education and the Economy: What Does It Really Mean to Be College and Work Ready?  The Mathematics Required of First Year Community College Students. (2013). http://www.ncee.org/college-and-work-ready/  Accessed 27 August 2017

National Council of Teachers of Mathematics, Curriculum and Evaluation Standards for School Mathematics, NCTM, Reston, VA (1989)

National Science Board:  Science and Engineering Indicators 2012. Appendix Table 2-12  (2012) http://www.nsf.gov/statistics/seind12/c2/c2s2.htm.  Accessed 27 August 2017

National Science Board: Science and Engineering Indicators 2002. Appendix Table 2-11 (2002)

National Science Foundation:  Science and Engineering Degrees: 1966–2012, Tables 5, 37, 38, 46 (2015) https://www.nsf.gov/statistics/2015/nsf15326/ Accessed 27 August 2017

Nelson, E.: Cognitive science and the common core mathematics standards. Nonpartisan Ed. Rev. 13.3 (2017)

OECD: Literacy, Numeracy, and Problem Solving in Technology-Rich Environments – Framework for the OECD Survey of Adult Skills.  OECD Publishing, Paris (2013).

Olson, S., & Riordan, D.: Engage to Excel: Producing One Million Additional College Graduates with Degrees in Science, Technology, Engineering, and Mathematics, Report to the President. Executive Office of the President, Washington, DC (2012)

Peterson, L., Peterson, M.:  Short-term retention of individual verbal items.  J. of Experimental Psychol. 58, 193-198 (1959)

Pinker, S.: The Language Instinct: How the Mind Creates Language. Wm. Morrow, New York (1994)

Sweller, J.: What human cognitive architecture tells us about constructivism.  In: Tobias S. and Duffy T. (Eds.) Constructivist Instruction: Success or Failure? pp. 127-143. Routledge, New York (2009)

Sweller, J., Clark, R., Kirschner, P.: Teaching general problem-solving skills is not a substitute for, or a viable addition to, teaching mathematics. Notices of the Am. Mathematical Society 57(10), 1303-1304 (2010)

Waits, B., Demana, F.: Calculators in mathematics teaching and learning. In: Burke, M. and Curcio, F (Eds.) Learning Mathematics For A New Century.  National Council of Teachers of Mathematics, Reston, VA (2000)

Willingham, D.:  How knowledge helps. Am. Educator 30(1), 30-37 (2006)

Willingham, D.: Practice makes perfect—but only if you practice beyond the point of perfection. Am. Educator 28(1), 31-33 (2004)

Willingham, D.: Students remember what they think about. Am. Educator 27(2), 37-41 (2003)

Willingham, D.: What will improve a student’s memory? Am. Educ. 32(4), 17-25 (2008)

Willingham, D.: Why don’t students like school?  Am. Educator 33(2), 4-13 (2009)

Willingham, D.: Is it true that some people just can’t do math?  Am. Educator 33(4), 14-19 (2009b)

Willingham, D.: Do students remember what they learn in school? Am. Educ. (Summer) 7 (2015)