Significance of PISA math results

A new round of two international comparisons of student mathematics performance came out recently and there was a lot of interest because the reports were almost simultaneous, TIMSS[1] in late November 2016 and PISA[2] just a week later. They are often reported as 2015 instead of 2016 because the data collection for each was in late 2015 that would seem to improve the comparison even more. In fact, no comparison is appropriate; they are completely different instruments and, between them, the TIMSS is the one that should be of more concern to educators. Perhaps surprising and with great room for improvement, the US performance is not as dire as the PISA results would imply. By contrast, Finland continues to demonstrate that its internationally recognized record of PISA-proven success in mathematics education – with its widely applauded, student-friendly approach – is completely misinforming.

In spite of the popular press and mathematics education folklore, Finland’s performance has been known to be overrated since PISA first came out as documented by an open letter[3] written by the president of the Finnish Mathematical Society and cosigned by many mathematicians and experts in other math-based disciplines:

“The PISA survey tells only a partial truth of Finnish children’s mathematical skills” “in fact the mathematical knowledge of new students has declined dramatically”

This letter links to a description[4] of the most fundamental problem that directly involves elementary mathematics education:

“Severe shortcomings in Finnish mathematics skills” “If one does not know how to handle fractions, one is not able to know algebra”

The previous TIMSS had the 4th grade performance of Finland as a bit above that of the US but well behind by 8th. In the new report, it has slipped below the US at 4th and did not even submit itself to be assessed at 8th much less the Advanced level. Similar remarks apply to another country often recognized for its student-friendly mathematics education, the Netherlands, home of the PISA at the Freudenthal Institute. This decline was recognized in the TIMSS summary of student performance[1]with the comparative grade-level rankings as Exhibits 1.1 and 1.2 with the Advanced[5] as Exhibit M1.1:

pastedimageBy contrast, PISA[2] came out a week later and…

Netherlands 11
Finland 13
United States 41

Note: These include China* (just below Japan) of 3 provinces, not the country – if omitted, subtract 1.

Why the difference? The problem is that PISA was never for “school mathematics” but for all 15-year-old students in regard to their “mathematics literacy[6]”, not even mathematics at the algebra level needed for non-remedial admission to college much less the TIMSS Advanced level interpreted as AP or IB Calculus in the US:

“PISA is the U.S. source for internationally comparative information on the mathematical and scientific literacy of students in the upper grades at an age that, for most countries, is near the end of compulsory schooling. The objective of PISA is to measure the “yield” of education systems, or what skills and competencies students have acquired and can apply in these subjects to real-world contexts by age 15. The literacy concept emphasizes the mastery of processes, understanding of concepts, and application of knowledge and functioning in various situations within domains. By focusing on literacy, PISA draws not only from school curricula but also from learning that may occur outside of school.”

Historically relevant is the fact that conception of PISA at the Freudenthal Institute in the Netherlands included heavy guidance from Thomas Romberg of the University of Wisconsin’s WCER and the original creator of the middle school math ed curriculum MiC, Mathematics in Context. Its underlying philosophy is exactly that of PISA, the study of mathematics through everyday applications that do not require the development of the more sophisticated mathematics that opens the doors for deeper study in mathematics; i.e., all mildly sophisticated math-based career opportunities, so-called STEM careers. In point of fact, the arithmetic of the PISA applications is calculator-friendly so even elementary arithmetic through ordinary fractions – so necessary for eventual algebra – need not be developed to score well.

 

[1] http://timss2015.org/timss-2015/mathematics/student-achievement/
[2] http://nces.ed.gov/pubs2017/2017048.pdf (Table 3, page 23)
[3] http://matematiikkalehtisolmu.fi/2005/erik/PisaEng.html
[4] http://matematiikkalehtisolmu.fi/2005/erik/KivTarEng.html
[5] http://timss2015.org/advanced/ [Distribution of Advanced Mathematics Achievement]
[6] https://nces.ed.gov/timss/pdf/naep_timss_pisa_comp.pdf

Wayne Bishop, PhD
Professor of Mathematics, Emeritus
California State University, LA

John Hopkins flawed report on Kentucky

It looks like a recent, very problematic report from Johns Hopkins University, “For All Kids, How Kentucky is Closing the High School Graduation Gap for Low-Income Students,” is likely to get pushed well beyond the Bluegrass State’s borders.

The publishers just announced a webinar on this report for August 30th.

Anyway, you need to get up to speed on why this report is build on a foundation of sand. You can do that fairly quickly by checking these blogs:

http://www.bipps.org/uneven-quality-kentuckys-high-school-diplomas/

http://www.bipps.org/quality-control-problems-kentuckys-high-school-diplomas-part-1/

A third blog will release at 8 am Eastern tomorrow. It will probably link at

http://www.bipps.org/quality-control-problems-kentuckys-high-school-diplomas-part-2/

I won’t know for sure until it releases, however.

Let me know if you have questions and especially if this Hopkins report starts making the rounds in your state.

101 Terms for Denigrating Others’ Research

In scholarly terms, a review of the literature or literature review is a summation of the previous research conducted on a particular topic. With a dismissive literature review, a researcher assures the public that no one has yet studied a topic or that very little has been done on it. Dismissive reviews can be accurate, for example with genuinely new scientific discoveries or technical inventions. But, often, and perhaps usually, they are not.

A recent article in the Nonpartisan Education Review includes hundreds of statements—dismissive reviews—of some prominent education policy researchers.* Most of their statements are inaccurate; perhaps all of them are misleading.

“Dismissive review”, however, is the general term. In the “type” column of the files linked to the article, a finer distinction is made among simply “dismissive”—meaning a claim that there is no or little previous research, “denigrating”—meaning a claim that previous research exists but is so inferior it is not worth even citing, and “firstness”—a claim to be the first in the history of the world to ever conduct such a study. Of course, not citing previous work has profound advantages, not least of which is freeing up the substantial amount of time that a proper literature review requires.

By way of illustrating the alacrity with which some researchers dismiss others’ research as not worth looking for, I list the many terms marshaled for the “denigration” effort in the table below. I suspect that in many cases, the dismissive researcher has not even bothered to look for previous research on the topic at hand, outside his or her small circle of colleagues.

Regardless, the effect of the dismissal, particularly when coming from a highly influential researcher, is to discourage searches for others’ work, and thus draw more attention to the dismisser. One might say that “the beauty” of a dismissive review is that rival researchers are not cited, referenced, or even identified, thus precluding the possibility of a time-consuming and potentially embarrassing debate.

Just among the bunch of high-profile researchers featured in the Nonpartisan Education Review article, one finds hundreds of denigrating terms employed to discourage the public, press, and policymakers from searching for the work done by others. Some in-context examples:

  • “The shortcomings of [earlier] studies make it difficult to determine…”
  • “What we don’t know: what is the net effect on student achievement?
    -Weak research designs, weaker data
    -Some evidence of inconsistent, modest effects
    Reason: grossly inadequate research and evaluation”
  • “Nearly 20 years later, the debate … remains much the same, consisting primarily of opinion and speculation…. A lack of solid empirical research has allowed the controversy to continue unchecked by evidence or experience…”

To consolidate the mass of verbiage somewhat, I group similar terms in the table below.

(Frequency)   Denigrating terms used for other research
(43)   [not] ‘systematic’; ‘aligned’; ‘detailed’; ‘comprehensive’; ‘large-scale’; ‘cross-state’; ‘sustained’; ‘thorough’
(31)    [not] ‘empirical’; ‘research-based’; ‘scholarly’
(29)   ‘limited’; ‘selective’; ‘oblique’; ‘mixed’; ‘unexplored’
(19)   ‘small’; ‘scant’; ‘sparse’; ‘narrow’; ‘scarce’; ‘thin’; ‘lack of’; ‘handful’; ‘little’; ‘meager’; ‘small set’; ‘narrow focus’
(15)   [not] ‘hard’; ‘solid’; ‘strong’; ‘serious’; ‘definitive’; ‘explicit’; ‘precise’
(14)   ‘weak’; ‘weaker’; ‘challenged’; ‘crude’; ‘flawed’; ‘futile’
(9)    ‘anecdotal’; ‘theoretical’; ‘journalistic’; ‘assumptions’; ‘guesswork’; ‘opinion’; ‘speculation’; ‘biased’; ‘exaggerated’
(8)    [not] ‘rigorous’
(8)    [not] ‘credible’; ‘compelling’; ‘adequate’; ‘reliable’; ‘convincing’; ‘consensus’; ‘verified’
(7)    ‘inadequate’; ‘poor’; ‘shortcomings’; ‘naïve’; ‘major deficiencies’; ‘futile’; ‘minimal standards of evidence’
(5)    [not] ‘careful’; ‘consistent’; ‘reliable’; ‘relevant’; ‘actual’
(4)    [not] ‘clear’; ‘direct’
(4)    [not] ‘high quality’; ‘acceptable quality’; ‘state of the art’
(4)    [not] ‘current’; ‘recent’; ‘up to date’; ‘kept pace’
(4)    ‘statistical shortcomings’; ‘methodological deficiencies’; ‘individual student data, followed school to school’; ‘distorted’
(2)    [not] ‘independent’; ‘diverse’

As well as illustrating the facility with which some researchers denigrate the work of rivals, the table summary also illustrates how easy it is. Hundreds of terms stand ready for dismissing entire research literatures. Moreover, if others’ research must satisfy the hundreds of sometimes-contradictory characteristics used above simply to merit acknowledgement, it is not surprising that so many of the studies undertaken by these influential researchers are touted as the first of a kind.

* Phelps, R.P. (2016). Dismissive reviews in education policy research: A list. Nonpartisan Education Review/Resources/DismissiveList.htm
http://nonpartisaneducation.org/Review/Resources/DismissiveList.htm