Hattie’s synthesis of problem-based learning and problem-solving teaching

In a previous post I critiqued John Hattie’s analysis of inquiry-based teaching as reported in his book Visible Learning: A synthesis of over 800 meta-analyses relating to achievement. In this post, I critique his analysis of problem-based learning and problem-solving teaching.

Problem-based learning (PBL) is an inquiry approach. A PBL cycle starts with a real-world, authentic problem, and involves gathering information and using it to solve the problem.

Problem-based learning is underpinned by problem-solving skills and processes. However, problem-solving per se doesn’t necessarily involve problem-based learning. As Killen (2014, p. 257) explains:

When you teach for problem-solving you concentrate on helping learners to acquire the knowledge, understandings and skills that are useful for solving problems (you provide them with the foundation for later problem-solving). When you teach about problem-solving you concentrate on the processes for problem-solving (you teach learners how to solve problems)….[when you teach through problem-solving you use] problem solving as a technique for helping students to learning other things.

Killen (2014) contrasts the examples of teaching students how to solve maths word problems versus planning and conducting an OH&S audit of their school. The former involves teaching students how to solve problems using their existing knowledge, whereas the latter involves students ‘developing new knowledge through solving problems’ (p. 259). It is this type of problem-solving which underpins problem-based learning. However, I have found that PBL as Killen describes it seems to be rarely referred to by this name in school settings. Alternative terms such as inquiry learning, project-based learning and, more recently design thinking are used instead.

What is problem-based learning?

When PBL is mentioned in the literature it generally refers to a particular higher education medical curriculum model where the curriculum is centred on a predetermined set of ill-defined problems in the form of clinical cases. The medical model of PBL was developed in the 1960s as an alternative to traditional teaching methods (Albanese & Mitchell, 1993). As Albanese and Michell (1993, p. 52) pointed out in their meta-analysis:

There has to be a better way to train physicians than to make medical students sit through endless hours of lectures and then test their ability to recall bits of trivia on a multiple-choice test.

Savery and Duffy (2001) outline a typical medical PBL cycle:

  1. Students are presented with the detail of a clinical case (real or hypothetical)
  2. Initial small group meeting where students reason through the problem, suggest hypotheses and identify knowledge gaps
  3. Independent study (i.e. seeking, gathering and summarising information from books and journal articles)
  4. Small group meeting where students present knowledge, evaluate their sources and re-examine the problem
  5. Peer and self –assessment

PBL aims at developing clinical reasoning, problem-solving and self-directed learning. In this process, the role of the teacher is to construct the problem and to act as consultant or tutor. At its purest, PBL does not include lectures, assigned texts, or an exam though there is a continuum of PBL that includes teacher-directed elements (Barrows 1989).

Hattie’s problem-based learning sources

I was able to obtain all eight sources used by Hattie in his synthesis. The studies Hattie includes are almost all studies of medical education. Of the eight studies, five investigate higher education medical curriculum (i.e. training medical doctors) (Albanese & Mitchell, 1993; Dochy, Segers, Van den Bossche, & Gijbels, 2003; Gijbels, Dochy, Van den Bossche, & Segers, 2005; Vernon & Blake, 1993). Of the reminder one is in a higher education nursing context (Newman, 2004), and one is in a range of higher education disciplines (Walker & Leary, 2009).

Only one study (Haas, 2005) is in a school context (secondary school). Haas’ study examines a range of teaching methods for secondary algebra, where PBL is addressed along with other methods such as cooperative learning, communication and study skills, and direct instruction.

As Hattie’s synthesis is aimed at informing the K-12 sector, it is hard to understand why Hattie included so many higher education studies in his synthesis. In particular, the medical PBL model is so distinct that it would be difficult to see how any K-12 teacher could draw conclusions from these studies for their own practice.

What is problem-solving teaching?

Hattie (2009, p. 210) describes problem-solving as:

The act of defining or determining the cause of the problem; identifying, prioritizing and selecting alternatives for a solution; or using multiple perspectives to uncover issues related to a particular problem, designing an intervention plan, and then evaluating the outcome.

This is a broad description, which relates well to an inquiry approach. The authors that Hattie draws on for the synthesis describe problem-solving in similar terms. However, they provide more nuances that illuminate the nature of the studies in their meta-analyses. For instance, Hembree (1992) points out that that there are different types of maths problems including standard maths problems (those found on standardised tests), puzzle problems and ‘real-world’ problems such as ‘How much paper of all kinds does your school use in a month?’ (p. 249). This aligns with Mellinger’s (1991) description of two sorts of problems as:

  • Well-structured – ‘one has access to both the information necessary and an appropriate algorithm or heuristic to find the correct solution’) (p. 2); and
  • Ill-structured – ‘problems are unclear, algorithms are lacking and there may not be a correct answer.’ (p. 3)

It’s likely that the sort of teaching method used to help students learn problem solving may differ if the problem is standard, linear and well-structured, as opposed to a real-world, open-ended and ill-structured. It’s this issue that makes it so difficult to evaluate meta-analyses that bundle teaching and learning approaches such as problem-solving into one metric. As I point out below, even when one teaching or learning approach can be examined, its effect may be different in different contexts such as age group and subject area, thus one metric is an inappropriate for teachers to make decisions on pedagogy and curriculum.

Hattie’s problem-solving sources:

I was able to obtain five of six sources used by Hattie in examining problem-solving teaching. They address quite different issues in different subject areas and different educational contexts:

Alemida and Denham’s meta-analysis (1984) deals with children’s inter-personal cognitive problem-solving skills relating to social competence and behavioural adjustment. This sort of study seems to be directed at behaviour management rather than the effectiveness of classroom teaching methods to learn domain knowledge.

Mellinger’s (1991) PhD examines the development of cognitive flexibility as one component of problem-solving (other components being creativity, originality, ideational and associational fluency, synthesis, reasoning, and insight’ (p. 8). She included students involving preschool to adult subjects, with the majority being in elementary school.

Curbelo’s PhD (1984) examined problem-solving in school maths and science teaching.

Taconis et al (2001) examined problem-solving in science from elementary school to first-year university. The authors explain that the range included ‘simple algorithmic problems of third- and fouth-grade mathematics to complex physics and chemistry problems at the level of first-year university courses. Some experiments worked with familiar problems, other with unfamiliar problems or with a mixture of both’ (p. 454).

Hembree’s (1992) paper reported a wide-ranging analysis of problem-solving in maths, with one component being teaching methods such as instruction versus practice, and use of heuristics. In relation to heuristics, the effect was different in the different levels of schooling e.g. the gains were small in primary/elementary school, substantial in high school and smaller in college (p. 263).

It’s worth noting the results of Hattie’s synthesis in terms of effectiveness in relation to student learning. In table 1 I have presented problem-solving teaching, inquiry-based teaching and problem-based learning in order of effect size.

Hattie deems an effect size (d) of 0.40 as being effective, as he argues that an effect size of 0.00-0.15 is what students could achieve with no schooling (due to maturation) while the effects of teachers would be d=0.15-0.40. He claims this is equal to one year of schooling which ‘is estimated from the findings in countries with no or limited schooling’ (p. 20). Unfortunately, there is no reference cited for this claim so I am unable to evaluate it. I would assume, however, that in countries with little or no schooling there are a number of variables that may affect learning such as war, famine, poverty and disease.

Table 1 demonstrates that problem-solving teaching has a strong effect size (0.61) with a CLE (common language effect) of 43% with an overall rank of 20/138. This means that 43% of students would gain in achievement using this approach (Hattie, 2008 p. 9). By contrast, problem-based learning has a low effect size (0.15) with a measly 11% of students gaining in achievement. In the case of the problem-based learning studies, as they are nearly all situated in higher education, I question the validity of an effect size illustrating gain (or not) of one year of schooling as this is not relevant to higher education. As an aside, if PBL is so poor as to be equal to no schooling at all, I wonder why so many university medical schools use this approach!

Approach Effect size (d) CLE Rank /138
Problem-solving teaching 0.61 43% 20
Inquiry-based teaching 0.33 22% 86
Problem-based learning 0.15 11% 118

Table 1: Effects of problem-solving teaching, inquiry-based teaching & PBL

The findings by Hembree (1992) that the effect size was different in the different levels of schooling illustrates a general problem with single metrics. This problem is recognised by Hattie in relation to homework, i.e. that the gains in primary school are much less (d=0.15) than the gains in secondary school (d=0.64) (p. 10). He also found that the effect size was different in different subjects. Despite this, Hattie provides an average effect size of 0.29 (p. 234) for homework, which is meaningless, given the variation in effects for different groups of students and contexts.

In conclusion, the weakness of Hattie’s synthesis is that:

  • The studies have different effect sizes for different contexts and different levels of schooling, thus averaging these into one metric is meaningless
  • The relevance of higher education contexts to K-12 contexts is dubious
  • The distinction between problem-solving in relation to solving well-structured problems and ill-structured problems should be teased out rather than bundled together



Albanese, M., & Mitchell, S. (1993). Problem-based learning: A review of literature on its outcomes and implementation issues. Academic Medicine, 68(1), 52-80.

Alemedia, C., & Denham, S. (1984). Interpersonal cognitive problem-solving: A meta-analysis. Paper presented at the Annual Meeting of the Eastern Psychological Assication, Baltimore, April 12-15.

Curbelo, J. (1984). Effecvts of problem-solving instruction on science and mathematics student achievement: Ameta-analysis of findings. PhD, Floria State University.

Dochy, F., Segers, M., Van den Bossche, P., & Gijbels, D. (2003). Effects of problem-based learning: a meta-analysis. Learning & Instruction, 13.

Gijbels, D., Dochy, F., Van den Bossche, P., & Segers, M. (2005). Effects of problem-based learning: A meta-analysis from the angle of assessment. Review of Educational Research 75(1), 27-61.

Haas, M. (2005). Teaching methods for secondary algebra: A meta-analysis of findings. NASSP Bulletin, 89(642), 24-46.

Hattie, J. (2009). Visible learning. A synthesis of over 800 meta-analyses relating to achievement. London: Routledge.

Hembree, R. (1992). Experiments and relational studies in problem solving: A meta-analysis. Journal for Research in Mathematics Education, 23(3), 242-273.

Killen, R. (2014). Effective teaching strategies. Lessons from research and practice (6th ed.). South Melbourne, Vic: Cengage Learning Australia.

Mellinger, S. (1991). The development of cognitive flexibility in problem solving: Theory and application. PhD, University of Alabama.

Newman, M. (2004). Problem based learning: An exploration of the method and evaluation of iots effectiveness in a continuing nursing education programme. London: School of Lifelong Learning & Education, Middlesex University.

Taconis, R., Ferguson-Hessler, M., & Broekkamp, H. (2001). Teaching ccience problem solving: An overview of experimental work. Journal of Research in Science Teaching, 38(4), 442-468.

Vernon, D., & Blake, R. (1993). Does problem-based learning work? A meta-analysis of evaluative research. Academic Medicine, 68(7), 550-562.

Walker, A., & Leary, H. (2009). A problem-based learning meta analysis: Differences across problem types, implementation types, disciplines, and assessment levels. Interdisciplinary Journal of Problem-Based Learning, 3(1), 12-43.




One response to “Hattie’s synthesis of problem-based learning and problem-solving teaching

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s