The Story of our Evaluation

Written by: Dr. Volker Ecke and Dr. Christine von Renesse

This is going to be a somewhat different blog than usual. We are going to describe how we tried to measure the success of IBL, how hard evaluation is, and what we learned from our process. It also serves as an overview of the many different types of evaluation efforts we used in DAoM over the course of two NSF funding periods.

A short Paper Survey (1997 - 2011)

In 1997, Julian Fleron started to work on the first DAoM inquiry-based learning guide, focused on Infinity, with a goal to shift students' perceptions of mathematics, their confidence to engage in mathematics, and their enjoyment of the process of doing mathematics.

To measure changes in students' beliefs and attitudes, he created a pre/post paper survey (not matched), which has been administered to over 1000 students over the years ( full report ). Strong changes are seen in students' like or dislike of mathematics, their enjoyment in doing mathematics, as well as their perception of beauty in mathematics, before and after taking this class.

Experimenting with CLASS-MATH survey (2008-09)

While continuing to use the short paper survey we started looking for a more powerful instrument to measure students' beliefs and attitudes. As part of our "Open Inquiry Classroom" collaboration with Mairead Greene at Rockhurst University, we learned about the "Colorado Learning Attitudes about Science Survey ( CLASS" ) at a RUME conference. Building on the existing survey for physics, we created a survey for mathematics that set out to probe student beliefs about what it means to do and learn mathematics. The goal was to detect a shift in student beliefs from more "novice"-like views to those closer to an "expert" mathematician. For example, one statement reads To learn mathematics I only need to memorize solutions to sample problems. While students' experience in K-12 mathematics might make that a reasonable conclusion, it is --in our view-- not sufficient to make sense of challenging, contextual, mathematical investigations we expect students to solve in our IBL classes.

We realized, however, that we didn't agree among ourselves on novice and expert answers to some of the survey statements, such as When I solve a mathematics problem, I locate an equation that uses the variables given in the problem and plug in the values. This may in fact be a reasonable approach for large segments of mathematics problems students may encounter in their lives (e.g. mortgage payments, credit card payoff calculations, or measurements in arts and crafts).

In addition to our unease with "novice" versus "expert" views on several survey questions, we did not trust our ability to validate the survey we created, or to apply the sophisticated methods to analyze the CLASS studies. So when a more established, validated alternative came along, we jumped ship.

Using the Colorado survey (2009-2011)

At the 2010 RUME conference, we met Sandra Laursen and Marja-Liisa Hassi from the Colorado Ethnography and Evaluation Research group , who were at the time leading a major study into Inquiry-Based Learning in Mathematics. Since their target audiences (mathematics majors and future teachers) were different from ours (mathematics for liberal arts) we adapted their survey slightly, you can download our POST survey here. The pre and the post survey data were matched this time.
All four professors of the DAoM team used the survey in our Math for Liberal Arts classes from 2009-2011.

Our results show for all items a change in the direction we hoped for. Most importantly, we found a significant increase in student enjoyment, interest and interaction. We were glad that our results were similar to the results of the large study by Sandra Laursen et al. We also found some interesting gender differences. For the scales of group work, instructor-centered and extrinsic goals, the changes were highly significant for women but not significant at all for men.

We noticed however, that some students didn't complete the survey and we were wondering if survey fatigue was influencing the data in some way. The survey was very long and it included some items that were not as relevant for our majors (e.g. talking explicitly about proof and rigorous reasoning) and missed some others (e.g. beauty of mathematics). When we received our next NSF grant we decided therefore to develop our own survey (see next section) with the help of our evaluators.

In addition to the survey above we also let students from all our MLA classes write journals given the following journal prompts:
Journal 1
Journal 2
Journal 3.

We read the journals and noticed that they show exactly what we were looking for! Unfortunately coding the hundreds of journals and looking for some nice data to prove our observations turned out to be very time intensive and difficult. After spending a lot of time coding journals we decided that we didn't have the manpower (and maybe the know-how) to create useful data from the journals. What we are left with is a nice word cloud:

We decided to include some of the open questions of our journals into our new survey:

Creating our own DAoM Survey (2013-2015)

We developed our current survey from the student goals that our grant is based on:

Student Meta Goals

• Students will appreciate mathematics as a human endeavor which is one of our most fundamental intellectual pursuits.
• Students will understand that mathematics is a vital, rapidly growing field of inquiry with a dedicated cohort of practitioners.
• Students will understand the continued impact of mathematics in shaping history, culture, logic, philosophy, and knowledge, as well as its role as a humanistic and aesthetic discipline.
• Students will understand the ubiquitous role of mathematics in the world around them.
• Students will strengthen their reasoning skills and become better problem solvers.
• Students will strengthen their skills in reading, writing, argumentation and speaking.
• Students will become more self-monitoring, reflective learners and take greater personal responsibility for their learning.
• Students will approach mathematics more positively and gain a balanced perspective of mathematics.
• Students will improve their mathematical confidence.
• Students will develop awareness of the negative impact of broadly-held societal views.
• Students will be capable of and interested in considering mathematics outside of the confines of the classroom, understanding the value of life-long learning in mathematics.

So far, we surveyed about 460 students and our results show that not only do all changes occur in the desired direction but also most changes are significant and the effect sizes are now larger. We are very excited that our new survey provides evidence of the positive effect of our classes on students' beliefs and attitudes. Some of the most significant changes occurred for the following prompts:

• A major reason why mathematicians engage in mathematics is because they find it beautiful.
• Mathematics requires curiosity.
• Mathematics requires creativity.
• Doing mathematics for its own sake is as valuable as composing music, creating art, or other intellectual endeavors.
• There are debates within mathematics.
• My thoughts and ideas matter when solving a mathematical problem.
• Thoughts and ideas that come to me while working on mathematical problems will often get me closer to a solution.
• I don’t think mathematics can be beautiful.
• I am able to understand and critique written or spoken mathematical arguments.

Again, we tried to match pre and post surveys, but unfortunately we had problems with students forgetting to submit their name (electronically). Therefore we are looking for now at the unmatched results. We hope that in the last year of our grant we can improve our matched numbers so we can present these as well.

So far, we have not see any interesting gender differences in our data, which is surprising to us.

Westfield State University Institutional Data Study

In 2014 we looked hard at our institutional data to see if we can make any larger impact claims about our classes. Maybe students' improved critical thinking skills would make them stronger students in general? Would they be more likely to graduate? At first glance the data looked very promising but when we thought about it more deeply we realized that we could not be sure that the observed changes were really due to our classes. Between different majors, other mathematics classes, different times at which students took our course etc it was impossible to find statistical evidence for our claims.

Our next goal is to survey seniors that took our class at some point. He want to investigate if their beliefs and attitudes changes are still the same a few years later. This would show the larger impact of our classes.

Assessing Meta Skills

In addition to our students' self reported beliefs and attitudes we also hope to more objectively measure students' meta skills such as communication skills, persistence, confidence, team work skills, self reflection, and reasoning skills. With the help of Mairead Greene we developed an assessment tool that we will try out this coming year. It has been very interesting (and difficult!) to think about how to assess meta skills instead of the usual content skills. More to come ... :)

Other tools:

The SALG (Student Assessment of Learning Gains) is a tool for self assessing learning gains.