Jayson Nissen, California State University – Chico
John B. Buncher, North Dakota State University
Paul Emigh, Oregon State University
Daryl McPadden, Florida International University, Michigan State University
Caleb Speirs, University of Maine
Ben Van Dusen, California State University - Chico

At the Foundations and Frontiers in Physics Education Research (FFPER) 2017 conference we formed a working group to discuss the use of statistics within the PER community. Our purpose was to identify common challenges that the PER community faces in using statistics and potential solutions. A major challenge we identified is that it is difficult for our community to educate graduate students about study design, statistical analysis, and reporting practices. Physics departments often require PER students to take many physics courses, leaving little flexibility in what courses they can take. When students want to take statistics courses in other departments, it is often very difficult to find courses that meet their needs: the pace in introductory courses is too slow, subsequent courses require the introductory courses, and faculty are reluctant to let students skip the introductory courses. While PER students in Schools of Education often have easier access to statistics courses, they also face similar limitations in their choices and may not be encouraged to enroll in statistics courses. This leaves students with few options other than to teach themselves or to learn from their advisors, who may have also taught themselves. To address this challenge and others we identified, the working group focused on solutions that used both formal and informal structures to leverage the substantial expertise in our community to strengthen the community’s overall knowledge and use of statistical methods.

The challenges of doing high-quality statistical work are not unique to PER. Instead, the PER community’s challenges with statistics represent a much larger issue within science as a whole. The widespread prevalence of poor statistical methods in science has led the American Statistical Association (ASA) to release a statement on p-values¹ in which they state, “Statistical significance is not equivalent to scientific, human, or economic significance. Smaller p-values do not necessarily imply the presence of larger or more important effects, and larger p-values do not imply a lack of importance or even lack of effect.” The ASA released this statement, in part, as a response to the growing discourse in the media around the validity of science and prominent statisticians pointing to p-values and poor statistical methods as a culprit in the ‘replication crisis’ that is besetting science. Transforming how science as a whole uses statistics is a daunting but necessary task². Focusing on improving our community’s use of statistical methods is a task well within the abilities and expertise of our community that can contribute to the scientific community as a whole. In drafting our proposed solutions we focused on activities that will enable rich conversations within our community.

The working group proposed five activities to improve the PER community’s use of statistics.

1. Develop a central location for sharing knowledge and resources about statistics.
2. Run workshops at the PER Conference and AAPT Meetings.
3. Develop or identify online learning materials to support graduate students and PER newcomers in learning statistics.
4. Produce resource articles directly for the PER community.
5. Create a space to critique each other’s work constructively.

We focused on the first three activities because these were broader in scope. We did not focus on producing resource articles for the PER community because the recent call for papers³ in Physical Review Physics Education Research on quantitative methods will hopefully meet this need. As for critiquing others work, we hope that this article and ongoing efforts to improve statistics in PER can support constructive critiques being published in peer reviewed journals where they can have the greatest impact on our community.

The working group expressed a strong desire for a central online location for sharing statistical knowledge and resources specific to our community. The location that seems most suited to meet this desire is the PER wiki (http://www.compadre.org/per/wiki/) hosted by PER-Central. As a wiki, this resource would be editable by members of the community and it could change fluidly to meet any changing needs and knowledge. While many members of our community already access PER-Central, links from other PER organizations (such as PERCoGs, PhysPort, and the AAPT) would help increase the visibility of the new resources. The largest barrier to developing a Wiki on PER Statistics is soliciting volunteers to generate the initial content. To kick start this development, we have surveyed several members of the community to identify individuals who have the requisite knowledge of statistics and can contribute to the project.

With the PER-Central Wiki serving as a centralized location for curated statistical resources for individual study, we wanted to create opportunities for practitioners to work together with experts. We identified AAPT/PERC workshops as an ideal structure for a focused session on applying and interpreting statistical methods for different types of data. A key aspect of these workshops will be to enable attendees to work with each other and with experts to analyze their own data and answer statistical questions from their current projects. We anticipate piloting the workshop at the 2018 PER Conference. The end goal is an AAPT workshop series comprising two parts: a morning workshop to introduce graduate students and PER newcomers to statistical methods for their current research projects and an afternoon session focused on advanced topics that would change from year to year (e.g., Hierarchical Linear Models or Network Analysis).

Regarding online learning materials, our group could not identify an existing online resource that would meet the needs of the PER community: a course that balanced mathematical rigor with the conceptual underpinnings of introductory statistics in contexts practical for PER. We propose gathering existing materials and writing short problem sets in the context of PER to create our own online course and creating an infrastructure to enable students to work through these materials together. During our discussion, we identified a series of existing tutorials in statistics (created by education researchers) as the backbone of the course. Links to these tutorials on the wiki and three problems per tutorial in the context of PER, which need to be developed, would be a foundation for online learning resources for our community. We did not identify ready solutions to several problems that still need to be addressed: developing the infrastructure for students to coordinate working through the materials, integrating the online material and the workshop, and getting support for community members developing these resources. The first step we identified is to gather existing tutorials and other resources in the online wiki and determine which to use in the online course.

The PER community has been a trailblazer for the broader Discipline Based Education Research community. For example, the creation and use of concept inventories as quantitative measures of student change began in PER and has driven significant transformation across the STEM disciplines. We have the responsibility as scientists to ensure that the methods used to collect, analyze, and share quantitative data in PER are high-quality and facilitate accurate interpretations of the data. The resources proposed by the working group at FFPER can support each other. The wiki can support developing the workshops and online materials, which can both work together to provide a far more effective learning experience than a short workshop or a handful of online tutorials could on their own. These resources build upon the high quality work our community produces and are a step forward in improving the breadth and depth of our community’s statistics knowledge and practices. Ultimately, it will be up to the community to develop, share, and implement these learning resources, and to hold itself accountable for using high-quality statistical methods.

Jayson Nissen is a Postdoctoral Researcher in the Department of Science Education at California State University - Chico.

John B. Buncher is an Assistant Professor of Practice at North Dakota State University.

Paul Emigh is a Postdoctoral Researcher in the Department of Physics at Oregon State University.

Daryl McPadden is a Doctoral Candidate in the Department of Physics at Florida International University and a Research Associate in the Department of Physics at Michigan State University.

Caleb Speirs is a Doctoral Candidate in the Department of Physics and Astronomy at the University of Maine.

Ben Van Dusen is an Assistant Professor of Science Education at California State University - Chico and Director of the LASSO Platform.

(Endnotes)

1. Wasserstein, R. & Lazar, N. “The ASA's Statement on p-Values: Context, Process, and Purpose”, The American Statistician, 70 (2), 129-133, (2016).

2. Nuzzo, R., “Statistical Errors”, Nature, 506 (13), 150-152, (2014).

3. https://journals.aps.org/prper/edannounce/10.1103/PhysRevPhysEducRes.13.010002

Disclaimer – The articles and opinion pieces found in this issue of the APS Forum on Education Newsletter are not peer refereed and represent solely the views of the authors and not necessarily the views of the APS.