Skip to content

Background

Open science and robust reproducibility practices are becoming increasingly adopted within numerous scientific disciplines. Within subfields of educational technology, however, the adoption and review of these practices are sparsely implemented, typically due to a lack of time or incentive to do so (Armeni et al.1, 2021; Nosek, 20228) with some notable exceptions (Cook et al., 20164; García-Holgado et al., 20215; Makel et al., 20197). Authors have numerous concerns and minimal experience in what can be made publicly available, such as datasets and analysis code (Haim et al., 20236). As such, there is a need for accessible resources, providing an understanding of open science practices, how they can be used, and how to mitigate potential issues that may arise at a later date.

At its core, Open Science seeks to make scientific research, data, and dissemination accessible to all, breaking down the barriers of closed-access publications. It is built on the principles of transparency, collaboration, and shared knowledge. The goals of Open Science are to advance the pace of discovery but also foster a more inclusive, equitable, and accountable scientific community. As with many things, translation from ideals and principles into real-world implementation comes with considerable challenges. For example, open-access publication typically comes with a higher cost for the researcher (in turn damaging goals of equity and accessibility). Similarly, in education research, data sharing often poses challenges. Data are typically collected in partnership with educators, administrators, and students, who authorize the collection of data for a specific study/set of research questions, and often actively prohibit the distribution of data to third parties. Data can be deidentified, but given how intrinsically personal educational data can be, this task can be labor-intensive. Worse, some of the easier forms of deidentification (such as removing all forum post data prior to sharing) lead to data no longer being usable for a wide range of research and development goals2.

Sharing data on a by-request basis and carefully crafting data agreements has long been a potential solution, but it is often ineffective. For example, (Wicherts, Borsboom, Kats, & Molenaar, 200610) contacted owners of 249 datasets, only receiving a response from 25.7%. Within education technology, (Haim et al., 20236) contacted the authors of 594 papers, only receiving a response from 37, or 6.2%, of which only 19 responded that their dataset is public or could be requested. Some of the cited reasons were a lack of rights necessary to release the dataset, personally identifiable information was present, or the dataset itself was part of an ongoing study. The task of sharing data requires significant time investment and can be stalled by changes in email addresses or institutions.

Open Education Science (van der Zee & Reich, 20189), a subfield of Open Science, seeks to address problems of transparency and access, specifically in education research, addressing issues of publication bias, lack of access to original published research, and the failure to replicate. The practices proposed by Open Education Science fall into four categories, each related to a phase of educational research: 1) open design, 2) open data, 3) open analysis, and 4) open publication.Of most relevance to the current tutorial are Open Data and Open Analysis. Open Data is about ensuring research data and materials are freely available on public platforms, aiding in replication, assessment, and close examination. However, there can be challenges, especially with educational data. There might be initial agreements that prevent the sharing of data or issues related to personal identifiable information (PII) which restrict what can be made public. Open Analysis emphasizes that analytical methods should be reproducible. This is commonly achieved by sharing the code used for analyses on platforms like GitHub or preregistration websites. But there is a catch; the code is often of limited value without the associated data. Simply put, without Open Data, achieving Open Analysis can be tough. Moreover, there are challenges like "code rot" and "dependency hell" (as highlighted by Boettiger, 20153), where changing libraries can render older code unrunnable.

  1. Armeni, K., Brinkman, L., Carlsson, R., Eerland, A., Fijten, R., Fondberg, R., Heininga, V. E, Heunis, S., Koh, W. Q., Masselink, M., Moran, N., Ó Baoill, A., Sarafoglou, A., Schettino, A., Schwamm, H., Sjoerds, Z., Teperek, M., van den Akker, O. R, van’t Veer, A., & Zurita-Milla, R. (2021). Towards wide-scale adoption of open science practices: The role of open science communities. Science and Public Policy, 48(5), 605-611. https://doi.org/10.1093/scipol/scab039 ↩

  2. Baker, M. 1,500 scientists lift the lid on reproducibility. Nature 533, 452–454 (2016). https://doi.org/10.1038/533452a ↩

  3. Boettiger, C. (2015). An introduction to Docker for reproducible research. ACM SIGOPS Operating Systems Review, 49(1), 71-79. https://doi.org/10.1145/2723872.2723882 ↩

  4. Cook, B. G., Lloyd, J. W., Mellor, D., Nosek, B. A., & Therrien, W. J. (2018). Promoting open science to increase the trustworthiness of evidence in special education. Exceptional Children, 85(1), 104-118. https://doi.org/10.1177/0741932516637198 ↩

  5. García-Holgado, A., García-Peñalvo, F. J., de la Higuera, C., Teixeira, A., Ehlers, U. D., Bruton, J., Nascimbeni, F., Zea, N. P., & Burgos, D. (2020). Promoting Open Education Through Gamification in Higher Education: the OpenGame project. In Eighth International Conference on Technological Ecosystems for Enhancing Multiculturality. 399-404. https://doi.org/10.1145/3434780.3436688 ↩

  6. Haim, A., Baxter, C., Gyurcsan, R., Shaw, S. T., & Heffernan, N. T. (2023, July). How to Open Science: Analyzing the Open Science Statement Compliance of the Learning@ Scale Conference. In Proceedings of the Tenth ACM Conference on Learning@ Scale (pp. 174-182). https://doi.org/10.1145/3573051.3596166 ↩↩

  7. Makel, M. C., Smith, K. N., McBee, M. T., Peters, S. J., & Miller, E. M. (2019). A path to greater credibility: Large-scale collaborative education research. AERA Open, 5(4). https://doi.org/10.1177/2332858419891963. ↩

  8. Nosek, B. (2022). Making the Most of the Unconference. Presented at the Unconference on Open Scholarship Practices in Education Research. Available at https://osf.io/9k6pd/ ↩

  9. van der Zee, T., & Reich, J. (2018). Open Education Science. AERA Open, 4(3). https://doi.org/10.1177/2332858418787466 ↩

  10. Wicherts, J. M., Borsboom, D., Kats, J., & Molenaar, D. (2006). The poor availability of psychological research data for reanalysis. American Psychologist, 61(7), 726–728. https://doi.org/10.1037/0003-066X.61.7.726 ↩