Reproducibility of artifacts is a cornerstone of most scientific publications. To improve the current state and strengthen ongoing community efforts towards reproducibility by design, we conducted a survey among the papers published at leading ACM computer networking conferences in 2017: CoNEXT, ICN, IMC, and SIGCOMM. The table below shows the responses of this survey. The "Easiness" column describes the difficulty to reproduce the results with given artifacts: "This self-assessment allowed the researchers judge their publications on a scale from 1 (“easy – an undergraduate can do that”) to 10 (“hard – only I can do that”)." The raw data is available here: DIANA website.
The objective of this effort is to assess the current state of artifact availability and reproducibility based on a survey. We hope that it will serve as a starting point for further discussions to encourage researchers to ease the reproduction of scientific work published within the SIGCOMM community. Furthermore, we hope this work will inspire program chairs of future conferences to emphasize reproducibility within the ACM SIGCOMM community as well as will strengthen awareness of researchers.
An associated editorial describing this effort is available
A Survey on Artifacts from CoNEXT, ICN, IMC, and SIGCOMM Conferences in 2017 →
Matthias Flittner, Mohamed Naoufal Mahfoudi, Damien Saucez, Matthias Wählisch, Luigi Iannone, Vaibhav Bajpai, Alex Afanasyev
ACM SIGCOMM Computer Communication Review (CCR), January 2018.
This webpage is created by Thomas Paul.