Reproducibility Checklist

This is a copy of the reproducibility checklist that the authors will be asked to answer during paper submission. See the Reproducibility Criteria section in the Call for Papers for more details.

For all reported experimental results:

  • A clear description of the mathematical setting, algorithm, and/or model
  • A link to a downloadable source code, with specification of all dependencies, including external libraries (recommended for camera ready, though welcome for initial submission)
  • A description of computing infrastructure used
  • The average runtime for each model or algorithm, or estimated energy cost
  • The number of parameters in each model
  • Corresponding validation performance for each reported test result
  • A clear definition of the specific evaluation measure or statistics used to report results.

For all results involving multiple experiments, such as hyperparameter search:

  • The exact number of training and evaluation runs
  • The bounds for each hyperparameter
  • The hyperparameter configurations for best-performing models
  • The method of choosing hyperparameter values (e.g. manual tuning, uniform sampling, etc.) and the criterion used to select among them (e.g. accuracy)
  • Summary statistics of the results (e.g. mean, variance, error bars, etc.)

For all datasets used:

  • Relevant statistics such as number of examples and label distributions
  • Details of train/validation/test splits
  • An explanation of any data that were excluded, and all pre-processing steps
  • For natural language data, the name of the language(s)
  • A link to a downloadable version of the dataset or simulation environment
  • For new data collected, a complete description of the data collection process, such as instructions to annotators and methods for quality control