Guidelines for Replication Studies in IREE

In the next paragraphs we clarify:

1. the aims of replication studies
2. kinds of replication
3. what to do with the authors of the original studies
4. what we expect when you consider submitting your replication study to IREE

1. What are the aims of replications?

Analysing random variables which are drawn from random samples means that outcomes will result with certain probabilities depending on the underlying distribution. Thus, empirical analyses produce unlikely results from time to time which do not withstand slight changes in the data or method.
Also, working with data is challenging and requires a lot of attention to details. Nevertheless, all effort may not protect against unintended errors when generating data, creating working data sets, writing programming codes or even transferring numbers into manuscripts.
Moreover, exact results depend to some extent on the chosen statistical software and its version.

These factors may compromise the internal and external validity of empirical results. However, in order to stand on the shoulders of previous findings and to provide evidence-based advice to economic policy, researchers need to distinguish between valid findings and random or erroneous results.

Replications are the scientific instrument to evaluate the robustness and generalizability of empirical findings.

2. What are replications?

In general, replications are the repetition of empirical analyses with the goal to identify robust and generalizable results.
Replications may vary in two dimensions from the original analysis: data and method.
Depending on the extent of the variations in these two dimensions we distinguish different kind of replications. Assertions about the original study drawn from replications are closely related to its kind.

	Same data	Different data (different time or different population)
Same model	Pure reproduction	Replication with focus on generalizability
Different model	Replication with focus on robustness	n.a.

Pure Reproduction: For this kind of replication the same data are analysed with the same empirical model as in the original study. Ideal starting points for pure reproductions are the original analyses data set and the original programming published with the original study or provided by its authors. If these resources are not available, the analyses data set and programming need to be reconstructed based on the information given in the original study. The goal of pure reproductions is to repeat the original study and to duplicate the exact results. Assertions drawn from such a replication are usually limited to the reproducibility of the original findings but allow no conclusions with respect to its robustness or generalizability.

Replications with focus on internal validity and robustness:

Using the same data but varying the estimation model as of the original study leads to the focus on robustness. The replication model may differ in (control) variables, the functional form assumed or estimation method. The goal of such replications is to verify whether the original results qualitatively hold against reasonable changes in the estimation. Robustness replications may also analyse different data as the original study as along as the investigated population (e.g. same industry, same region, or same age group but from a different data source as used in the original study) and the estimation model remain the same.

Replications with focus on external validity and generalizability: Generalizability requires that findings hold for a larger population than the specific individuals or items observed in the analysis data set. Replications which apply the same as the original estimation model to different populations may assert the applicability and the extent of generalizability of the original findings to other populations or units.

Reproductions and replications with focus on robustness or generalizability can all be applied to one original study as the goal and thus possible assertions of each kind of replication is different. As usual, in order to identify potential causal effects of the changes compared to the original analysis only one kind of change should be applied at a time. Thus, different kinds of replications should be applied to the original study one after another.

 

3. What to do with the authors of the original study?

The authors of the replicated study (i.e. the original study) should always be informed about any replication attempt. If a replication study is accepted for publication in IREE, the Editors will invite the original authors to comment the replication study. Such a reply will be published along with the replication study. Therefore, the replicator should make sure that the original authors got involved or at least informed before submitting to IREE.

• • In the case of a successful replication, the replicator should notify the original authors about the results.
  • If a replication attempt fails, the replicator should try to clarify the information which would be needed for a successful replication. With additional information provided by the original authors, an initially failed replication may turn into a successful replication. In such a case, the replicator should describe this certain replication process which involved the original author and still submit the replication to IREE. That information which turns a study from non-replicable to replicable is highly valuable for the understanding and usability of these findings.
  • The replicator should also document if the original authors are unable to provide necessary information.

 

4. What do we expect?

If you consider submitting a replication study to IREE your paper should include the following aspects:

• • What kind of replication is done in your study?
  • Explain the modifications between the original and your analysis (method or data) and to which end you chose these changes.
  • Discuss your results and sources of possible differences to the original findings.
  • Do your results support or do they challenge the original findings? Why and to which extent? What are the implications of the similarities or differences in the results? In your discussion, please bear the specific possible assertions drawn from each kind of replications in mind.
  • Document your contact with the original authors. Please allow the original authors to answer within a reasonable time, e.g. four weeks.
  • Your replication must be replicable, too. Make sure that the source and access of data, the construction of your analysis data set, and your programming is described in detail such that your replication can be replicated instantly by others.