After the submission phase, all eligible experimental design proposals enter a transparent, structured, and fully pre-specified evaluation process. The goal of this stage is to collectively select the experimental design that is most suitable for large-scale implementation across all participating labs.
The rating and selection process is one of the central elements of the crowd-science approach of ManyLabsDACH.
Before proposals are sent for evaluation:
All submitted designs undergo a plausibility and feasibility screening by the Project Team.
The screening checks:
Compliance with the design requirements
Ethical feasibility (IRB compatibility, no deception)
Clarity and internal consistency of the proposal
If necessary, the Project Team may request minor clarifications or revisions.
Only proposals that pass this screening stage proceed to the rating phase.
All proposals that pass the screening are distributed anonymously to all participating Research Teams (RTs).
Each RT evaluates all eligible proposals except their own.
Ratings are submitted at the RT level, not by individual RT members.
Each proposal is rated on a scale from 0 (“very low priority”) to 10 (“very high priority”), based on the following criteria:
Importance of the tested hypotheses
Does the proposal address research questions that are theoretically or empirically meaningful?
Quality of the experimental design
How well does the design test the stated hypotheses?
Are treatments, outcomes, and analyses clearly specified?
Novelty and contribution to the literature
Does the design offer new insights or advance existing research?
RTs are instructed to consider all three dimensions jointly when assigning their rating.
The experimental design with the highest average rating across RTs is selected for implementation.
There are no discretionary overrides: the selection strictly follows the pre-specified rating rule.
Once selected:
The proposing lab will be reimbursed for participant payments made at its own lab.
The proposing Research Team formally joins the Project Team.
The selected design becomes the single experimental protocol implemented identically across all participating labs.
The entire rating and selection procedure is pre-registered as part of PAP 1.
All participating labs commit ex ante to:
Accepting the outcome of the selection process
Implementing the selected design according to the standardized protocol
All contributing Research Teams remain co-authors on the final publication, regardless of whether their design is selected.
All participating laboratories implement the same experimental protocol, ensuring full comparability of results across sites. The protocol is designed to be simple, transparent, and strictly standardized, minimizing implementation heterogeneity while preserving internal validity.
The entire procedure is pre-registered in PAP 2 and applied identically in all labs.
To ensure harmonized data collection, the following elements are identical across all participating labs:
Experimental instructions
Fully standardized and centrally provided.
Software and implementation
The same experimental software is used in all labs (e.g. oTree).
Randomization procedures
Identical random assignment to experimental conditions within sessions.
Identical session size
The session size must be identical across all labs.
Incentives and payment structure
Uniform compensation scheme across labs, adjusted only where strictly required by local regulations.
Language
All experiments and surveys are conducted in German.
Devices
All the experiments must be conducted on computers. Other Devices like Smartphones or Tablet's are not possible.
No lab-level adaptations are permitted beyond those explicitly approved and documented in PAP 2. Moreover, we actively keep track of aspects that may vary across labs during data collection: Who is the experimenter? How many participants were invited/signed up/showed up? How does the lab look? Who else is present during the experiment? What questions were asked? How were they answered? …
After completing the experiment, participants complete a standardized follow-up survey (approximately 10 minutes).
The survey includes:
Incentivized measures of economic preferences
Additional non-incentivized survey items
Basic demographic information
The content of the follow-up survey is identical across all labs and finalized prior to data collection.