At ecoCompute Science every submission includes a tangible artifact that reviewers run, use or inspect. The rest of the process follows from that.
The review process
1. Submission
Two pages and a tangible artifact that reviewers run, use or inspect. See the Call for Papers and Artifacts.
2. Kick-the-tires phase
The opening days of the review period are reserved for setup. Reviewers attempt to get the artifact running and report any issue that prevents them from starting. Authors are given a short window in which to correct it.
Artifacts should not fail on setup problems. A missing dependency is not a scientific judgement and should not be treated as one.
3. Artifact evaluation
A dedicated artifact evaluation committee runs, uses or inspects the artifact of every submission. The committee assesses whether the claim, as formulated by the authors, holds when somebody else works with the artifact:
- Does the guide work from beginning to end on a machine other than the authors'?
- Does the artifact support the points the write-up makes, in the way the authors argue it does?
- Does every number in the write-up have a source, and do the values attributed to the artifact follow from it?
- Does the property stated to be stable, whether an ordering, a direction or an effect size, hold within the stated tolerance?
- Is the reported variance consistent with what the reviewer observes?
- Where data was anonymised, does it preserve the properties on which the claim depends?
4. Publication of the outcome
The review outcome and the artifact status are published together with the paper. Readers can see whether an artifact was reproduced, partially reproduced or not reproduced, and on what grounds.
A paper whose artifact reproduced only in part is not for that reason a failure. A paper that conceals which parts did not reproduce is.
5. The event
Accepted two-page submissions are discussed at the online event: three sessions per submission, with limited group size, the author present in every session, and all participants having read the submissions one week in advance. Session notes are produced with LLM assistance and returned to the author.
Publication in two stages
Stage one: the write-up and the artifact
Reviewed, accepted and presented at the event. This is the version the community discusses.
Stage two: the full paper
Written after the event, using the session notes as input, and shepherded by a committee member.
In addition to the usual responsibilities, the shepherd has one specific task: confirming that the claims do not exceed the evidence that was reviewed. A full paper that broadens a result from a particular hardware configuration and workload to the general case does not pass shepherding.
Stage two is published in ECEASST, the Electronic Communications of the EASST. It is a diamond open access journal hosted by Berlin Universities Publishing, properly indexed, with no fee for authors or readers. ECEASST provides the submission and reviewing platform for the full papers through to publication. Anna-Lena Lamprecht is responsible for the publication track.
Writing quality
A sound idea and a working artifact do not by themselves produce a readable paper, and a language model does not resolve this. Shepherding at stage two addresses structure and clarity as well as the discipline of the claims. Authors for whom English is not a first language are invited to indicate this and will be paired with a shepherd who can assist rather than penalise them.
At stage one the write-up is deliberately short and bullet points are acceptable, so writing proficiency presents a considerably lower barrier than at a typical academic conference. This is intentional.
Committees
Two committees are being assembled:
- A programme committee, responsible for scope, acceptance and shepherding.
- An artifact evaluation committee, responsible for running, using or inspecting the artifacts.
Prospective reviewers are invited to contact the organisers, stating which kinds of artifacts they are able to run, use or inspect and what hardware they have access to. Reviewers are named in the published outcome unless they request otherwise.
Open questions on review
The following questions are genuinely undecided, and input is welcome before they are settled.
How can artifact review be kept affordable? Artifact review is considerably more resource-intensive than reading a PDF. The options under consideration are a cap on the number of submissions, rolling review, a first reviewer who produces a working recipe that the others then follow, and AI assistance for the first reproduction pass. Where an artifact is genuinely reproducible by anyone who has read the paper and holds the relevant domain knowledge, that first pass is a reasonable candidate for automation. The judgement remains with people.
Single-blind or fully open review? Artifacts are difficult to anonymise reliably. Repository names, author machines, internal paths and licence headers all disclose identity. A decision is required between accepting single-blind review and adopting a fully open process.
Confidential industry data. Data that can be anonymised without losing the properties on which a claim depends is accepted (see Artifacts). The open question concerns the remainder. Where a reviewer cannot inspect the data in any form, reproducibility in the conventional sense is unattainable, and synthetic sample data is frequently not representative enough to be of value. The current answer is that such work submits another tangible artifact, such as the design of the system or the method applied to a public configuration, and that its claim is limited to what that artifact supports.
Non-attendance. Registration free of charge produces high rates of non-attendance, which is untenable for discussion groups of limited size. A nominal fee, a short written application, and an invitation model following the example of Dagstuhl, under which participants are reluctant to disappoint the person who invited them, are all under consideration.