Read on Twitter
The adjusted deadline / dual submission problem between @icmlconf and @NeurIPSConf is a serious issue in the making that could severely detriment not only the authors that are at the heart of both conferences but both of the conferences themselves. 1/?
The unfortunate truth is that the ML conference deadline cycle is brutal: good papers not being submitted can mean they never get published due to the fast-moving nature of the field and unhealthy reviewing culture that quickly dismisses papers as behind the curve. 2/?
This can have serious knock-on consequences for junior researchers and I think the decisions that organizers are making will have a noticeable detrimental impact on some people& #39;s lives in ways that could have been avoided. 3/?
Though I am sure these decisions have been made in good faith, I implore the organisers to reconsider and will provide some suggestions of things that could be done below. The two conferences are intertwined and this should not be an issue for one alone. 4/?
The current setup is making authors gamble: do I withdraw and resubmit, or hold out and hope the paper gets in. Putting this responsibility on authors is unfair and means the burden is largely bared by PhD students, who are also the most vulnerable to this issue. 5/?
It is not helpful for the conferences either. Perhaps counterintuitively, it is mostly strong papers that are going to be victim, which will be a loss to both conferences. 6/?
The “NeurIPS experiment” showed us that the review process is extremely noisy and very few papers are so good that they are nearly guaranteed to be accepted when submitted. No conference should discourage resubmissions. 7/?
As such, many papers on the borderline for ICML will have a good shot of being accepted to @NeurIPSConf, while discouraging dual sub will not stop the many poor quality resubmissions that have bad reviews and can be safely withdrawn from ICML. 8/?
This not only a detriment to the @NeurIPSConf submissions, but those for @icmlconf as well. Some papers that would actually get in may be prematurely withdrawn as authors decide their resubmission chances are actually better. 9/?
My group has at least one paper where we think it has a reasonably good chance of being accepted given the review scores (say 50%) but we are thinking of withdrawing because it is an extremely strong paper such that its resubmission chance is estimated to be better (say 75%). 10/
We also have quite a few that have a reasonable chance (20%-30%) but where the strategic choice is likely to be to withdraw. As such, both @icmlconf and @NeurIPSconf will lose strong papers where the authors make the wrong gamble. 11/?
Moreover, it is unlikely to have the desired effect of having lots of withdrawn papers from @NeurIPSconf as many authors will effectively dual submit with changed titles etc in the hope that it goes unnoticed, particularly for papers whose @icmlconf chances are already good. 12/?
So given these issues, what can we do? The first thing would be for the conferences to endeavor to get the deadlines as close together as possible. In particular, the @NeurIPSconf deadline is earlier than it often is so there should be some scope to push it back slightly. 13/?
This would also help with people who’s plans have been disrupted by COVID-19 and I think the @NeurIPSconf reviewing community would appreciate slightly condensing some aspects might be necessary, such as timelines given to reviewers, discussions, and area chairs. 14/?
Secondly, I think @icmlconf should look to release updated review scores from the discussion period and ideally also meta-reviewer recommendations before the @neurips deadlines, doing this for as many papers as possible even if it cannot be done for all. 15/?
In particular, post-discussion updated review scores should be available in time and could be released without causing much harm while doing a lot of good. 16/?
Thirdly, I think @NeurIPSconf should reconsider providing some flexibility on dual submissions up to the deadline for @icmlconf decisions. Reviewing is not due to start until June 12th and so this is a complication of assignment, not the reviews themselves. 17/?
The number of papers submitted that are later withdrawn due to being accepted at @icmlconf will only make up a small proportion of total, given how low accepted rates are anyway and not all will be dual submitted anyway. 18/?
Given COVID-19 is likely to impact on submission numbers anyway, I would postulate that the number of submissions received is likely to be very similar to that originally anticipated before the delay, such that withdrawals do not end up being too much of an issue. 19/?
At worst, it will lead to slightly uneven reviewer / area chair / senior area chair assignments, which could probably still be corrected for anyway after any “withdrawal” period. 20/?
One would also hope that not many papers that are accepted to @icmlconf become candidates for desk-rejects at @NeurIPSconf, such that this should again be manageable. 21/?
In summary, I feel that this is a problem that can be managed with only limited conference on @NeurIPSconf, but will cause serious hardship for a significant number of authors and worsen both conferences if not 22/22
