Well Quite!

Yesterday, I got a paper rejected from the Haskell Symposium 2008, and I'm going to talk about the reviews I got. I realise this is highly unprofessional, but at this stage I really don't care, especially as in my opinion, the behaviour of some of the reviewers is also highly unprofessional. Also, this is the "calm" version of this rant: fortunately a friend came over yesterday and we went out for a drink or two which stopped me writing this yesterday. Finally, I'm not trying to take anything away from the authors who did get papers accepted, or the work they have done. My complaints are very specifically about the process that takes place though, perhaps unsurprisingly, I don't really have many ideas about how it can be improved.

To every system, every activity, there is a game to play, and to succeed, you must learn to play the game. The British Education system is an excellent example of this: to succeed you don't have to be bright, you don't have to put in any effort, you just have to learn to regurgitate facts as the exams will never actually require you to think. The same can be said for almost all exams I've taken at university. The problem that I increasingly see with the academic game of "getting papers published" is that the rules change far too frequently and actually hinder research and progress.

My understanding of how a conference or workshop or symposium are run is as follows:

1. A programme committee is formed of, maybe a dozen people. These should collectively have an interest spanning pretty much everything that the conference aims to cover.
2. Papers are submitted.
3. Each member of the committee selects which papers they wish to review. They must declare conflicts of interest and each paper typically gets about three reviewers. But, there will often be a load of papers left over that too few committee members are interested in. For whatever reason, the authors thought it was appropriate to submit the paper to this particular conference, but there would seem to be not enough on the committee who are interested in reviewing these papers. It's basically up to the programme chair to get these papers sufficiently reviewed by someone.
4. The reviewers may well do the reviews. They may farm the reviews out to PhD students or research assistants if they're think that the student or assistant actually has a better current knowledge of the area about which the paper is written than they do.
5. Reviewers then submit their reviews. These normally have a "score" indicating what the reviewer thought of the paper, and a "confidence" which indicates how seriously the reviewer thinks their review should be taken. Here, the papers left on the pile which no one wanted to review could well end up with three reviews, all done by people who readily accept they are not confident of their own knowledge of this particular area.
6. Then the programme committee meet. Papers which were slammed by everyone typically don't get discussed and get rejected. Papers which were raved about by everyone don't get discussed, and get accepted. The papers remaining get discussed and somehow some get accepted, up to a limit of how many papers can be presented in the time allowed at the conference.
7. The reviews get sent back to the authors. These reviews are anonymous and sometimes the "scores" and "confidences" get sent back and sometimes they don't.

There are variations on this, and I've never actually been involved in this process other than being an author. However, I'm lead to believe that this is basically what happens.

So, what does this process mean for authors? Well, there are several things to consider. Firstly, no one actually wants to do reviews. They take time away from your own research and people are just doing them to remain, or become, well known. It's also good publicity for the conference if the programme committee is full of distinguished and well known academics. Programme chairs often choose to organise conferences because they want a promotion and have to demonstrate that they have run conferences and are an international figure. Sure, I'm being cynical and probably unfair in many cases. But this is undeniably sometimes the case. Secondly, reviews can get left to the last minute. If a reviewer has to produce reviews of 10 papers by tomorrow then these reviews will be done quickly. They will often be farmed out to PhD students and RAs who will also be under pressure to get the review done quickly. Furthermore, if your paper is in the pile that no one really wanted to review then the review will certainly be a rush job.

So the ideal situation is to have your paper reviewed by people who are really interested in your area and who are organised and will put the time in to do a decent job. Fortunately for the reviewers, even if they do a really shoddy job, they won't be found out because the reviews go back to the authors anonymously.

Now let's say that your paper is on the pile which no one really wants to review. Your best bet here is to produce a trivial paper - something with almost no content. The reason is that if you start producing anything even vaguely complicated, the resulting rushed and uninterested reviewer will hate you because they will realise that to do a decent job of reviewing the paper will take time and effort that they really can't be arsed to expend. Whereas, if you produce something trivial then they will understand it easily, it won't produce any surprises for them, hopefully they won't have to read anything you cited and they'll be able to top and tail the paper quickly, with a minimum of effort and fuss and still have a decent knowledge of what you were trying to say.

Thus to win this game, plan in advance, for as far in the future as you possibly can. Work out which conferences you are going to submit to and look at their programme committees. Then look at what the members of the programme committees are researching and completely abandon any hope that you can do research on what you want to do, especially if it doesn't correlate perfectly with the interests of the programme committee. Oh and make sure that your topic doesn't suddenly go out of fashion, otherwise you will never get any subsequent publications. It helps if your supervisor or co-authors are widely respected members of whichever community you're working in.

Now I got three reviews back, anonymously, without scores. They paint a very telling story. The first review is excellent. I have never received such a review nor have I ever expected to. The guy has obviously thoroughly read the paper. He has also been to my website and read the tutorials. He has downloaded the software, installed it and used it. By my reckoning, he must have spent at least six hours doing this. It is incredible - as I say, I would never expect anyone to put that much effort in. What I surmise from this is that he wanted to review the paper: he clearly is interested in it and picked it and put a huge amount of effort in. Sadly for me, he then admits that he's sceptical about the value of Session Types in the first place (so it's difficult to see this as being an objective unbiased review, though given the effort he's gone to, he's clearly tried very hard) and then he repeats issues which I have discussed myself in the evaluation, about how error messages are huge and very difficult to make sense of as criticism of the work. And indeed it is - it significantly makes it harder to use the library given this issue. However, he basically does end up admitting that, given the way I have written the library, this is unavoidable and actually demands programmatic control of the errors coming back from GHC, rather than a series of modifications to the library. He suggests that this work could be published as motivation for making such changes to GHC. So he doesn't really like the work, he's sceptical of the value Session Types add and he quite justifiably questions the usability of the library (though oddly not commenting on any of the proposals I make in the paper for solving the error message problem). Overall though, I simply can't have anything but awe for the amount of work and time he has invested in the review. I have absolutely no issue with being endlessly rejected if reviewers go to this much effort. It is simply unfortunate that his default position on Session Types is one of scepticism.

The second review, by comparison is a complete joke. I don't believe the reviewer read past page 4, with the possible exception of additionally reading the conclusion. I question whether this guy knows anything about Haskell at all or left himself more than 5 minutes in which to review the paper. Typically when you do a review, you start by giving a description of what the paper is about, in order to demonstrate that you read the paper. In this review, even this description is wrong:

The paper presents a library that implements session types as a Haskell DSL.

Wrong. A DSL embedded within Haskell is used. And it is used to define a session type. The rest of the library which allows you to implement a given session type has nothing further to do with the DSL. The second paragraph is equally brilliant:

Unfortunately, I find the paper practically impossible to follow. For the most part, this is because it uses a large number of combinators which are never really introduced and explained. In particular, although the paper is about types it doesn't provide any type signatures. This makes it impossible to understand if and how the goal of type-safe communication is achieved. Here are just some of the questions that the paper doesn't seem to answer...

So in the first sentence he admits he certainly hasn't understood and probably hasn't read the paper. He is very correct in the third sentence: I don't have any type signatures in the paper. This is because they are huge, absolutely enormous. If I put the type signatures in then I would have about 3 of the 12 pages permitted left to describe the work. Thus sentence 4 is ridiculous: there is simply no way that adding type signatures would make it easier to understand. Go look up the type signatures for doing subtraction on numbers in the type system, it's not trivial; and that's a simple case for my library: the type signatures in my library are typically over ten lines long. And the use of seem in sentence 5 further reinforces the notion that he hasn't read the paper. He then goes on and raises six questions, all of which come from work discussed before page 4, and all of which are addressed within the paper. Some of them are only fleetingly covered (due to space constraints), but they are all covered. I particularly love his first question: Is a channel associated with a session type? Looking at page 2 of the paper, I write:

Having specified a session type, that session type can then be used to parameterise the type of a communication channel, which restricts operations involving that communication channel to those specified by the session type.

So he gave up before page 2. Good job. At no point does he refer to anything that comes after page 4.

Let's take the first two sentences of his second paragraph and contrast them with the first review. The first reviewer raised no such objections. Not only did the first reviewer read the paper and my tutorial and raise no such objection, but they also used the software to the extent that they demonstrated knowledge of it in their review. This sits a little oddly with the notion that the paper is practically impossible to follow. This is amateur hour at its very peak. Maybe the reviewers should actually compare notes before submitting their reviews because frankly, this is pathetic. The first reviewer has effectively demolished every criticism the second reviewer makes, and demonstrates that the second reviewer didn't care and didn't put any effort into the review. It is utterly obvious that the second reviewer did not pick to review this paper, but rather the paper was left on the pile with not enough volunteered reviewers and so was pushed to the second reviewer. But it's okay, because the reviews are anonymous. So even though I know exactly what happened, and surely the reviewers likewise realise that I will be able to work it out, they have no problem with making these reviews because I can't identify anyone. I would be perfectly happy receiving one review from someone who knows what they're talking about, like the first review, than this shit. And the reason should be obvious: when it comes to the programme committee meeting, the second review is wrong from top to bottom, I can't defend myself against it, but it will completely destroy my chances of being published.

The third review is okay: it's more along the lines of what I've come to expect. It's clearly not the super-human effort that the first review is, but I believe they've read the whole paper and understood it. Sadly, the specific issues they raise with regard to the operation of the library are on the whole addressed in the paper. For example, they query the rather difficult to read notation for session types. Yeah I agree, but I have explained that I can't use do-notation in the paper. They also raise the issue of a lack of compelling example. Now wait a minute! The paper uses an n-queens checker as a running example. This is deliberately trivial in order to keep the code fragments short. It's pretty unlikely that the reviewer would want to be presented with multiple pages of code. I repeatedly state in the paper sentiments such as:

As mentioned in section 2, the bulk of the code of this example is concerned with the communication and very little is actually doing the work of checking for collisions: the detectCollision function is trivial. However, the communication patterns involved here are entirely non-trivial and demonstrate how our Session Type library can be used to tackle such patterns.

In fact, once again, this charge is voided by the first review:

The running example is an implementation of the n-queens check, with one process per queen, which communicate to check that no queen threatens another. It's a trivial example, but the communication pattern is dynamic and fairly complex.

Again, from the paper:

This example problem demonstrates how our library can be used for a much broader class of problems where work is not only farmed out to a number of workers (the queens, in this case) but how the workers must also communicate between themselves in order to complete their work before sending results back to the root process. Note that our example works for any number of workers: the number of queens is not known statically and so this shows how session types can work in the most general case.

What do you want? You want me to enumerate every possible situation where such a pattern is of use? Not to mention all the other communication patterns that can be covered by this library. And finally, in the third review, we get to the nub of the matter:

Much of the code is too intricate to easily read and understand.

Ahh, the displeasure of the reviewer, again, I believe, having the paper foisted upon them and wanting to do a quick job of it. But don't get me wrong, the third review is by no means in the same league of incompetence as defined by the second review. The third reviewer is wrong though: I don't believe there is a code example in the paper which is not explained in English.

All three reviews say that the paper is dense and at least in places hard to follow and I agree with that entirely. But the library is very featureful and is a complex beast, and I only have 12 sides in which to describe it. I give as high a level description of it (in particular, its implementation) as I possibly can without making the whole thing superficial. It's very tricky to write a paper about something this complex and bleeding edge (I really haven't come across anything else that does as much within the type system as I do) and keep it readable. So basically, the conclusion is that this work is simply unpublishable: if I try and give anything other than a totally superficial explanation of it then it becomes too dense and complex, and if I make it lighter and more readable then surely the reviews would come back demanding a more thorough and detailed covering of the work. How can I ever win?

What makes me so angry is that the months of work that went into writing the library, and the weeks of work that went into writing and rewriting the paper can be so easily dismissed by reviewers who really can't be bothered or are just uninterested in the work and rely on the cloak of anonymity or, perhaps more frustratingly, by the fact that the conference which is most suitable for the work and should be the ideal location for this work has a programme committee which is simply the wrong audience for this work.