One thing that was done in my alma matter, and which I try emulating in academic spaces I am in, is a paper swap session. Different people, all working towards the same deadline, exchange papers and provide feedback on how to improve the paper, where it’s making a certain argument clearer, bringing in additional literature, or how to make the paper more concise. Inevitably when organizing these sessions, someone always asks:
But Abraham! I am but a novice in the world of academia. I have never given feedback on a paper, how do I go about doing so?
This is a fair question. My first response is to point them to the various resources that already exist, such as Ken Hinckley’s guide on how to give excellent reviews, or the CHI guide to reviews. These are extremely useful and insightful guides which I myself have used in the past. But on rereading these guides, I realized there are two small issues. The first is that they are focused on how to review a paper rather than providing feedback on a paper: these two overlap, but are distinct (feedback meaning providing helpful advice to a colleague on how to improve a paper; review is judging a paper for its academic merit and whether it is worthy of being published or not). But more than that, these guides had a lot of high level feedback rather than low, on-the-ground advice. Advice such as have a positive attitude, give constructive feedback, and focus on the positives as well as the negatives is all good, but how does one know what feedback to give? What are the steps people should take to give feedback?
In this blog post, I try to write a more granular, detailed guide for how to give feedback on a paper. The primary audience are those who are early in their academic journeys, and those insecure on how to give feedback, but I hope everyone will benefit from this! Furthermore, this is written from the point of view of someone in the Human Computer Interaction field. other fields and disciplines may have other ways of providing feedback.
First, leave the impostor syndrome at the door
The first thing to get out of the way is that a lot of first-time feedbackers can feel insecure or unsure of the feedback they provide. They may feel they do not know enough, or are not experienced enough, to provide useful insights into a paper. Let’s get this out of the way. If you have done, are doing, or have thought of doing research, your opinion on a paper is valid. You are the paper’s intended audience, and so, any opinion of yours, positive or negative, is valid. Finally, even if your worst nightmares are true and your feedback is somehow worthless, you are giving feedback, not orders. It is up to the authors to decide whether to accept your feedback or not.
So get your confidence going, and get your feedbacker hat on, son, because it’s about to get serious!
How to give feedback
Step 1: Read and summarize the paper.
The first step is to read the paper from start to finish, without looking out for flaws or critiques. Just read it. Then, summarize the paper in 2-3 sentences.
This is useful because it helps you get the full context of the paper, as well as serving as a check on yourself. Did you understand the paper? Or have you not understood what the paper is about after reading it (which can be a point of feedback for the authors: ideally a paper should be clear enough so that a simple skim gives the reader a rough idea of what the paper is trying to do).
Moreover, if you then give this summary to the authors, it can be helfpul for them too. By summarizing what you think the paper is about, the authors can contextualize the feedback accordingly, and know what areas to tackle. For example, if they conducted an ethnography, but my summary says “The authors ran an experiment”, then clearly there is some mistranslation going on and the authors need to make their method clearer.
Step 2: Give feedback
The next step is to go through the paper and give feedback on it. Ok this is a bit too broad. Basically, you want to write down aspects of the paper you liked and aspects that you didn’t like. Feedback is in the eye of the beholder: anything you want to mention, you can (and should!), ranging from to even low level like the number of typos and such.
For those seeking a bit more guidance, the way I think about this is there are four main categories of feedback you can give: (1) writing; (2) prior work; (3) logic / argumentation; and (4) contribution. For each area of feedback, I give several questions you ask yourself and should write about.
Writing
This part concerns itself with the clarity of the writing. The questions you should ask yourself are:
- Is the writing clear?
- After reading through the paper, did you understand what was going on? Or are you confused?
- If the writing is confusing, what made it confusing? Did they use too many weird terms that you didn’t understand? Were the paragraphs too dense to read through?
- And finally, did you enjoy reading the paper? Did it flow smoothly, or did you find yourself needing to reread various sections and are still confused.
Write a few sentences on these prompts. For this (and other prompts) it is important to not only mention the bad things, but the good things too. This lets the authors know what they are doing well and should continute doing.
For guidance:
- A good paper is easy to follow, enjoyable to read, and leaves you knowing exactly what the authors did and why.
- A bad paper is one that is difficult to follow, confusing to read, and has you rereading the same paragraph 3 times and going “Huh? What are you saying?”
Prior Work
Prior work refers to how well the authors talk about work that was done before. Research is meant to build on top of each other, so a good paper explains what this other work is and how the current paper builds on top of it.
Write a few sentences on these prompts:
- Do you feel the authors provided enough background knowledge to (roughly) understand the problem they were addressing?
- If you are aware of other papers that have been done in the area, does the paper cite them?
For guidance:
- A good paper is one that gives the reader enough information to understand the basic of a problem and what current research on the topic says
- A bad paper is one that does not do that.
Logic
Next up is logic, or argumentation. All academic papers are basically full of logical arguments. The world has this problem, therefore it is important to address it. We wanted to answer this research question, so we used this method. We got this data, so therefore, we make these recommendations.
For the arguments the authors make, ask yourself:
- Does the paper’s arguments make sense?
- For example, does their method justify their findings? Are their conclusions supported by their data?
- One way to think about this is to ask, can you think of counterarguments to their conclusions? For example, can you think of alternative way to think about the data?
- But perhaps most importantly, do you believe their arguments? Or are you still not convinced?
For guidance:
- A good paper is one where everything makes sense, and you can easily draw a logical line between point A to point B to point C.
- A bad paper is one full of logical fallacies, random jumps, and conclusions that are not supported by the evidence or data.
Contribution
The final area of feedback is contribution. This might be the most complicated one to gauge (CHI even wrote a whole blogpost about the different types of HCI contributions, but the way I see it, contribution is showing that the paper is important and that it helps improve the world by existing, whether it is because it adds knowledge, insights, or somehow improves humanity.
After reading the paper, write a few sentences on these prompts:
- Did the paper explain why the problem they are studying is important? Do you agree with them that their problem is important?
- Does the paper clearly explain why their findings matter?
- Does the paper clearly state what its contribution is?
For guidance:
- A good paper is one where everything makes sense, and you can easily draw a logical line between point A to point B to point C.
- A bad paper is one full of logical fallacies, random jumps, and conclusions that are not supported by the evidence or data.
Other areas of feedback
If those areas aren’t enough, here are a few additional areas of feedback you can comment on.
- Use of Figures: Figures are useful for summarizing, at a glance, key concepts of the paper. Does the paper use figures? If yes, are they clear at a glance what it is about? Or are they confusing? If there are no figures, could a figure be used to summarize their main points?
- Organization: This goes hand in hand with writing. Is the paper properly organized? Do the sections flow well, or would you want to rearrange certain sections?
- Length: Is the paper too long? What are ways to cut it down? Is the writing too fluffy? Are there sections that can be cut? Generally, the shorter and more concise a paper is, the better (if you can say something in 100 words instead of 500, then write 100. The other 400 can be better spent elsewhere)
Advanced tips: It’s all about the context
Some advanced tips is to think about the contex tof a paper. By context I mean things beyond what is written in the paper: I am talking about who is writing the paper, where they are writing to, and so on so forth. Here are three examples of how to take context into account when giving feedback:
Context 1: What type feedback do the authors want?
Whenever someone asks me to provide feedback on one of their papers, I always counter-ask what type of feedback do you want me to give? The reason I say this is because not all papers ask or require the same type of feedback. In some cases, the authors already have finished product and they just want to cut down on space. In a different paper, the authors may be concerned about motivating their problem because they are writing to a hostile audience who they are not sure is convinced by the importance of their research questions. Once you know what type of feedback the authors want, it can help you drill down on the specific areas they are asking about, meaning you can provide more focused feedback (as wella s saving you some bandwidth by focusing on a few areas!).
Similarly, I also ask what is the preferred method for receiving feedback? Some people like comments in a pdf file. Others like comments in an Overleaf document or Google Docs. Some people prefer a written report. And finally, some people may just want a quick 3 minute conversation. Your goal as a feedbacker is to help these authors improve their paper, and therefore, by knowing what type of feedback they would like to have and in what format, it helps you help them improve their paper. Of course, you may have your own preferences as a feedbacker: if you despise the Overleaf interface and hate making comments there, you shouldn’t have to bend over backwards and inconvenience yourself just for their sake (you are the ones doing them a favor after all). But if you are indifferent to the format you can give, then something as simple as changing the medium through which the feedback is delviered can make a huge difference.
Finally, you can add the tone of feedback the authors want. Some people brutal, honest, and aggressive feedback (“Give it to me straight, doc”). Others prefer gentler more encouraging feedback. Neither method is inherently better than the other one, but changing the tone of your feedback can vastly impact how the feedback is received and whether the authors leave with a sense of motivation and ambition or whether they are demotivated, even if the piece of advice being transmitted is the same. So contact the authors and figure out what type of feedback they prefer.
Context 2: What can be done before the submission?
Another advanced tip when giving feedback is to ask yourself whether the authors are capable of implementing the feedback you give. Take the following two scenarios:
- Situation (A): the authors are writing a paper that is due in 2 months. There are six authors working on this paper, all of them senior academics and have a lot of expertise writing papers.
- Situation (B): the paper is due in 48 hours, and it is being handled by an overworked, underpaid, stressed out graduate student who has two other papers to submit.
Obviously, given the difference in both ability and the amount of time left for the deadline, there are changes that can be implemented in Situation (A) that cannot be implemented in Situation (B) (e.g., rerun an experiment, or redo the entire analysis). Therefore, if the goal is to help improve the papers in time for the submission, one might consider tailoring their feedback accordingly. It does not make sense to give people feedback that they cannot reasonably implement in time: if anything, it can be counterproductive, because (a) that is more feedback the authors have to sift through and (b) the authors are stressing out about flaws of their paper that they don’t have time to address. If your goal is to prevent the person from submitting the paper at all, then sure, include that feedback. But if you do want them to submit the paper, and the feedback isn’t mission critical, maybe focus on small level changes and leave the big level comments for a point where the authors are more relaxed and can address the feedback. Feedback is not neutral. While true that the ultimate responsibility lies with the author as to whether they accept your feedback or not,
Context 3: Contextualize how important your feedback is.
Another way to incorporate context is by providing the authors information as to how crucial or important you think your feedback is. Some feedback is very important: it makes a paper unreadable if not addressed. Other feedback is more of an improvement, where having it would be better, but not having it is fine too. By letting the authors know what feedback you are giving is a big deal versus which one is not, it lets them kno: plus, in case the authors do not have enough time to address all of your points before a submission, it helps them prioritize feedback accordingly.
I myself use a 1-2-3-4 system, as follows:
Feedback with a 1 is mission critical feedback. If I was reviewier, and I saw this flaw, I would reject the paper. For example, you did not properly anonymize participant quotes, and I can easily identify who said what.
Feedback with a 2 is important feedback. Doing this will fix the paper, but it isn’t mission critical. For example, the findings section is confusing to read, and it would benefit from a reorganizing for clarity.
Feedback with a 3 is my own opinion. It is my own preferred style of doing things, but, and the authors can easily ignore this. For example, when writing reports of an interview study, I like highlighting quotes in italics so they stand out in the paper.
Feedback with a 4 is additional notes that isn’t necessarily feedback or something to do with the paper, but rather, miscellaneous thoughts that appeared after reading the paper. For example, “This makes me think of a cool follow-up study involving squirrels and the quantum computing”.
Resources
I created a paper guide, or template, for providing feedback. It can be found here.
>> Home