The responses have been edited for clarity and brevity.
I consider four factors: whether I'm sufficiently knowledgeable about the topic to offer an intelligent assessment, how interesting I find the research topic, whether I’m free of any conflict of interest, and whether I have the time.
Then, right in the Introduction, you can often recognize whether the authors considered the full context of their topic.
After that, I check whether all the experiments and data make sense, paying particular attention to whether the authors carefully designed and performed the experiments and whether they analyzed and interpreted the results in a comprehensible way.
Do the hypotheses follow logically from previous work? To what extent does the Discussion place the findings in a wider context and achieve a balance between interpretation and useful speculation versus tedious waffling? (Then, throughout, if what I am reading is only partly comprehensible, I do not spend a lot of energy trying to make sense of it, but in my review I will relay the ambiguities to the author.) I should also have a good idea of the hypothesis and context within the first few pages, and it matters whether the hypothesis makes sense or is interesting. I do not focus so much on the statistics—a quality journal should have professional statistics review for any accepted manuscript—but I consider all the other logistics of study design where it’s easy to hide a fatal flaw.
Mostly I am concerned with credibility: Could this methodology have answered their question?(In my field, authors are under pressure to broadly sell their work, and it's my job as a reviewer to address the validity of such claims.) Third, I make sure that the design of the methods and analyses are appropriate. I also pay attention to the schemes and figures; if they are well designed and organized, then in most cases the entire paper has also been carefully thought out.First, I read a printed version to get an overall impression. When diving in deeper, first I try to assess whether all the important papers are cited in the references, as that also often correlates with the quality of the manuscript itself.The only other factor I pay attention to is the scientific integrity of the journal.I would not want to review for a journal that does not offer an unbiased review process.Unless it’s for a journal I know well, the first thing I do is check what format the journal prefers the review to be in.Some journals have structured review criteria; others just ask for general and specific comments. I almost never print out papers for review; I prefer to work with the electronic version.I usually consider first the relevance to my own expertise.I will turn down requests if the paper is too far removed from my own research areas, since I may not be able to provide an informed review.I'm more prone to agree to do a review if it involves a system or method in which I have a particular expertise.And I'm not going to take on a paper to review unless I have the time.