For details of my books, click here.

Some of this document is reasonably polished, but other parts are rough notes only.

I have also put up a webpage on "How to study statistics", aimed particularly at those taking their first statistics course: click here.

My name is Paul Hutchinson, and I work in the Department of Psychology, Macquarie University. However, this document is not really specific to psychology --- in previous existences I've worked in Statistics and Civil Engineering departments, and I think most of what is said below is quite widely true. Anyway, here is my advice on

The first thing to say is that if it has to be done, it's you who's got to do it.

• You. There's no-one else to do the project for you. Very likely, there's no-one who will give you any help. You're the one who cares, the drive has got to come from within you. You're the one who knows how to answer the question, no-one else knows as much.

  The above is not the whole truth --- as your project evolves, you'll find yourself receiving technical help, wisdom, and emotional support from a great many people --- but you need to be aware that self-reliance is vital.

  There may be other types of research, e.g., in which a neophyte is fitted into a slot in the God Professor's program. I don't have experience of them.
• Do. It is important that you start the main activities of the research --- experimentation, or survey, or simulation, or data analysis --- sooner rather than later. A common mistake is to think and plan and read, and never get round to the main game. When you start, you may find that the real difficulties and issues are quite different from what you thought in the abstract.

Research has several components, for example: thinking of the question, answering the question, communicating the answer. It may be worth mentioning that not all of these are essential.

• Even without much of a question, one can collect some data, summarise it, and report the results. Certainly if you're the first to think of this kind of data, that's all you need to do. Certainly if the data keeps changing from year to year (e.g., accidents, and many other society-level phenomena), this type of study is always useful to a greater or lesser extent.
• Even without an answer, the question is sometimes such a good one that it is worth telling others about. (It might be that you don't have time to answer it, or are uninterested, or not competent in the necessary techniques.)
• Sometimes you don't want to communicate your question and answer. They might have turned out to be less interesting than you thought when you started. Or you might want to wait until you've done something additional.

• The fact that you're doing research is strong evidence that you have the skills necessary to answer the question. Not all of them, perhaps, but enough to work out a way of attacking the research and to serve as a foundation for additional skills to be learnt along the way. Incidentally, as a rule of thumb, if there is a course on something, people think of it as a low-level skill --- it is not necessarily below your dignity as a researcher, but you don't get any merit badges for mastering it. (I am a little concerned at the increasing amount of structured studies that universities are prescribing for their first-year research students. To my mind, the overwhelming priority at that stage is to do a doable piece of research, not attend lectures on bibliographic tools and the institution's policy on intellectual property.) As to non-technical aspects of answering the question, there is some advice below.
• There are lots of textbooks and courses to aid you in your scientific writing. Essentially, though, it is all a matter of practice.

  One of the reasons I recommend psychology to students is that you can't escape either the use of numbers or the use of words. (Economics is another good subject from this point of view, but it doesn't have much of an experimental component.)

  As soon as you've got a message to communicate, you should write a report on it, not wait until the whole project is finished. That probably means writing 1000 words per week, every week. Just occasionally, supervisors forget to tell new writers the obvious --- the easiest sections to write are the descriptions of the results, and of what you did (because these sections are highly constrained by the facts), and you should write these first; what the results mean, and why you did what you did, are altogether more problematic, so the Discussion and Introduction sections should be written later. If you're desperate, imagine you're speaking to a friend about your work, and write down what you would say. (Then, as always, revise and edit, revise and edit, revise and edit.)

Some research has to be done to a tight deadline --- for example, the final-year projects carried out by undergraduates. The big difference this makes is that you don't have time (which you do in most situations) to get things wrong and then do them again. The consequence is that there's a good deal of luck involved at this stage, so I think that if your final-year project falls flat on its face, it doesn't mean you won't be good at research in a more normal environment.

It is only natural for a student to ask "What will get me good marks?" In assessing theses, some universities have marking schemes --- u marks are allocated for the project design, v marks are allocated for the conduct of the experiment or survey, w marks are allocated for the general writing of the thesis, x marks are allocated for the review of previous literature, y marks are allocated for the presentation of the results, z marks are allocated for the interpretation of the findings, and so on. Other universities have no such marking scheme. (Even where one exists, it is very often impracticable to use it strictly.) My advice in such a situation may be different from that of others. But, for what it's worth, here it is. My impression is that students typically spend too much time reviewing past literature, and not enough time thinking about it and criticising it. The student's ideas need to be firmly grounded in established wisdom, not plucked out of cloud cuckoo land. But the thing that will most impress the marker of a thesis is a focus on the central ideas, a willingness to criticise imprecise thinking by previous authors, the dissection of what is essential from what is less relevant, and the demonstration of how the student's ideas sharpen what has previously been blunt. In other words, intellectual oomph.

An important issue is that of what to study. If you have ideas, there's no problem. But, is there anything you can do to prompt ideas to come to you? Or, in the absence of ideas, how can you do something worthwhile? I don't suppose there's a complete answer to these questions, but the following comments are intended to be helpful.

To get ideas, expose yourself to them.

• Talk to someone about your research. Your mum or dad, for example. Or (though I've never tried this) the top stream of 13-year-olds at your local selective high school. Brainstorming with your colleagues is better than nothing, but you're not really forced to decode your talk sufficiently. Anyway, there are other uses for your colleagues, such as being rude to you.

  It is tremendously valuable if there's someone in your circle who can bring off the trick of being rude without causing offence. Suppose you are a cognitive psychologist, and in a seminar you claim that "activation flows through the levels" of your model. You benefit if someone from another tradition confronts you with the claim that "activation", "flows", and "levels" are all meaningless. It is generally up to the young to do this.

  Informal seminars, where you listen to others, and talk yourself, are an important part of this.
• Read widely. (But remember what I said earlier: reading is not what research is about. Indeed, I would say there is a type of personality for whom too much reading is a major danger to their ever doing anything worthwhile.)
  • How to read: read the title, skim the abstract, look at the pictures and maybe the tables, and if there's anything interesting, then consult the text, looking for that specific point. (No-one starts at the beginning of a paper and works their way through to the end.)
  • What to read: many of the "prestigious" journals are best avoided, as they have got into the habit of attending to a lot of details that everyone knows are unimportant, thus diluting the real message. The papers are polished and bland, and the reader is sucked into taking them on their own terms. You need something rougher, that you can get to grips with.
    • Comments and letters in any journal. Whatever the point is, it is made quickly. Furthermore, an area of controversy may be highlighted for you.
    • Journals from outside your main discipline that are relevant to your topic. For example, if your own discipline is psychology, see what the management, sociology, medicine, and engineering journals are saying about your topic. They may very well take sufficiently different an attitude to provoke thought. And interdisciplinary journals, e.g., on the borders of music and psychology, or law and statistics.
    • For the same reason, journals from countries that Americans have never heard of (such as Germany, India, and Japan).
    • Low prestige journals sometimes publish simple data on unusual questions --- if the topic attracts you, ideas for improving the method or generalising the results will soon come.
    • For the same reason, you can often find mental stimulation in journals from 60 or 80 years ago.
  • Your attitude: sympathetic criticism. So much junk is published that a degree of sceptical hostility is appropriate when approaching most papers (unless the author is on the Approved List). But temper this with sympathy: a paper may be useless to you, and it may even be clear that by any reasonable standard the research was a waste of time, but yet the method may be useful to one reader, a detail in the results may be just what a second reader is looking for, and a third reader may find her thoughts clarified by a point in the discussion.

    If someone says your paper is trash, retort that anyone who writes a decent paper can get it published, but it takes real talent to get rubbish into print.

• Forget harsh realities for a moment, and think seriously about what question really interests you, gets you passionate. Even though you're trying to keep your mind on the ideal, you will find practicalities obtrude themselves. Some of these practical problems may turn into research projects.
• Look to a leisure interest of yours for inspiration. Many students are interested in at least one of music, sport, and politics; well, if their academic subject is psychology, surely this has some interesting things to say about all of these? The same goes for many other academic subjects.
• Many of us are lucky enough to have easy access to computerised databases. Try searching them with an unusual combination of words, e.g., statistics and music, or laughter and Wagner, or measurement and theology.
• Ask successful researchers in your department whether they have any particular methods for getting ideas to come.
• I find making lists, and then organising them, often useful. Lists of what? Of questions, possible experiments, possible surveys, sources of data, ways of operationalising a concept, ways in which your supervisor could be more helpful,...
• Draw an analogy. (Theoretical, or method of analysis.)
• Thought experiments. If the results were to be ..., then we would conclude .... On the other hand, if they were to be ..., our conclusion would be ....

Suggestions for what to do if you don't have any ideas.

• Criticise a paper by someone else.
  • Can I improve the experiment?
  • Can I improve the theory? In statistical modelling, one might replace a discrete variable with a continuous variable, or vice versa --- e.g., if "blue" and "white" types have been proposed, replace them with a trait ("blueness").
  • Can I refine the definition?
  • Can I broaden the circumstances in which the effect occurs?
  • Can I simulate this on the computer?
  • Can I improve the statistical analysis? ("Two of a trade never agree", it is said, and certainly no two statisticians have agreed with each other this century.) Improvement does not necessarily mean additional complication. Sometimes, simple descriptive results are overlooked in the rush to perform a complicated hypothesis test. If nothing better comes to mind, try confidence intervals in place of hypothesis tests, or nonparametric methods in place of parametric, or Bayesian weight of evidence in place of a p-value.
  If you do comment on a published paper, you're actually paying it quite a high compliment. (If you come across 24-carat dross, you should follow the convention of politely ignoring it.)
• You can always write a paper on:
  • The useability of university Calendars.
  • The comprehensibility of ergonomics textbooks.
  • The rise and fall of the use of particular cliches in the titles of learned articles.
  • Do readers want more raw data in journal articles?
  • Bayesian weight of evidence in psychology journals, 1970--1996.
  • How first-year students view research participation.
  • Comparison of several fields in respect of their tolerance of low survey response rates.
  • Why do students hate statistics?
  Get the idea?
• Think of an experiment that no-one else has done, and do it. (The emphasis here is on doing something because it can be done, not because you actually understand the implications.)
• The different packaging of something unoriginal.
• Go to a journal that is less quantitative than you are, and teach the readers something (be tactful, now).

General advice.

• If you have a rigid deadline to work to, it is vital that you are not reliant on someone else for anything important. Among the possible areas of difficulty are: administrative approval from an outside body, construction of equipment, computing. You must have a workable plan that can be implemented if approval is refused/the equipment isn't delivered/the computing expertise is unavailable.
• Most people beginning research underestimate the importance of the physical aspects --- going through 500 files a second time because you didn't extract a particular item of information the first time, coming in to the lab on Sunday afternoon to try out an idea you've just had, driving at 4am to another city because the evening news told you of a "natural experiment" there that day, taking 60 journal volumes off the shelf to skim through for the key paragraph you wish you had made a note of, and so on.
• Write down an idea when you have it, because you may easily have forgotten it by the time you get to the office. Some people even keep an ideas book for this purpose; but the trouble with this is that it really need to be physically very small, so that it can be always with you.
• Work around the problem, if you can't solve it.
• Sometimes a problem is best handled by defining it away.
• There is some research that simply isn't worth doing. It may be obvious to the drover's dog roughly what the problem is... and that the institutional barriers to solving it are insurmountable. Surveys, especially, can be a substitute for actually doing anything about the problem.
• Importance of flexibility.
• Luck. Have several projects going --- diversification --- they won't all be disasters.
• Don't be afraid to admit your ignorance. The nature of academic research is that one is continually venturing into areas where one is an ignoramus. It is unpleasant to admit this. But the earlier you do, the sooner you get an explanation and the quicker you can put that little bit of extra learning to use.
• There can be a danger in overmuch personal involvement. Some people love people with Alzheimer's disease, or autism, or depression, and want to research the condition. For a beginning researcher, the problem of separating scientific evidence about the mass of people with the condition from the loved one's personal experience is one problem too many --- it is better to select another subject for research.
• Perfectionism and attention to detail is a virtue. Taken to excess, it is a vice.
• Some researchers seem to suffer from a "fear of success", which I think is different from obsessive perfectionism. I think what happens is that they perceive the successful completion of their research as the end of one stage in their life and the beginning of another, and they are reluctant to move on to the new stage.
• Universities have become very bureaucratic about documenting the progress of research students. One can appreciate the reasons for this, and yet there is a sense in which it is all a waste of time --- if the student is successful, there's no need to have a file of routine documentation; if he or she isn't, no amount of paperwork will rescue the situation.

  The supervisor may feel the student is wasting his or her time, and yet be reluctant to say this (or say it forcefully enough) because it may actually be wrong and because in any case it will be discouraging. The supervisor may proffer clear advice, and yet the student be stubborn. If I had an answer, I'd give it here...

• All sorts of stuff has been written about the role of the supervisor. Much of it is sensible. But it's all a long-winded way of saying that it is reasonable for the student to expect the supervisor to listen to him or her for something like 20--50 hours per year. Anything more than that is taking a responsibility away from where it properly belongs (i.e., with the student).
• Don't miss any deadlines that have been agreed with your supervisor. If you do, then (depending on the nature of your excuse) you'll go on to either the list of people who probably won't make it, or the list of people who probably won't make it but who deserve sympathy. You do not want to be on either of these lists. And don't be late for meetings with your supervisor.
• Don't expect much praise from your supervisor. If your supervisor thought about it, he or she would realise that a few words of praise were both justified (by the brilliance of your work) and desirable (for the benefits of positive reinforcement). Sad to say, he or she doesn't think about it: from about your second week of research, you've been accepted as a grown-up and been judged by the same standards as the professors.
• You might find it worthwhile reading How to get a PhD. Managing the peaks and troughs of research by E M Phillips and D S Pugh (Milton Keynes: Open University Press, 1987).

  Here are three points that Phillips and Pugh make, all of which I mildly disagree with. (i) Intelligence-gathering (i.e., description of the facts) is not research. (ii) Research of the testing-out type (as contrasted with exploratory research and problem-solving research) is much the most appropriate for a PhD student. (iii) It is necessary for your thesis to have a "thesis", in the sense of a connected argument and message.

  It may be that Phillips and Pugh are substantially correct in the context of relatively mature fields of study; and that my opinion that (i) intelligence-gathering may be research, (ii) exploratory research is useful and appropriate, and (iii) the research may be too messy to have much of a "thesis" to it, is formed by my experience being mostly in relatively immature fields of study.

• Taste. Is it permissible to use phrases like "glass ceiling" or "evidence-based medicine" or "gold standard" in reporting your research? Fastidious writers avoid cliches most of the time, but not every use of them should be condemned. (In some allegedly scholarly fields of study, they seem to be compulsory.)
• Don't worry about the possibility of someone else duplicating your research. It is rare for this to happen, and, when it does, it is stimulating to examine the similarities and the differences.
• Don't despair if it becomes necessary to tear up two years' work. This is quite common, and often means you are so expert and can see things so clearly that you can finish the whole project in only six more months.
• If you do happen to discover something important, be sure to do it at the right time and place. Perhaps you remember the inventor of the Infinite Improbability Drive: "just after he was awarded the Galactic Institute's Prize for Extreme Cleverness he got lynched by a rampaging mob of respectable physicists who had finally realized that the one thing they really couldn't stand was a smartass" (The Hitch Hiker's Guide to the Galaxy, Chapter 10). Read here about an earlier savant who suffered an equally dire fate.

Applying to do research.

There is quite a lot of advice available on the WWW about how to choose which departments to apply to (for a research degree), and how to increase your chances of success. I'm a bit sceptical about the need for this --- I think that by the end of their undergraduate careers, most students know in what sub-field of the subject they want to work, or what style of approach they want to use. Furthermore, they know which departments at which universities study that sub-field or take that style of approach. Consequently, their list of desirable departments is already a short one; if they have made personal contacts, the list will be very short indeed. However, if you think you need advice, here is mine.

• You should appreciate that there are an enormous number of specialisations available in the scholarly world. Flick through one of the directories of scholarly societies if you don't believe me. For example, there is a Colour Society of Australia, an International Association for Cross-Cultural Psychology, and a Stress and Anxiety Research Society. If some specialisation appeals to you (surely you'll like something in all your undergraduate studies!), then sometime about the middle of your undergraduate career, you should stop thinking of yourself as (for example) an embryo psychologist, and start thinking of yourself as (for example) an embryo colour psychologist. You will then only be interested in departments that are strong in colour studies.

  If you are in this position, do not limit yourself to departments of psychology. You might find the most suitable place is in a group within a chemistry or physiology department.

• Make personal contacts. If colour psychology really does fascinate you, make contact with people in the field. Academics are enthusiasts for their own little patch, and will be delighted to find someone who shares that enthusiasm. If possible, get some lowly job in their department during the vacation. But be warned --- I would say it is impossible to successfully fake this. Alleged enthusiasm without an appropriate level of specialist knowledge and native intelligence will get you nowhere.
• If you really are unsure --- if you just feel you'd like to do research in "psychology" or "chemistry" --- you may do better to delay your entry into research until your ideas have matured a little further. (You cannot be expected to have a fully-thought-out research project at this stage; what I mean is that you should know (for example) that your interest is in cognitive psychology studied by computer-controlled experiments on normal humans.)

A little advice of a statistical or technical nature.

This is not the real theme of the present document, but a few comments of this type may be worthwhile. You might also like to read Pitfalls of data analysis by Helberg.

• There is often tension between the respective advantages of standardising on one particular method of answering a question, and of using a variety of different methods. Think of measuring unemployment, or television viewing, for instance. On the one hand, one wants a standard, reproducible, method --- comparisons from month to month are important, and one has reason to think that whatever defects are present remain pretty constant over time. On the other hand, any single practicable method is bound to have peculiarities and biases, and the "real, true" answer should probably be synthesised from several different methods, even if each of them makes use of only a small sample size.

  When controversies break out, one factor is often the intrusion of a new player into the system: the existing interests have learned to co-exist with each other and with the defects of a question-answering method, but this doesn't suit someone new.

  The following is very much a broad-brush statement, to which there must be numerous exceptions: too often, a standard method is adopted too early in the development of a subject. In research, one usually wants the "real, true" answer, whereas for administration, one may prefer a standardised, reproducible, answer.

  "Figures often beguile me, particularly when I have the arranging of them myself" (Mark Twain). Presumably sociologists have studied the use of statistics in the wielding of power?

• Meta-analysis: this term refers to quantitatively combining the results of previous studies of a subject, in order to arrive at the right answer. I'm not terribly keen on the idea, but this may be because I have not worked in fields where there have been lots of previous studies of the same question. There are some difficult principles and practicalities involved, and inadequate handling of these led critics to refer to meta-analysis as being "meta-silliness". I think there have been substantial improvements in the methodology, and these were winning me over, but then I read a paper by Sohn (Clinical Psychology Review, 16, 1996, 147--156), which argues that publication bias is so serious as to vitiate all conclusions from literature reviews in the field he examines, effectiveness of psychotherapy (and I think the implication of the paper is that publication bias has a more serious impact on a meta-analysis than on a review in traditional narrative format).
• Contrast between exploring data, and using data for hypothesis testing.
• Maybe you want to do both. One easy technical solution is to randomly split your data into two portions, explore one half and generate hypotheses from it, and then test the hypotheses using the other half.
• Importance of a random sample, or random allocation. I have the feeling that often it is foolish to prefer a rubbishy sample of 200 to a random sample of 20, yet often this is done.

  It might be possible to argue that a rubbishy sample of 200 is the better for hypothesis generation, whereas a random sample of 20 is the better for hypothesis testing.

• My impression is that the subject which is leading the way as regards strictness of methodology is medicine. Many people will be familiar with the basics. In order to measure the effect of something, it is necessary to compare it with something else. That is, we have a treatment group and a control group. Patients are assigned at random to the one group or the other. It is desirable that neither the patient nor the doctor who evaluates the patient's condition know whether the patient received the drug or not. (This is termed a double-blind experiment.) But did you know this idea is being taken so far that Gotzsche (Controlled Clinical Trials, 17, 1996, 285--293) seriously suggests that the data should be blinded during statistical analysis? That is, two reports are written. The first assumes A to be the treatment and B to be the control, and the second is based on the reverse assumption. Both are completed before the code is broken. This is to avoid bias creeping in during the data analysis and writing.
• Moderate-sized effects imply large studies. What may turn out to be the most important statistical paper of 1996 was published in the Oxford Textbook of Medicine, of all places: the convincingly-written article by Collins et al (1996) will surely encourage tens of thousands of medical professionals towards good statistical practice in their research. One of the messages is that there may be a number of treatments that are only moderately beneficial, but yet would be very worthwhile because the disease is common or the treatment is cheap or both; and which, because the treatment vs. control difference is not great, require large sample sizes in order to establish statistical significance of the difference. These sample sizes are so large that they imply collaboration between many investigators. Collins et al give clear and interesting examples of both large single clinical trials (e.g., aspirin in treating heart attacks), and of meta-analyses of many small trials (e.g., adjuvant therapy for early breast cancer treated surgically). The effect sizes in these studies were large enough to be well worth having --- a reduction in 35-day mortality from 11.8% to 9.4% among patients with acute myocardial infarction who were treated with aspirin, and a reduction in 10-year mortality from 48% to 38% among women aged 50+ with stage II breast cancer who had tamoxifen included in their treatment --- yet were also small enough that clinical trials of the usual sizes were too small to detect them.

  It would not surprise me if there were to be a trend towards simple, large, collaborative studies in many areas of scientific enquiry besides medicine. Now, consider that very many students conduct some sort of small research project as part of their studies. I suggest that if different students in different universities together decided on a common topic and worked out a common methodology (and stuck to it), then (i) the results would be of much greater value than results from a single institution, and (ii) the experience of collaborative work would itself be valuable training, if indeed we are entering an era when this will be increasingly common.

  Collins R, Peto R, Gray, R, Parish S (1996). Large-scale randomized evidence: Trials and overviews. In Weatherall D J, Ledingham J G G, Warrell D A (Editors), Oxford Textbook of Medicine. 3rd Edition, pp. 21--32. Oxford: Oxford University Press.

• Don't get carried away with the statistical analysis. Tell the story of your research using the tools from your first course in statistics, not your last --- that is, using nothing more complicated than well-chosen descriptive techniques plus the concept of standard error. As much as is practicable, present your results in such a way that the reader can see that the conclusions from your more complicated calculations are probably correct.

  Suppose your research is experimental; and involves obtaining a measurement (e.g., an accuracy score) from each subject in each of four conditions; that the four conditions consist of each combination of two levels (e.g., big and small) of one factor with two levels (e.g., light and dark) of another factor; that the subjects are in two groups (e.g., males and females); and that interest centres on whether the average interaction effect is different for males from what it is for females. Such an experimental question would be recognised by some people as being about the three way interaction in a three-factor experiment, one factor being a between-subjects factor and two of the factors being within-subjects factors. Such people would know that a concise way of analysing the data and presenting the results would be via Analysis of Variance. I recommend: (a) Yes, do that. (b) But also, present your results in a way that enables them to be roughly checked. (Otherwise, your readers will presume you've got them wrong.) In the example, the interaction can be calculated for each subject. It is the difference between two differences, [(big, light) - (big, dark)] - [(small, light) - (small, dark)]. So calculate this for each subject, and summarise the values for the males by their mean and s.d., and the values for the females likewise. It will then be clear how similar or different are the interactions for the two genders.

  It might be worth going to quite some trouble to make sure you, your supervisor, and your examiners agree on this one. Having gone to the trouble of really understanding your data and then explaining it comprehensibly, it would be vexing to come across an examiner who wants to see 20 pages of ANOVA tables. Or vice versa.

• Just occasionally, it may happen that careful attention to notation in writing an equation will reveal that you really did not understand something you thought you did (or even that everyone else did not understand it).

  This can happen with any type of mathematical equation, not necessarily one involving random variables. But, to give a simple example of this latter type, there might be an averaging process somewhere, and one can sometimes get confused about what domain it is that one is averaging over.