Friday, 2 August 2013

Guessing on Multiple Choice Tests

You've probably heard someone give advice along the lines of "when in doubt, guess 'C'" when taking multiple choice tests. The alleged reasoning behind it is that there's a statistical advantage in guessing 'C' consistently rather than randomizing your answers.

I decided to investigate this claim. Let's first check the case when the professor distributes the correct answers evenly and randomly, not favouring any one letter. Take for example a test with 30 questions, where each answer is independent of the others and there are four choices per question (A, B,C, or D). Assuming that you really have no clue which is the correct answer, any guess has a 25% probability of being correct. This is true regardless of whether you choose randomly (let's ignore for the moment that humans are notoriously bad at generating random values on their own) or if you decide to choose the same letter consistently. Here's why:

The instructor randomly chooses which letter is correct, without bias, so the probability of any letter happening to be the correct answer on a particular question is 25%. If you also guess randomly without favouring any letter (i.e. each letter is guessed, on average, 25% of the time), you should then expect that 25% of the correct answers are 'A', 25% of your guesses are 'A', thus 6.25% of your guesses (25% of 25%) are correct. The same is true of 'B', 'C', and 'D', so that overall you expect 4*6.25% = 25% of your guesses to be correct. Now if you consistently choose 'C' for the same random test, the probability that 'A' is correct is 25% but the probability that you guess 'A' is 0%. This is also true of 'B' and 'D'. When you choose 'C' 100% of the time, overall you should expect that 3*0.25*0 + 1*0.25*1 = 25% of your guesses are correct.

Since the guess is either right or wrong, this is a problem for the binomial distribution. The mean is n*p and the variance is sqrt[n*p*(1-p)], where n is the number of trials and p is the probability of the desired outcome. In our example multiple-choice test, whether you guess randomly or choose 'C' every time, you would expect, on average, to get 30*0.25 = 7.5 correct answers. To reinforce the point that both approaches to guessing are equal here, I simulated 50,000 of these tests in Excel and generated the following plot.
Probability distribution for correct guesses on tests with random answers to four-choice questions.
The average was 7.49 correct answers with a standard deviation of 2.37 when all guesses were random. When 'C' was guessed consistently, the average was 7.50 with a standard deviation of 2.38. The binomial distribution with n = 30 and p = 0.25 predicts an average of 7.50 and a standard deviation of 2.37. It's pretty clear that both guessing schemes give the same results when the correct answers are random and evenly distributed among the possible choices. 

Now that it's clear that there's no advantage if answers are randomly and evenly distributed, let's investigate the case where the instructor favours one letter over the others. It turns out that no matter how biased the instructor's distribution of correct answers might be, if you randomly guess each letter 25% of the time, your probability of choosing the right answer is still 25%. Let's assign unknown probabilities for each letter being assigned as correct by the instructor: pA, pB, pC, and pD. The sum of these unknown probabilities must be unity. So when the probability of guessing A, B, C, or D is 25% each, the probability of being correct is then 0.25pA+0.25pB+0.25pC+0.25pD = 0.25(pA+pB+pC+pD) = 0.25*1 = 25%. That all changes if you consistently choose 'C' though. In this case, your probability of getting the right answer is 0pA+0pB+1*pC+0pD = pC. I varied pC between the most unfavourable case (where 'C' is never the correct answer) to the most favourable (where 'C' is always the correct answer) to generate the following plot:

Probability distribution for correct answers when 'C' guessed consistently for different values of pC.
Obviously, there's some benefit to choosing 'C' if you know that the instructor favours 'C' over the other letters. But you also get screwed if the instructor doesn't like to use 'C' or simply chooses to favour another letter because he wants to penalize the people who always guess 'C'. While it looks pretty nice that more of the curves I plotted are shifted to the right rather than left of the curve for random guessing, if the choice of which letter gets favoured is random, the chance that 'C' is never the right answer is higher than the chance that it is always correct. To illustrate, I ran a simulation which I think is slightly more realistic than the examples above.

I assumed that pB + pC is 54% on average, which is just a guess on my part but I feel it is reasonable because answers aren't randomly assigned a letter. Numerical answers are often ordered from smallest to largest and statements like "All of the above" are reserved only for 'D' because they'd be confusing if they weren't. I used the normal distribution to generate the random variations in individual tests, so that pB + pC can vary from 0 to 1, but is usually close to the average. I similarly used normally distributed random numbers to split up pB and pC, so that on average pC is half of (pB + pC), but can vary from 0 to 100%. Same idea with pA and pD. Random guesses by the test taker are still equally distributed on average between the four choices. Plotted below are the results of 250,000 simulated tests.
Probability distribution for correct answers based on "realistic" multiple choice tests.
With random guessing we expected to be right 25% of the time on average and to see a binomial distribution after many tests. When 'C' is guessed consistently, things get more complicated, but a simple approximation can be found using the normal distribution. As you can see in the plot, the approximation works fairly well. It is clear that random guessing gives you more consistent results than guessing 'C' all the time. Always guessing 'C' increases your chances of getting 1 in 3 guesses right, but it also increases your chances of doing worse than 1 in 6, simply because your results are influenced by how the distribution of answers was biased by the instructor for the particular test. Overall, consistently guessing 'C' resulted in 2% more correct answers. But of the 250,000 simulated tests, random guessing beat consistently guessing 'C' on a total of 125,568 tests (i.e 50.23% of the time).

To summarize, if you happen to know that the creators of the test favour 'C' for correct answers, guessing 'C' consistently gives you an edge. In the long run, sticking with 'C' probably gives you a very slim advantage over random guessing, though guessing randomly gives you better consistency in the results of your guesses.

15 comments:

  1. Or you could look to see what letter has been used for other answers that you know are correct and extropolate which letter is being preferred, then use that as your constant guess... which means skip the questions you do not know until you answer those you do know. Then go back and guess, if necessary.

    ReplyDelete
    Replies
    1. PERFECT LOTTERY SPELL THAT WORK FAST WITHIN 24 HOURS WITH DR RAYPOWER LOTTERY SPELL POWERS
      I saw so many testimonies about Dr Raypower a great lottery spell caster that will help you cast a lottery spell and give you the rightful numbers to win the lottery, i didn't believe it, at first but as life got harder i decided to take a try,  I contacted him also and told him i want to win a lottery he cast a lottery spell for me which i use and i play and i won $3,000,000 (THREE MILLION DOLLARS). I am so grateful to this man Dr Raypower and I am making this known to every one out there who have been trying all day to win the lottery, believe me this is the only way to win the lottery, this is the real secret we all have been searching for. Do not waste time contact him today for you to also be a winner contact info below.

      Email: Urgentspellcast@gmail.com or Urgentspellcast@yahoo.com

      WhatsApp: +1 (424) 330-8109

      Website: https://urgentspellcast.wordpress.com/

      Delete
  2. What you've described is essentially an example of the gambler's fallacy. In a small sample of random questions, it is likely that one answer will be selected more often than the others. However, all the correct answers are independent of each other (assuming that the test questions don't refer you back to the answers of previous questions). Assuming no bias by the instructor, if you're part way through a test and find that 1/3 of your answers are 'C', there's still no guarantee that 'C' is actually more likely to be correct. It might be a clue, but it could also just be how the random selections worked out. If you see one letter showing up more often, you'd have to perform a statistical analysis and decide if this observation is statistically significant. Constant guessing is only advantageous over random guessing when your choice really does have a higher probability of being correct. Therefore, you must be confident that there really is some bias in the test.

    ReplyDelete
  3. Answering C will fail True or False questions. Make sure you answer A or B on a True-Or-False.
    Answering "all of the above", on questions that have "all of the above" as an answer, will likely be beneficial as well.

    ReplyDelete
  4. There are a lot of online apps and the courses that are helping the candidates to study better and that too without leaving the home. Even I love to study without any distractions so my room is just perfect place for that. This is why I have picked one of the best Mobile Bar Review Courses that is helping me immensely.

    ReplyDelete
  5. PERFECT LOTTERY SPELL THAT WORK FAST WITHIN 24 HOURS WITH DR RAYPOWER LOTTERY SPELL POWERS
    I saw so many testimonies about Dr Raypower a great lottery spell caster that will help you cast a lottery spell and give you the rightful numbers to win the lottery, i didn't believe it, at first but as life got harder i decided to take a try,  I contacted him also and told him i want to win a lottery he cast a lottery spell for me which i use and i play and i won $3,000,000 (THREE MILLION DOLLARS). I am so grateful to this man Dr Raypower and I am making this known to every one out there who have been trying all day to win the lottery, believe me this is the only way to win the lottery, this is the real secret we all have been searching for. Do not waste time contact him today for you to also be a winner contact info below.

    Email: Urgentspellcast@gmail.com or Urgentspellcast@yahoo.com

    WhatsApp: +1 (424) 330-8109

    Website: https://urgentspellcast.wordpress.com/.

    ReplyDelete
  6. I do not think I would have ever stop wiping if not for Dr Osasu When my lover left me the hole world was like hell I could not cope with it neither can I breathe like i real human. But all the-same I thank God for using Dr Osasu to restore my lose love and making the rest of my life full of joy and happiness by bringing back my ex lover back to me with his powerful spell. All praise and honor be onto you. if not for you Dr Osasu I don't thing the rest of life would have been sweet, so all I will have to do now is to introduce you to people out there who are in similar problem or another to contact you and get there solution just like me.

    Here are his details
    (drosasu25@gmail.com) or call him : +2347064365391

    ReplyDelete
  7. I do not think I would have ever stop wiping if not for Dr Osasu When my lover left me the hole world was like hell I could not cope with it neither can I breathe like i real human. But all the-same I thank God for using Dr Osasu to restore my lose love and making the rest of my life full of joy and happiness by bringing back my ex lover back to me with his powerful spell. All praise and honor be onto you. if not for you Dr Osasu I don't thing the rest of life would have been sweet, so all I will have to do now is to introduce you to people out there who are in similar problem or another to contact you and get there solution just like me.

    Here are his details
    (drosasu25@gmail.com) or call him : +2347064365391

    ReplyDelete
  8. How I Got My Husband back... Am so excited to share my testimony of a real spell caster who brought my husband back to me. My husband and I have been married for about 6 years now. We were happily married with two kids, a boy and a girl. 3 months ago, I started to notice some strange behavior from him and a few weeks later I found out that my husband is seeing someone else. He started coming home late from work, he hardly cared about me or the kids anymore, Sometimes he goes out and doesn't even come back home for about 2-3 days. I did all I could to rectify this problem but all to no avail. I became very worried and needed help. As I was browsing through the Internet one day, I came across a website that suggested that Dr. Wale can help solve marital problems, restore broken relationships and so on. So, I felt I should give him a try. I contacted him and told him my problems and he told me what to do and I did it and he did a spell for me. 48 hours later, my husband came to me and apologized for the wrongs he did and promised never to do it again. Ever since then, everything has returned back to normal. My family are living together happily again.. All thanks to Dr. Wale If you have any problem contact him and I guarantee you that he will help you. He will not disappoint you. you can WhatsApp him +1(978) 406-9575 or Email him at : everlastingspellcast@gmail.com or or view his website for more review https://everlastingspellcaster.website2.me/contact

    ReplyDelete
  9. REAL URGENT EFFECTIVE SPELL CASTER TO HELP BRING BACK EX LOVER AND SAVE MARRIAGE

    Hello everyone, my name is Flores. I am so overwhelmed with joy all thanks to Dr Odianosen spell. My husband left me for another woman a few years back and I was very devastated cause I never did anything wrong to him, I was left with my two kids and a job that pays little. I was almost giving up until I saw a testimony online about Dr Odianosen and I decided to contact him. I explained my problem to him and he assured me I'll see a positive result after 24 hours, surprisingly my husband came back the day after the spell begging me to forgive him and promised to never leave. My husband has been back for 6 months now and we've never had any issues, am glad I didn't doubt Dr Odianosen cause he sure can solve any problems with fast relief you can also contact him for help now

    Email: abelodianosen11@gmail.com or

    WhatsApp: +2348107771101

    ReplyDelete
  10. Good day to you, here is a quick one, just in case you are desperately in need of a trustworthy spell caster to help revive your crumbled relationship. Here is one that is truly versatile, compassionate and reliable when it comes to spell casting.. Doctor Abolo spelling home is the right choice for you. Contact and explain to him and I can assure you that the rest would be complete magic, my love life was on the verge of collapse until I ordered for a love spell while all due procedures were properly followed, I am a living witness that Doctor Abolo spell magic does work that is why i'm using all medium to let the world know of this great witch doctor who brought back peace and unity to my home, Talk to him today and have your emotional setbacks wiped out totally.
    You reach him at these contacts! Abolospell@gmail.com

    ReplyDelete
  11. Ready to live, work or study in Australia or Canada? Buy your IELTS certificate now with
    We can help you obtain high scores in IELTS Certificates online without you attending the exams. All our certificate are registered and can be verified online, you can use this certificates for Universities admission and any immigration use . The regions we cover are UAE, Qatar, Oman, Saudi Arabia, Kuwait, India, Pakistan,visit.


    Philippines, Jordan, Australia ,Asia, Eu(UK), Canada, Italy, Ireland, Germany, USA etc.
    Because the business is risky, very little information is provided to the public and details of the certificates are only provided to paying clients so Contact us for more info. We are confident in the quality of our services, but if suddenly you are not satisfied with the result of our work, we will refund you all the money for the work. Sometimes it happens that we do not fully satisfy our client and we have to refund the money paid for.

    Buy IELTS certificate online | Available ielts certificate without exam | Where to get ielts certificate | Get Ielts certificate without exam | ielts certificates without exam in europe | legal Ielts Certificate for sale online | Ielts certificate for sale without exam | Need Another genuine ielts certificate | Buy Genuine IELTS Certificate Without Exam | Genuine IELTS Certificate Without Exam | Original IELTS Certificate For Sale | Buy real ielts certificate online | Original IELTS Certificate for sale | Real original ielts certificate to buy | ielts certificate for sale in Canada | valid ielts certificate without exam in USA | Obtain legit ielts certificate in India | Order registered ielts certificate in Parkistan | registered ielts certificate for sale in Australia | Buy real British Council certificates in Africa | Buy real ielts certificate online | Buy valid ielts certificate 100% without exam | Get a genuine ielts certificate without exam | Ielts certificates for sale online | We issue ielts certificate without exam | How to get ielts certificate without exam | Buy genuine ielts certificate | Real and Valid ielts certificate without exam | Original ielts certificate for sale | BUY Genuine 7+ Band Score in IELTS | Buy Real IELTS CERTIFICATE ONLINE | Buy Original Ielts Certificate Without Exam | Buy Ielts Certificate | Buy Registered Ielts Certificate Without Exam | Buy registered ielts certificates for immigration | IELTS is acceptable every where you go | Travel with IELTS | Buy ielts certificate online | get toefl certificate online | idp ielts muscat | ielts rea | Idp ielts certificate | ielts ukvi Bahrain | b2 certificate Cambridge | british council in Milan Italy | cae exam online | b2 english certificate online | buy registered ielts certificate online | registered ielts certificate for sale | cae scale score | ielts without exam in Pakistan | ielts Portugal | cae certificate | Cambridge cae test online | ielts bari | idp ielts bari | idp ielts oman | ielts score for sale | ielts certificate without exam | | british council Soudan ielts registered | british council Italy ielts | buy gre test | ielts without exam | ielts ukvi Soudan | ielts Bulgaria | british council porto Portugal | buy ielts band | ielts b2

    Excited to know more about us, visit our website or write us on
    (WhatsapCONTACT DETAILS/WHATSAPP+1901-878-9747
    TELEGRAM=@Passportsservicese
    EMAIL ADDRESS………..passportsservices247@gmail.com

    ReplyDelete
  12. I have been playing the lottery since i was 19 years and now i am 47 meaning i have been playing the lottery for 28 years. The biggest amount I have ever won in my life was 1200 bucks. But one day my story turns to history after I find this man's name on the Internet that he is the best when it comes to winning the lottery. This man is a very strong voodoo Dr who gives out the lottery winning numbers that can never fail. After all my years of laboring and struggling to win the lottery i finally won ( $12,000,000) Dr Winner, is the name,EMAIL him now: (Winnersspellcast@gmail.com this is the only way to win the lottery and the best way via his website:https://winnersspellcast.blogspot.com) Call or text him: +2348138289852

    ReplyDelete
  13. Hello viewer am Albert mike from Kenya 29 years old am here to share my testimony on how i joined the brotherhood of illuminati.. four months back when i wanted to join the brotherhood i was scammed by fake agents and people pretending to be Grand Master use the website.. until i met a friend MR BERRY WARD who gave me the genuine procedures i followed and made me a member of the great brotherhood of illuminati. today i am living a good life and i have enough money to do all i ever wanted .if u are interested and u need the help of my Initiator Berry. Email grandmasterilluminati29@gmail.com MR Berry Ward
    thanks Berry, am living the dream i will forever be grateful.

    ReplyDelete