Hypothesis Testing Overview

View All Blogs

Would you believe me if I say that I can predict the outcome of a coin toss? You would probably discount what I say, but what if I had a wand in my hand and had magician’s clothes? You may wonder – maybe I am telling the truth but most likely not unless you can see some proof. Let’ arrange for the proof. We toss a coin and I predict Heads and the coin does in fact come out to be heads. Would you believe me now? Probably still not since there was a 50-50 chance that the coin would have landed on heads anyway. What if I can predict the outcome of two consecutive tosses? I predict the first one will be Heads and the second one will be Tails. If in fact the outcome of your experiment was Heads on the first turn and Tails on the second, would you believe me now? Well, you may compute that there were four possibilities on the two tosses (Heads-Heads, Heads-Tails, Tails-Heads, Tails-Tails) and my prediction may still have been due to chance as there was a 25% chance (1 in 4) that I just lucked out and predicted the right sequence. You may still hesitate to believe me – what if I can do this prediction for 10 coins in a row. If you calculate the probability of that happening it is around 0.09% (1 in 1024). Now, you may wonder maybe I have some trick up my sleeve which is helping me predict the coin toss because the probability that this occurs by chance is so low that we would not see this in the normal course of our lives.


But such things do happen if we are not careful in our analysis. For example, we may have heard about famous stock pickers who can consistently pick good quality stocks in the long-run. We may even consider investing our hard earned money with such agents. There may be great stock pickers out there (such as Warren Buffet) but let me show you have you could claim to be such a picker. Let’s say that there are 5 quality stocks A, B, C, D and E. You invest a modest amount of money in each of the 5 stocks. Let’s say in one year stock A does exceptionally well. You now discard the other stocks and invest again next year in F, G, H, I, and J. Let’s say that this year stock G does exceptionally well. You again discard the rest and continue to do so every year. In about 10 years, you can claim that I was so good at investing that first year I picked stock A, second year I picked stock G, third year I picked stock P and so on and if you had invested with me 10 years ago, your cumulative return would have been 5000%. Here, we are only looking at the successes and not at the failures – so we encounter what is called a survivor bias. We cannot draw good conclusions if we only look at successes without also looking at the failures.


Similarly, when we are trying to solve a business problem, say P, and want to find out if R is a root cause. We need to collect some data to answer this question. When usually work with sample data and there are two types of errors we can make. Based on our data analysis, we may incorrectly conclude that R is not a root cause when in fact it really is a root cause and secondly, we may conclude that R is a root cause when in fact it is not the root cause of our problem. Ideally, we don’t want to make these types of mistakes in our analysis because these mistakes can be expensive. If R was a root cause and we miss it, we cannot put solutions in place to solve the real problem. If R was not a root cause and we picked it, we may be wasting valuable company resources addressing issues of minor relevance.


That is where hypothesis testing comes to our rescue – done properly, we can avoid the types of mistakes we discussed above. The power of statistics comes into play when we can make good conclusions with limited amount of data! There are a lot of different hypothesis tests out there and each test is applicable to a different situation, each have their own assumptions and requirements and if we want to do proper analysis, we need to pick the right hypothesis test. The hypothesis test we pick depends on the type of data we have for our primary metric or the main question we are trying to answer (Y) and the type of causes (X). For example, if we want to determine whether using software A or software B has an impact on our time to complete a task. The primary metric (Y) would be the time to complete the task (which would be continuous in this case) and the cause would be the software (discrete in this case with 2 possible values). The following chart will help you pick the right hypothesis test for your problem. Selecting the right test is just one step to solving the problem. Other steps include formulating the right hypothesis, selecting the right data for analysis, drawing the right conclusions etc. These topics will be covered in a separate blog.


The objective of this blog is to highlight that in the real-world we deal with limited set of sample data and in order to make good business decisions, we need to use the right statistical tools (hypothesis testing in this case). Of course, you may be wondering why go through all this complex stuff when simple tools would be enough to solve the problems. You would be right depending on the problem we are attacking. However, there are some classes of problems where you must perform proper data analysis and draw the right conclusions. Problems where too much money is involved, problems that require significant amount of time and resources to solve, or that entail significant risk. In these cases, we cannot just "hope" or "guess" through intuition that we have the right answer. If we do that, we might just as well be the mystic who can predict the future.

---

Follow us on LinkedIn to get the latest posts & updates.