Setting Improvement Targets In The Presence of Limited Data

View All Blogs

Target Setting With Limited Data
In this article, we will look at whether the improvement target set for a project makes sense in the presence of historical variation present in the process and the data and/or the time we usually have to validate the improvements. If you have the luxury of unlimited data/time, you can choose to set any target for your project. However, there are times when we are limited by the amount of data that is available for our project or we are limited by the time available to complete the project. This article is applicable to those cases where we have limited time/data available for our improvement project.

Importance of Targets
We set targets at the beginning of any improvement project to not only provide direction to the team working on the project but also to be able to check at the end of the project whether we were able to accomplish the goals set for the project. A project without a target is like playing a game where we don’t keep score. At the end of the game, we have no idea if we “won” or “lost”. Target or goal setting has to be SMART – Specific, Measurable, Actionable, Relevant, and Time Bound. Setting the target is a delicate balance. If the target is too difficult to achieve based on our current process performance, the team may not be successful, or it may be a de-motivating factor for the team if they are not able to accomplish the targets. Often, these targets are also mentioned on employee performance improvement plans and not achieving the targets could be a disincentive to the employees. If the target set is very easy to accomplish, then the company does itself a disservice by not setting aggressive targets and only achieving mediocre improvements in performance.

How to Set Targets
There are many factors that one needs to consider when a target is being set – inputs from management, requirements by our customers or other stakeholders, to stand apart from our competitors, in order to meet certain stated cost reduction or other targets, or meet the benchmarks set historically by the same process or by other processes either within or outside the company. If you are already the best-in-class compared to others, the target may just be an internal mechanism to continually raise the bar and challenge your workforce to do better. This target or the goal is typically documented on a project charter and the actual improvement achieved at the end of a project is compared to this goal to determine whether the project was a success.

Data Collection for Validation
In order to prove that the team was able to achieve the stated performance, data must be collected at the start of the project to establish the current baseline and at the end of the improvement exercise in order to show that the targets were met. Each process is different and the data that is collected depends on the availability of existing or deployed data collection plans. If no existing data collections exists, then the team may put in temporary or permanent data collection methods to collect the data on the metric(s) of concern. In order to prove that an improvement was made enough data points are required. Having limited data is a problem since the results observed may be due to “chance” occurrence. Hence, sufficient data is required to establish that the goals were met.

How Much Data to Collect
The question is how many data points should one collect to establish an improvement? Too few data points may point a false picture and the improvement may not have been sustainable (calling in the victory too early or the results may be just due to the Hawthorne effect). If we collect too many data points, then it may take a long time after the improvement is made to determine if the required improvements have been achieved.

The number of data points we collect also depends on the process under consideration and the difficulty in collecting the required data. In some cases, data is only available on a monthly or quarterly basis – for example quarterly or monthly feedback surveys, monthly reports on performance of metrics such as inventory, sales, revenue, profitability etc. If we have access to only monthly data, then even to collect 3 data points, we would have to wait 3 months after the improvements are completed before we have evidence of the improvement. Which is not usually acceptable. Often, the management is not willing to wait for a long period of time to demonstrate improvement. Hence, it becomes important for these types of improvements that we have at least weekly data points on the metric(s) of consideration. If you have weekly data points, then you will get at least 4 data points in a month. Of course, having a higher frequency of data collection such as daily or hourly is even better. However, in some cases, it becomes difficult or even cost prohibitive to get access to a lot of data. For this article, we assume that the data collection frequency is at least weekly.

Example
Let’s look at an example for improving the revenue generated by the sales team. Assume that the average current revenue is $1000/week and as in the real-world let’s assume that there is variation in our revenue collection, the revenue varies from worst-case scenario of $520/week to a best-case scenario of $1470/week. This variation can be reported as the range of $950 (max - min) or as a standard deviation (roughly $200). Assume that we have weekly reporting of the revenue numbers and we would like to make an improvement of 10% in our revenue collection from $1000/week to $1100/week. If we want to complete this project in a few months, and we use up one month to come up with the improvements, we would have at most 4 data points available for validation of our improvements. The question is whether this improvement target makes sense given the amount of variation we see in the process?



Analysis
For the example shown above, the coefficient of variation (CV) is 0.2 which is obtained by dividing the standard deviation (200) with the mean value (1000). From sampling theory, we can estimate the minimum difference in mean value for this case using the following formula:

μ_2-μ_1=(Z_(α/2)*σ)/(√n)

This formula was derived for a confidence level of (1- α) and a power of 50%, where μ_1 is the mean of the process before the improvement, and μ_2 is the mean of the process after the improvement, Z_(α/2) is the standard normal variable, σ is the standard deviation of the process, and n is the number of data points. If we plug in the following formula for the coefficient of variation and re-order the terms, we get the following:

μ_2-μ_1=(Z_(α/2)*CV*μ_1)/(√n)

(μ_2-μ_1)/μ_1 =(Z_α*CV)/(√n)

For a 95% confidence level, the Z_(α/2) value is around 1.96 and if n is around 4, then the minimum difference we would be able to detect would be

(μ_2-μ_1)/μ_1 ≈ CV

In other words, if we only have around 4 data points for validation, we would not be able to detect a % improvement in our means that is less than the coefficient of variation. The following table provides a relationship between the number of data points available for validation and the % improvement we can hope to demonstrate:

Months	Num Points	Minimum Demonstrable Improvement
1	4	100% of CV
2	8	1/√2 ~ 70% of CV
3	12	1/√3 ~ 60% of CV
4	16	1/2 ~ 50% of CV

For our example, the coefficient of variation was 0.2. Hence, with only one month of validation data, we can only hope to demonstrate a minimum of 20% improvement. In our case, we had tasked our team to generate a 10% improvement – this would be hard to detect. In this situation, we have three options:

• We could pick a different target for example demonstrate an improvement larger than 20% (if feasible),
• We could ask management for more time to complete the validation exercise (4 months in this case),
• We could try to collect data at a higher frequency if possible – such as daily data (if practical).

Conclusion
In summary, we have shared a simple heuristic guideline for checking if the goal/target set on a continuous improvement project can be validated. For any project that you would like to undertake, calculate the historical coefficient of variation (CV) and make sure that if you have limited data points available for validation, then the value you select for your goal is greater than the minimum improvement you can demonstrate. If your goal is smaller than this number, you will not be able to validate the improvements on such a project. In such a scenario, you can either pick a higher goal/target, change your data collection plan to collect data at a higher frequency, or extend the duration of the project so that more data can be collected for validation.

---

Follow us on LinkedIn to get the latest posts & updates.