Sigma Magic Help Version 15

Compare Samples

Overview

Compare Samples can be used to perform hypothesis testing analysis for the case when the output Y is continuous or discrete and the input X(s) is discrete. The analysis automatically determines the right test to use and applies that test to the input data points.

If the primary metric, Y data, is continuous and if there is only one data set that needs to be compared to an external standard, then the analysis either applies a 1-sample t-test if the data is normal or a 1-sample sign test or a 1-sample Wilcoxon test if the data is not normal. If both the data sets are normal, then a 2-sample t-test is used. If they are not normal, then a Mann-Whitney test or the Kruskal-Wallis test is used. If tests are conducted in a paired fashion, then a paired test is used. If there are more than 2 data sets and the data sets are normal, then ANOVA analysis is used, otherwise, a Mood's Median test is performed. A flowchart of the different methodologies is shown in the figure below.

The flowchart below shows the flowchart for comparing the means of continuous data. The flowchart below shows the flowchart for comparing the standard deviations of continuous data. If the primary metric, Y data, is discrete and we have one or two sets of data to compare then if we are working with defects, then the Poisson rate tests can be selected otherwise if we are working with defectives, then the Proportion tests can be selected. If there are more than 2 data sets, then the Chi-Square tests can be selected.

The flowchart below shows the flowchart for comparing the defects for discrete data. This tool can be added to your active workbook by clicking on Stats and then selecting Compare Samples.

Analysis setup

Click on Analysis Setup to open the menu options for this tool.

Setup

A sample screenshot of the setup menu for continuous data is shown below.

1	Data Type: Specify the type of data you are dealing with. The available options are: Option Description Continuous Continuous data refers to measurements on a continuum or scale that can be meaningfully subdivided into infinitely small increments, depending on the precision of the measurement system. Examples of continuous measurements are time, temperature, weight, currency etc. Defects Defects are any item or service that exhibits a departure from specifications. A defect does not necessarily mean that the product or service cannot be used. Defectives Defectives refer to the entire product or service and refer to the condition that the product or service is not usable. A product may have many defects - not all of these defects may cause the product to be defective.
2	Input Data: Specify the format for your input data. If you need to change your input data format, click on the Format Data button. Option Description Group Data Group data means that there are two columns with group description in one column and the data in another column. Enter the data column under the box labeled Analysis Variables and the group column under the box labeled Categorical variables. Raw Data Raw data implies that each data set is in a separate column. Enter the columns you want to analyze in the box labeled Analysis Variables. Summarized Data Summarized data refers to properties of the raw data such as mean, standard deviation etc. Enter the summarized data in the appropriate text boxes shown in the Data tab.
3	Num Data Sets: Specify how many data sets you are trying to compare. This is typically the number of X variables you have in your analysis. For example, if you are trying to compare the mean values delivery times of Supplier A and Supplier B. Then the number of data sets is 2 as we are comparing 2 sets of data.
4	Comparison: Specify the type of comparison you want to make. Option Description Mean or Median Continuous data refers to measurements on a continuum or scale that can be meaningfully subdivided into infinitely small increments, depending on the precision of the measurement system. Examples of continuous measurements are time, temperature, weight, currency etc. Std Dev Defects are any item or service that exhibits a departure from specifications. A defect does not necessarily mean that the product or service cannot be used. Proportion Defectives refer to the entire product or service and refer to the condition that the product or service is not usable. A product may have many defects - not all of these defects may cause the product to be defective.
5	Flowchart: Click on this button to open and view the flowchart for this tool. You can also click and choose the test you want to perform directly from this flowchart.
6	Help Button: Click on this button to view the help files for this topic.
7	Cancel Button: Click on this button to cancel your changes and exit from changing the settings.
8	OK Button: Click on this button to save your settings and try to compute the analysis results. Note that if all the required data have not been specified then your analysis results will not be computed.

A sample screenshot of the setup menu for discrete data is shown below.
Most of the menu options are similar to the continuous case. The two changes you would find are highlighted below:

1	Data Type: For discrete data you will need to change the data type to either defects or defectives.
2	Comparison: For discrete data the comparison changes to proportion. This refers to the comparison of either the proportion of defects or defectives between different data sets.

Hypothesis

A sample screenshot of the hypothesis menu for continuous data is shown below. However, the menu options for the discrete case are very similar as well.

1	Hypothesis Test: Specify the hypothesis test that you want to perform for your analysis. You can leave this setting at Auto and the software will pick the appropriate hypothesis test for your case. Use the flowchart to determine which would be the best test to use for your analysis.
2	Delta: For this analysis, you will need to specify the difference that is practically important for you. For example, if you are comparing the time it takes to repair TV sets, you may be interested in detecting any difference greater than 0.5 hours. If you leave the delta value blank, then the software will not compute the minimum sample size required for this analysis.
3	Null Hypothesis: Specify the null hypothesis. If you are comparing one set of data with an external standard, the null hypothesis value should be the external standard. Note that for proportions, the null hypothesis should be between 0 and 1. If you are comparing multiple sets of data with each other, enter a value of 0 to check if the mean or proportion are equal and enter a value of 1 if you want to check if the standard deviations are equal.
4	Alt Hypothesis: Specify the alternate hypothesis (less than, greater than, or not equal). The default setting is Not Equal. If you select either less than or greater than, then the software will perform a one-sided hypothesis test.
5	Confidence Level: Enter the confidence level required for your analysis. This controls your Type I or Alpha error (1 - Confidence Level). The default value for this is 95%.
6	Power: Specify the Power required for your test. This value controls your Type II or Beta error (1 - Power). The default value is 90%.
7	Hypothesis Statements: The hypothesis test that you are performing is shown as a hypothesis statement. Make sure you closely read this statement to ensure this is the test you are interested in performing. If not, make appropriate changes to the dialog box to reflect the test you want to perform. The hypothesis statement assumes some default settings for the null and alternative hypotheses. If you want to change these settings you will need to go to the Hypothesis tab.

Data

If you click on the Data button, you will see the following dialog box. Here you can specify the data required for this analysis.

1	Search Data: The available data displays all the columns of data that are available for analysis. You can use the search bar to filter this list and to speed up finding the right data to use for analysis. Enter a few characters in the search field and the software will filter and display the filtered data in the Available Data box.
2	Available Data: The available data box contains the list of data available for analysis. If your workbook does not have any data in tabular format, this box will display "No Data Found". The information displayed in this box includes the row number, whether the data is Numeric (N) or Text (T), and the name of the column variable. Note that the software displays data from all the tables in the current workbook. Even though data within the same table have unique column names, columns across different tables can have similar names. Hence, it is important that you not only specify the column name but also the table name.
3	Add or View Data: Click on this button either to add more data into your workbook for analysis or to view more details about the data listed in the available data box. When you click on this button, it opens up the Data Editor dialog box where you can import more data into your workbook, or you can switch from the list view to a table view to see the individual data values for each column.
4	Required Data: The code for the required data specifies what data can be specified for that box. An example code is N: 2-4. If the code starts with an N, then you will need to select only numeric columns. If the code starts with a T, then you can select both numeric and text columns. The numbers to the right of the colon specify the min-max values. For example, if the min-max values are 2-4, then you need to select a minimum of 2 columns of data and a maximum of 4 columns of data in this box. If the minimum value is 0, then no data is required to be specified for this box.
5	Select Button: Click on this button to select the data for analysis. Any data you select for the analysis is moved to the right. To select a column, click on the columns in the Available Databox to highlight them and then click on the Select Button. A second method to select the data is to double click on the columns in the list of Available Data. Finally, you can also drag and drop the columns you are interested in by holding down the select columns using your left mouse key and dragging and dropping them in one of the boxes on the right.
6	Selected Data: If the right amount of data columns has been specified, the list box header will be displayed in the black color. If sufficient data has not been specified, then the list box header will be displayed in the red color. Note that you can double-click on any of the columns in this box to remove them from the box. The data you specify for this analysis depends on the options you have specified in the Setup tab. Option Description 1 If you want to analyze a single column of data, you can specify that column under Analysis Variables and leave the Grouping Variable blank. 2 If you want to analyze multiple columns of data, you can specify the multiple columns under Analysis Variables and leave the Grouping Variable blank. 3 If you are working with grouped data then you enter the variable data under the Analysis Variables and the column that contains grouping information under Grouping Variables.
7	View Selection: Click on this button to view the data you have specified for this analysis. The data can be viewed either in a tablular format or in a graphical summary.

If you have specified the input data format as summarized data, your data tab will look similar to the following dialog box: For summarized data, you can either specify a data file for entry or type the values right within the dialog box.

1	Summarized Data: For continuous data, you will need to specify the average, standard deviation, and number of data points. For discrete defects or defectives data, you will need to specify the count of the number or defects or defectives and the total number of samples tested.
2	Load from File: Click on Load from Table and specify the data columns that contain the data you are interested in. The software will fill up your summarized data table based on the data you have specified. Note that you can always edit and overwrite the values on the screen and these values will be used for analysis.

Click on OK to save the dialog box options and compute the analysis results.

Charts

If you click on the Charts button, you will see the following dialog box.

0	Pick Charts: Select the charts you would like to display for this analysis.
1	Title: The system will automatically pick a title for your chart. However, if you would like to override that with your own title you can specify a title for your chart here. Note that this input is optional.
2	Sub Title: The system will automatically pick a subtitle for your chart. However, if you would like to override that with your own subtitle you can specify a subtitle for your chart here. Note that this input is optional.
3	X Label: The system will automatically pick a label for the x-axis. However, if you would like to override that with your own label for the x-axis you can specify a different label here. Note that this input is optional.
4	Y Label: The system will automatically pick a label for the y-axis. However, if you would like to override that with your own label for the y-axis you can specify a different label here. Note that this input is optional.
5	X Axis: The system will automatically pick a scale for the x-axis. However, if you would like to override that with your values for the x-axis, you can specify them here. The format for this input is to specify the minimum, increment, and maximum values separated by a semi-colon. For example, if you specify 10;20 then the minimum x-axis scale is set at 10 and the maximum x-axis scale is set at 20. If you specify 10;2;20, then, in addition to minimum and maximum values, the x-axis increment is set at 2. Note that this input is currently disabled and you will not be able to change this setting.
6	Y Axis: The system will automatically pick a scale for the y-axis. However, if you would like to override that with your values for the y-axis, you can specify them here. The format for this input is to specify the minimum, increment, and maximum values separated by a semi-colon. For example, if you specify 10;20 then the minimum y-axis scale is set at 10 and the maximum y-axis scale is set at 20. If you specify 10;2;20, then, in addition to minimum and maximum values, the y-axis increment is set at 2. Note that this input is optional.
7	Horizontal Lines: If you want to add a few extra horizontal reference lines on top of your chart you can specify the values here. The format for this input is numeric values separated by semi-colon. For example, if you specify 12;15 then two horizontal lines are plotted at Y = 12 and Y = 15 respectively. Note that this input is optional.
8	Vertical Lines: If you want to add a few extra vertical reference lines on top of your chart you can specify the values here. The format for this input is numeric values separated by semi-colon. For example, if you specify 2;5 then two vertical lines are plotted at X = 2 and X = 5 respectively. Note that this input is optional.

Verify

If you click on the Verify button, the software will perform some checks on the data you have entered. A sample screenshot of the dialog box is shown in the figure below. The objective of this analysis as well as any checks that are performed is listed in this dialog box. For example, the software may check if you have correctly specified the input options and entered the required data on the worksheet. The results of the analysis checks are listed on the right. If the checks are passed, then they are shown as a green-colored checkmark. If the verification checks fail, then they are shown as a red-colored cross. If the verification checks result in a warning, they are shown in the orange color exclamation mark and finally, any checks that are required to be performed by the user are shown as blue info icons.

Outputs

Click on Compute Outputs to update the output calculations. A sample screenshot of the worksheet is shown below.

1-Sample t Test

The following analysis was for comparing the mean values for one set of data to an external standard using a 1-sample t-test.

1	Input Summary: Shows you a summary of the inputs used for this analysis. The key points to note are that we are entering raw data in a column for this analysis. We have one set of data that we are comparing to an external standard (specified as 10 in Ho) and the alternative hypothesis is not equal. In effect, we are checking if the mean of the proc time data is 10 or not. The methodology is set at Auto, so the software will pick the right method to use for this analysis.
2	Assumption Checks: Lists any assumptions checked for this analysis and the status of the checks. A check mark indicates that the assumptions were satisifed and a cross mark indicates that the assumptions were not met. In our case, the software checks if the data is normally distributed and it determined that yes the data is normal.
3	Hypothesis Test: Lists the null and alternative hypothesis that is being conducted based on the inputs specified in the dialog box. Always make sure that you have correctly specified the hypothesis for your test.
4	Analysis Results: Lists the results of the analysis including any test statistic, P values, and confidence levels. The T statistic is 0.43 which has a P value of 0.673. Since, the P value is greater than alpha, we can conclude that there is no reason to believe that the mean of the data is different from 10. The confidence interval specifies with 95% confidence that the mean values could vary between 9.42 and 10.88.
5	Conclusions: Lists the primary conclusion from this study that the mean is not different from 10.
6	Graphs: Shows the graphs generated for this analysis such as hypothesis P plot, confidence interval plot, box plots, etc. In our case, we have the confidence interval plot and the box plot of the data.

Mann-Whitney Test

The following is a case where we are comparing two data sets with each other using a non-parametric test such as the Mann-Whitney test.

1	Input Summary: Shows you a summary of the inputs used for this analysis. In our case, we are comparing two sets of data the processing time for the north and east zones. Note that we are interested in comparing the mean/median values and the methodology is set to Auto which means the software will pick the best method for this analysis.
2	Assumption Checks: Lists any assumptions checked for this analysis and the status of the checks. A check mark indicates that the assumptions were satisifed and a cross mark indicates that the assumptions were not met. In our case, the data is NOT normally distributed.
3	Hypothesis Test: Lists the null and alternative hypothesis that is being conducted based on the inputs specified in the dialog box. Note that for the non-parametric tests, the comparison is for the median values. Here we are comparing if the medians are the same or different.
4	Analysis Results: Lists the results of the analysis including any test statistic, P values, and confidence levels. The W statistic for the Mann-Whitney test was 919 which corresponds to a P value of less than 0.001.
5	Conclusions: Lists the primary conclusion from this study. In our case, since the P value is very low, we can safely conclude that the median values between the two populations are different.
6	Graphs: Shows the graphs generated for this analysis such as hypothesis P plot, box plots, etc. Note that only the box plot is shown between the two groups. The confidence interval plots are only shown for the parametric tests.

Levene Test

The following is a case where we are comparing the standard deviation values for four sets of data with each other using the Levene's test.

1	Input Summary: Shows you a summary of the inputs used for this analysis. Note that in this example, we are comparing the standard deviations of four sets of data (North, East, West, and South). The methodology is set at Auto so the system will pick the right methodology for this case.
2	Assumption Checks: Lists any assumptions checked for this analysis and the status of the checks. A check mark indicates that the assumptions were satisifed and a cross mark indicates that the assumptions were not met. In our case, at least one of the data sets is not normal.
3	Hypothesis Test: Lists the null and alternative hypothesis that is being conducted based on the inputs specified in the dialog box. The null hypothesis is that all the standard deviations are equal and the alternative hypothesis is that at least one of the standard deviations are different.
4	Analysis Results: Lists the results of the analysis including any test statistic, P values, and confidence levels. The W statistic is computed for the Levene's test and this is in turn used to determine the P value. The P value was determined to be less than 0.001.
5	Conclusions: Lists the primary conclusion from this study. From this study we can conclude that the standard deviations are different between the 4 sets of data.
6	Graphs: Shows the graphs generated for this analysis such as hypothesis P plot, box plots, etc. Since this is a non-parametric test, no confidence intervals are displayed. The box plot shows the differences in the variations between groups.

ANOVA Test

The following is a case where we are comparing the average values for four sets of data with each other using the ANOVA Test.

1	Input Summary: Shows you a summary of the inputs used for this analysis. Note that our data is in the Raw data format and we are comparing the mean values of the 4 sets of data namely North, East, West, South. We have specified the methodology as ANOVA and not let it be selected as Auto as we want this specific methodology to be used for this analysis.
2	Assumption Checks: Lists any assumptions checked for this analysis and the status of the checks. A check mark indicates that the assumptions were satisifed and a cross mark indicates that the assumptions were not met. In our case, at least one of the data sets is not normal. Ideally, we should not be using the ANOVA test for this case.
3	Hypothesis Test: Lists the null and alternative hypothesis that is being conducted based on the inputs specified in the dialog box. The null hypothesis states that all the means are the same and the alternative hypothesis states that at least one mean is different.
4	Analysis Results: Lists the results of the analysis including any test statistic, P values, and confidence levels. The ANOVA test results in a F value of 85.47 which translates to a P value of less than 0.001.
5	Conclusions: Lists the primary conclusion from this study. Since the P value is less than alpha, we can conclude that at least one of the means is different.
6	Graphs: Shows the graphs generated for this analysis such as hypothesis P plot, box plots, etc. The graphs section plots the confidence interval of the means for each group along with the box plot of the data.

Chi-Square Test

The following is a case where we are comparing the proportion values for three sets of data with each other using the Chi-Square Test.

1	Input Summary: Shows you a summary of the inputs used for this analysis. Note that we have specified summarized data for this study. There are 3 data sets that we want to compare and 2 categories (defectives and non defectives).
2	Assumption Checks: Lists any assumptions checked for this analysis and the status of the checks. A check mark indicates that the assumptions were satisifed and a cross mark indicates that the assumptions were not met. For this analysis, we just require that each data set has at least 5 data points which is satisfied.
3	Hypothesis Test: Lists the null and alternative hypothesis that is being conducted based on the inputs specified in the dialog box. Note that for the non-parametric tests, the comparison is for the median values. The null hypothesis is that all the proportions are equal and the alternative hypothesis is that at least one proportion is different.
4	Analysis Results: Lists the results of the analysis including any test statistic, P values, and confidence levels. The test statistic is the Chi-Square value which evaluates to 0.04. This translates to a P value of 0.98.
5	Conclusions: Lists the primary conclusion from this study. Since the P value is high (compared to alpha of 0.05), we can conclude that there is no reason to believe that the proportions are different.
6	Graphs: Shows the graphs generated for this analysis such as hypothesis P plot, box plots, etc. The graph shown in this case is a 3D bar chart. We can use the bar chart to identify where the Chi-square values contribution is coming from.

Notes

Here are a few pointers regarding this analysis:

• You can specify your tests in the Analysis Setup menu. However, it is recommended that you set the determination of the test to "Auto" so that Sigma Magic can pick the right test for you based on certain checks it performs on the data sets.
• The minimum number of data points depends on several factors such as the difference you are interested in detecting, the amount of type 1 and type 2 error you are willing to tolerate, and the amount of variation you have in your data. Use the sample size calculator to determine the minimum sample size required.

Examples

Following examples can be found in the Examples folder.

• Data is collected for the number of hours worked to process a claim. Determine if there is reason to believe that the processing time is greater than 10 hours (Compare Data Set 1.xlsm).
• Compare the number of defects generated by the department for the last year and determine if it is greater than the industry average of 2% (Compare Data Set 2.xlsm).
• Compare the performance of the north region with the east region for the meantime to repair data and determine if there are any differences between the two groups(Compare Data Set 3.xlsm).
• Compare the performance of all the regions of the country (north, south, east, west) for the mean repair time and determine if there are any differences between them (on average) (Compare Data Set 3.xlsm).
• Compare the performance of all the regions of the country (north, south, east, west) for the mean repair time and determine if there are any differences in variation between the data sets (Compare Data Set 3.xlsm).