Help Manual

Contents






Sigma Magic Help Version 15

Pre Process Data

Overview

Sometimes we need to pre-process the data before we can use the data. For example, the data may not be in the format required for the specific analytical tool. Even if the data is in the right format, some analytical tools can handle missing data, while others cannot handle any missing input data. In these cases, we will need to pre-process the data before we can use them. There are three main operations we can perform with this tool. The first is to handle missing data, the second is to center and scale the data, and the third is to convert any text data to numeric data.

The figure below shows examples of pre-processing of data. The first data set shows data with missing values. There are several methods to handle missing data, and the second table shows the missing data replaced with the best estimate (the mean value of the column). The table in the center shows data that is centered and scaled. By centering, we subtract the mean value from each of the data points, so that the average value is 0 (it is centered) and scaling the data makes sure that its overall standard deviation is 1. Some analysis algorithms work better when the data is centered and scaled. Finally, the table on the right shows the transformation of a text column into numeric columns. You can use this tool to perform all of these types of operations. Pre-Process Data

Inputs

First, add this tool to your workbook by clicking on Pre Process Data within the Analytics menu on the main menu bar. Click on Analysis Setup to open the menu options for this tool. The Analysis Setup has four tabs as described below.

Setup

If you click on the Analysis Setup button on the main menu bar, the Analysis Setup menu options are shown. A sample screenshot of the menu is shown below: Pre-Process Inputs 1
1
Analysis Tabs: There are 4 tabs for this analysis.
OptionDescription
SetupClick on this button to open the Analysis Setup options.
DataClick on this button to specify the data for this analysis.
ProgramClick on this button to view and/or edit the RScript source code for this analysis.
VerifyClick on this button to check if you have correctly specified all the inputs for this analysis.
2
Missing Values: Specify how you want to handle missing values in your data. Here are the available options. Note that it is a radio button, and you can only pick one option to handle your missing values.
OptionDescription
Ignore missing valuesThis will leave your missing values as-is without taking any action. Your output will also contain these missing values.
Delete missing valuesThe software will delete any missing values from your records. Note that your record's size will get smaller after this operation.
Replace with central valueThe software will replace the missing values with the most likely value. For continuous data, this would be the mean of the rest of the data points in this column. For attribute data, this could be the mode of the remaining non-missing records.
Replace with imputed valueThe software will try to determine the best option for these missing values using the MICE algorithm. For example, for continuous data, it may try to develop a regression model and then use the best-fit approach to estimate the missing values.
3
Numeric & Factors Columns: You can further process your input data using the following options. Note that you have a checkbox here, so you can perform multiple operations on your data depending on what you pick.
OptionDescription
Center the columns Some algorithms are sensitive to the scale of the data and require that data be scaled for it to work properly. If the column contains text data, then the entire column is determined to be a text column and the centering and scaling operation is not performed on that column. The following analysis only applies to numeric columns. The software will calculate the average value for that column and then subtract each value in that column with the column average.
Scale the columnsThe software will divide the column with its standard deviation in order to scale the column to have a unit standard deviation. Note that this operation only applies to numeric columns.
Convert to numericUse this option to convert text factors to numeric factors. Note that if a column contains only two distinct values, then 2 output numeric columns are created. If a column contains three distinct values, then 3 output numeric columns are created.
Maintain full rankIf this checkbox is selected, then the number of output columns is one less than the number of factors in that column. Hence, we avoid the issue of singularity.
4
Additional Options: This field is optional. You can specify any additional options for the R software program directly by typing it here.
5
Help Button: Click on the Help Button to view the help documentation for this tool.
6
Cancel Button: Click on the Cancel Button to discard your changes and exit this menu.
7
OK Button: Click on the OK Button to save your changes and try to execute the program. Note that you will need to specify the required data in order to complete the analysis and generate outputs. If there are any missing data, then the software will remind you to specify the data and click on Compute Outputs to generate analysis results.

Data

If you click on the Data button, you will see the following dialog box. Here you can specify the data required for this analysis. Data
1
Search Data: The available data displays all the columns of data that are available for analysis. You can use the search bar to filter this list and to speed up finding the right data to use for analysis. Enter a few characters in the search field and the software will filter and display the filtered data in the Available Data box.
2
Available Data: The available data box contains the list of data available for analysis. If your workbook does not have any data in tabular format, this box will display "No Data Found". The information displayed in this box includes the row number, whether the data is Numeric (N) or Text (T), and the name of the column variable. Note that the software displays data from all the tables in the current workbook. Even though data within the same table have unique column names, columns across different tables can have similar names. Hence, it is important that you not only specify the column name but also the table name.
3
Add or View Data: Click on this button either to add more data into your workbook for analysis or to view more details about the data listed in the available data box. When you click on this button, it opens up the Data Editor dialog box where you can import more data into your workbook, or you can switch from the list view to a table view to see the individual data values for each column.
4
Required Data: The code for the required data specifies what data can be specified for that box. An example code is N: 2-4. If the code starts with an N, then you will need to select only numeric columns. If the code starts with a T, then you can select both numeric and text columns. The numbers to the right of the colon specify the min-max values. For example, if the min-max values are 2-4, then you need to select a minimum of 2 columns of data and a maximum of 4 columns of data in this box. If the minimum value is 0, then no data is required to be specified for this box.
5
Select Button: Click on this button to select the data for analysis. Any data you select for the analysis is moved to the right. To select a column, click on the columns in the Available Databox to highlight them and then click on the Select Button. A second method to select the data is to double click on the columns in the list of Available Data. Finally, you can also drag and drop the columns you are interested in by holding down the select columns using your left mouse key and dragging and dropping them in one of the boxes on the right.
6
Selected Data: If the right amount of data columns has been specified, the list box header will be displayed in the black color. If sufficient data has not been specified, then the list box header will be displayed in the red color. Note that you can double-click on any of the columns in this box to remove them from the box.
7
View Selection: Click on this button to view the data you have specified for this analysis. The data can be viewed either in the tablular format or you can view a graphical summary of the data selected.

Program

If you click on the Program button, the software will display the program code - an example screenshot is shown in the figure below. Pre-Process Inputs 3
1
R Program: You can view the R program that will be executed here. This program is usually automatically generated from the options you have specified in the setup earlier. This is the program that will be executed by the R program to generate analysis outputs. If you like, you can edit this program.
2
Auto Mode: If the radio button is selected as Auto, then the software will automatically update this code based on any changes you make in the input dialog box. We recommend that you use this option to generate the R program so that all your input settings are used to generate analysis results.
3
Manual Mode: If you use the Manual option, then you will be allowed to edit the R program before the program is executed. Make sure that you specify a syntactically correct program; otherwise, the R program may report errors.

Verify

If you click on the Verify button, the software will perform some checks on the data you have entered. A sample screenshot of the data is shown in this figure. Pre-Process Inputs 4
1
Verify Checks: The objective of this analysis as well as any checks that are performed are listed in this dialog box. For example, the software may check if you have correctly specified the input options and if you have specified the data correctly for analysis.
2
Check Status: The results of the analysis checks are listed here. If the checks are passed, then they are shown as a green-colored checkmark. If the verification checks fail, then they are shown as a red-colored cross. If the verification checks result in a warning, they are shown in the orange color exclamation mark and finally, any checks that are required to be performed by the user are shown as blue info icons.

Outputs

Click on Compute Outputs to pre-process the data and enter the processed data results into the output columns. A sample screenshot of the worksheet is shown below. In this analysis, missing data values were imputed using the MICE package. Outputs Check the outputs to make sure the transformation is what you expect, if not make appropriate changes and click on Compute Outputs again. There are no analysis outputs or graphs for this analysis. The data entered in the input columns are reformatted and displayed in the outputs area.

Notes

Here are a few notes regarding this analysis:
  • The maximum number of variables you can manipulate using this analysis is 30. If you have more than 30 variables, you will have to do these multiple times with different sets of data.
  • If you do not want to transform all the factor columns to numeric or you only want to scale some of the columns but not others, you will have to create separate worksheets for each, transform the worksheets that you desire, and re-assemble the data back to make a complete set.

Examples

Following examples can be found in the Examples folder within the software.
  • Pre-process the data to remove missing values (Pre Process 1.xlsm).
  • Pre-process the data to center and scale the numeric values (Pre Process 2.xlsm).
  • Pre-process the data to convert all factors to numeric (Pre Process 3.xlsm).

References

For more information on this topic, please refer to the following articles. Do note that if any external links are mentioned below, they are for reference purposes only.