Help Manual


Sigma Magic Help Version 17

Pre Process Data


Sometimes, we need to pre-process the data before we can use the data. For example, the data may not be in the format required for the specific analytical tool. Even if the data is in the right format, some analytical tools can handle missing data, while others cannot handle any missing input data. In these cases, we will need to pre-process the data before we can use them. There are three main operations we can perform with this tool. The first handles missing data, the second centers and scales the data, and the third converts text data to numeric data.

The figure below shows examples of data pre-processing. The first data set shows data with missing values. There are several methods to handle missing data, and the second table shows the missing data replaced with the best estimate (the mean value of the column). The table in the center shows data that is centered and scaled. By centering, we subtract the mean value from each data point so that the average value is 0 (it is centered), and scaling the data ensures that its overall standard deviation is 1. Some analysis algorithms work better when the data is centered and scaled. Finally, the table on the right shows the transformation of a text column into numeric columns. You can use this tool to perform all of these types of operations. Pre-Process Data


First, add this tool to your workbook by clicking on Pre Process Data within the Analytics menu on the main menu bar. Click on Analysis Setup to open the menu options for this tool. The Analysis Setup has four tabs, as described below.


The Analysis Setup menu options are shown if you click on the Analysis Setup button on the main menu bar. A sample screenshot of the menu is shown below: Pre-Process Inputs 1
Analysis Tabs: There are three tabs for this analysis.
SetupClick on this button to open the Analysis Setup options.
DataClick on this button to specify the data for this analysis.
VerifyClick on this button to check if you have correctly specified all the inputs for this analysis.
Missing Values: Specify how you want to handle missing values in your data. Here are the available options. Note that it is a radio button, and you can only pick one option to handle your missing values.
Ignore missing valuesThis will leave your missing values as-is without taking any action. Your output will also contain these missing values.
Delete missing valuesThe software will delete any missing values from your records. Note that your record's size will get smaller after this operation.
Replace with central valueThe software will replace the missing values with the most likely value. This would be the mean of the rest of the data points in this column for continuous data. This could be the mode of the remaining non-missing records for attribute data.
Replace with imputed valueThe software will try to determine the best option for these missing values using the MICE algorithm. For example, continuous data may be tested by developing a regression model and then using the best-fit approach to estimate the missing values.
Numeric & Factors Columns: You can further process your input data using the following options. Note that you have a checkbox here, so you can perform multiple operations on your data depending on what you pick.
Center the columns Some algorithms are sensitive to the scale of the data and require that data be scaled for it to work properly. If the column contains text data, the entire column is determined to be a text column, and the centering and scaling operation is not performed on that column. The following analysis only applies to numeric columns. The software will calculate the average value for that column and then subtract each value from the column average.
Scale the columnsThe software will divide the column with its standard deviation to scale the column to have a unit standard deviation. Note that this operation only applies to numeric columns.
Convert to numericUse this option to convert text factors to numeric factors. Note that two output numeric columns are created if a column contains only two distinct values. If a column contains three distinct values, then three output numeric columns are created.
Maintain full rankIf this checkbox is selected, then the number of output columns is one less than the number of factors in that column. Hence, we avoid the issue of singularity.
Additional Options: This field is optional. By typing it here, you can specify any additional options for the R software program.
Help Button: Click on the Help Button to view the help documentation for this tool.
Cancel Button: Click on the Cancel Button to discard your changes and exit this menu.
OK Button: Click on the OK Button to save your changes and try to execute the program. Note that you must specify the required data to complete the analysis and generate outputs. If there are any missing data, the software will remind you to specify the data and click on Compute Outputs to generate analysis results.


You will see the following dialog box if you click the Data button. Here, you can specify the data required for this analysis. Data
Search Data: The available data displays all the columns of data that are available for analysis. You can use the search bar to filter this list and speed up the search for the right data for analysis. Enter a few characters in the search field, and the software will filter and display the filtered data in the Available Data box.
Available Data: The available data box contains the list of data available for analysis. If your workbook has no data in tabular format, this box will display "No Data Found." The information displayed in this box includes the row number, whether the data is Numeric (N) or Text (T), and the name of the column variable. Note that the software displays data from all the tables in the current workbook. Even though data within the same table have unique column names, columns across different tables can have similar names. Hence, it is crucial that you not only specify the column name but also the table name.
Add or View Data: Click on this button to add more data to your workbook for analysis or to view more details about the data listed in the available data box. When you click on this button, it opens up the Data Editor dialog box, where you can import more data into your workbook or switch from the list view to a table view to see the individual data values for each column.
Required Data: The code for the required data specifies what data can be specified for that box. An example code is N: 2-4. If the code starts with an N, you must select only numeric columns. If the code begins with a T, you can select numeric and text columns. The numbers to the right of the colon specify the min-max values. For example, if the min-max values are 2-4, you must select a minimum of 2 columns of data and a maximum of 4 columns in this box. If the minimum value is 0, then no data is required to be specified for this box.
Select Button: Click on this button to select the data for analysis. Any data you choose for the analysis is moved to the right. To select a column, click on the columns in the Available Databox to highlight them and then click on the Select Button. A second method to choose the data is to double-click on the columns in the list of Available Data. Finally, you can drag and drop the columns you are interested in by holding down the select columns using your left mouse key and dragging and dropping them in one of the boxes on the right.
Selected Data: The list box header will be displayed in black if the right number of data columns is specified. If sufficient data has not been specified, then the list box header will be displayed in red color. Note that you can double-click on any of the columns in this box to remove them from the box.
View Selection: Click on this button to view the data specified for this analysis. The data can be viewed in a tabular format or a graphical summary of the selected data.


If you click the Verify button, the software will perform some checks on the data you entered. A sample screenshot of the data is shown in this figure. Pre-Process Inputs 4
Verify Checks: The software checks if you have correctly specified the input options and if you have specified the data correctly for analysis.
Check Status: The results of the analysis checks are listed here. If the checks are passed, they are shown as a green checkmark. If the verification checks fail, they are shown as a red cross. If the verification checks result in a warning, they are shown in the orange exclamation mark. Finally, any checks that are required to be performed by the user are shown as blue info icons.
Overall Status: The overall status of all the checks for the given analysis is shown here. The overall status check shows a green thumps-up sign if everything is okay and a red thumps-down sign if any checks have not passed. Note that you cannot proceed with generating analysis results for some analyses if the overall status is not okay.


Click on Compute Outputs to pre-process the data and enter the processed data results into the output columns. A sample screenshot of the worksheet is shown below. In this analysis, missing data values were imputed using the MICE package. Outputs Check the outputs to ensure the transformation is what you expect; if not, make appropriate changes and click on Compute Outputs again. There are no analysis outputs or graphs for this analysis. The data entered in the input columns are reformatted and displayed in the outputs area.


Here are a few notes regarding this analysis:
  • The maximum number of variables you can manipulate using this analysis is 30. If you have more than 30 variables, you must do these multiple times with different data sets.
  • If you do not want to transform all the factor columns to numeric or you only want to scale some of the columns but not others, you will have to create separate worksheets for each, transform the worksheets that you desire, and re-assemble the data back to make a complete set.


The following examples are in the software's Examples folder.
  • Pre-process the data to remove missing values (Pre Process 1.xlsx).
  • Pre-process the data to center and scale the numeric values (Pre Process 2.xlsx).
  • Pre-process the data to convert all factors to numeric (Pre Process 3.xlsx).


For more information on this topic, please refer to the following articles. Do note that if any external links are mentioned below, they are for reference purposes only.

© Rapid Sigma Solutions LLP. All rights reserved.