Using Weka Experimenter Environment

1. Open Weka – from the front – rather than double clicking on a data file

2. Click on the Experimenter button

3. Set up an experiment:

a. Make sure the “Setup” tab is selected

b. Click on “New” button

c. Make sure “Results Destination” is “ARFF file”

d. Click on the “Browse” button 

e. Use the Save dialog box to navigate to where the results file should be saved

f. Make sure “Experiment Type” is “Cross-validation”

g. Make sure “Number of folds” is 10

h. Make sure “Classification” is selected if doing nominal predictions and “Regression” is selected if doing numeric predictions

i. Under “Iteration Control”, “Number of repetitions” should probably be 10

j. Choose “Data sets first” to run looping through data sets inside looping through algorithms, or “Algorithms first” to loop through algorithms inside looping through datasets.

(Data sets first follows the following pseudocode:

                For each algorithm

                                For each dataset

                                                For each repetition

                                                                For each fold

)

k. Under “Datasets”, click “Add new …” to add datasets to the experiment

l. Use the “Open” dialog box to navigate to and specify the dataset to include

m. Repeat two previous steps for however many data sets are desired

n. You can change your mind and delete datasets if desired

o. If may move things to another machine, may choose “Use relative paths”

p. Under “Algorithms”, click “Add new …” to add algorithms to the experiment

q. Choose algorithm, and any desired options, as when classifying in the “Explorer” environment

i. The first algorithm added will be the “baseline” against which other algorithms will be compared.

r. Repeat two previous steps for however many algorithms are desired (different options are considered a different algorithm)(“Edit Selected” button allows changing

4. Save the experiment set up into a configuration file so that it can be repeated

a. Click on the “Save” button near the top

b. Use the Save dialog box to navigate to where the configuration file should be saved

5. Run the experiment using the configuration file (can come back in and “Open” if did the set up another day and saved the configuration file)

a. Select the “Run” tab

b. Click on the “Start” button

c. The experiment could take fairly long

6. Analyze the results of the experiment

a. Select the “Analyze” tab

b. Click on the “File” button

c. Use the “Open” dialog box to navigate to where the results arff file was saved

d. Click “Perform test” button

e. Compare results across algorithms and datasets. Look for v or * indicating statistically significant differences in results (v indicates that the algorithm is statistically significantly BETTER than the baseline algorithm. * indicates that the algorithm is statistically significantly WORSE than the baseline algorithm)