Error Analysis

2014-05-03 18:39 | Source

Error Analysis

Choosing the most simplest way to be implemented quickly.
Never PREMATURE OPTIMISATION (famous advice in computer programming). Always build something simple.Let learning curve evidence direct us to what effort to do first.
From that, we can use the learning curve to decide if we add more complexity(parameters) or training set. In this way we can combat "gut feeling"
Alternatively, often people use error analysis. it is used to spot error pattern.
Even simple one still give vague answer about what to do decide whether it has high bias/ high variance

Here are the specific example
Observe the 100 emails errors(misclassify) from 500 training examples.
Based on that, we can do those two things mentioned.
From the first method(type of email) we know that for example our algorithm is mostly misclassify "steal passwords".
What features may be most helpful. When we cross-validate, we know that we are vulnerable in detecting punctuation. We can build the algorithm to analyse that rather spend time on others.
Simple and quick dirty helps us to make a better understanding on which hard or easy examples

We also want to have some single row number(numerical evaluation) that indicates some kind of accuracy
For example this is NLP problem to detect similar words using stemming software
Based on numerical evaluation, we know that using stemming or without stemming would increase our performance or not.
Or distinguish upper vs lower would increase/decrease our performance
There's a lot of implementation method or different version of learning algorithm, and it's good to have numerical evaluation to test which has the lowest error.

Use CV to choose whether we need stemming or not.
lots of new version of learning algorithms keep getting implemented every time and use error analysis whether the performance of our learning algorithms is increasing or not
Most of the time the simplest method is the most correct one
The fast result will give us insight about which specific problem need specific actions to handle, and better method to handle / generalize it
Don't afraid about being too quick or to dirty implementation. Once we got our first initialisation, we can then have a grip about which direction we should go into.
So once again, take a quick and dirty way to implement the algorithms. From the result , we will then spend some time wisely go which the direction of complexity of the learning algorithm for particular set of problems