Algorithms are created to simplify the calculations and operations man has to accomplish. Its main purpose is to aid humans in their day-to-day operations. But there’s a catch. Algorithms are constructed based on a set of scenarios. They are designed to behave in such a way if a certain qualification is met. But what happens if the input is not part of the scenarios the algorithm is designed to tackle? That’s when inductive bias enters the picture.
The best algorithms are the ones that are able to learn how to behave over time. Even during the times when the input is not part of their program, they need to be able to adapt to responding to it. To help these algorithms, a set of assumptions are made that will guide them as to how to behave in the event they are faced with scenarios they have yet to encounter.
Most inductive biases are based on logic wherein a logical formula is used as a guide to determine the true nature of the input and to either validate or negate the hypothesis of the algorithm. The hypothesis in question is the proposition of behavior posed by the algorithm.
There are different types of inductive biases available. Minimum cross-validation error is perhaps the most popular of the bunch. It narrows the hypotheses that are derived and chooses the one that has the lowest cross-validation error. Minimum length of description is another wherein the optimal hypothesis to be chosen is the one that has the shortest length in terms of description. Why? It works on the belief that simplicity is key. The same goes for minimum features. Maximum conditional independence, on the other hand, utilizes the Bayesian framework while the nearest neighbors operates on the assumption that the nearer the scenario is to the one that is known by the algorithm, the higher the probability that both belong to the same class and should thus be treated in the same manner.
The Wonderful World of Algorithms 