Error tolerance (PAC learning) explained

In PAC learning, error tolerance refers to the ability of an algorithm to learn when the examples received have been corrupted in some way. In fact, this is a very common and important issue since in many applications it is not possible to access noise-free data. Noise can interfere with the learning process at different levels: the algorithm may receive data that have been occasionally mislabeled, or the inputs may have some false information, or the classification of the examples may have been maliciously adulterated.

Notation and the Valiant learning model

In the following, let

be our

-dimensional input space. Let

l{H}

be a class of functions that we wish to use in order to learn a

\{0,1\}

-valued target function

defined over

. Let

l{D}

be the distribution of the inputs over

. The goal of a learning algorithm

l{A}

is to choose the best function

h\inl{H}

such that it minimizes

error(h)=P_x}(h(x) ≠ f(x))

. Let us suppose we have a function

size(f)

that can measure the complexity of

. Let

Oracle(x)

be an oracle that, whenever called, returns an example

and its correct label

f(x)

When no noise corrupts the data, we can define learning in the Valiant setting:^[1] ^[2]

Definition:We say that

is efficiently learnable using

l{H}

in the Valiant setting if there exists a learning algorithm

l{A}

that has access to

Oracle(x)

and a polynomial

p( ⋅ , ⋅ , ⋅ , ⋅ )

such that for any

0<\varepsilon\leq1

and

0<\delta\leq1

it outputs, in a number of calls to the oracle bounded by

p\left(	1	,
	\varepsilon

	1
	\delta

,n,size(f)\right)

, a function

h\inl{H}

that satisfies with probability at least

1-\delta

the condition

error(h)\leq\varepsilon

In the following we will define learnability of

when data have suffered some modification.^[3] ^[4] ^[5]

Classification noise

In the classification noise model^[6] a noise rate

0\leqη<

	1
	2

is introduced. Then, instead of

Oracle(x)

that returns always the correct label of example

, algorithm

l{A}

can only call a faulty oracle

Oracle(x,η)

that will flip the label of

with probability

. As in the Valiant case, the goal of a learning algorithm

l{A}

is to choose the best function

h\inl{H}

such that it minimizes

error(h)=P_x}(h(x) ≠ f(x))

. In applications it is difficult to have access to the real value of

, but we assume we have access to its upperbound

η_B

.^[7] Note that if we allow the noise rate to be

1/2

, then learning becomes impossible in any amount of computation time, because every label conveys no information about the target function.

Definition:We say that

is efficiently learnable using

l{H}

in the classification noise model if there exists a learning algorithm

l{A}

that has access to

Oracle(x,η)

and a polynomial

p( ⋅ , ⋅ , ⋅ , ⋅ )

such that for any

0\leqη\leq

	1
	2

0\leq\varepsilon\leq1

and

0\leq\delta\leq1

it outputs, in a number of calls to the oracle bounded by

p\left(	1
	1-2η_B

	1	,
	\varepsilon

	1
	\delta

,n,size(f)\right)

, a function

h\inl{H}

that satisfies with probability at least

1-\delta

the condition

error(h)\leq\varepsilon

Statistical query learning

Statistical Query Learning^[8] is a kind of active learning problem in which the learning algorithm

l{A}

can decide if to request information about the likelihood

P_f(x)

that a function

correctly labels example

, and receives an answer accurate within a tolerance

\alpha

. Formally, whenever the learning algorithm

l{A}

calls the oracle

Oracle(x,\alpha)

, it receives as feedback probability

Q_f(x)

, such that

Q_f(x)-\alpha\leqP_f(x)\leqQ_f(x)+\alpha

Definition:We say that

is efficiently learnable using

l{H}

in the statistical query learning model if there exists a learning algorithm

l{A}

that has access to

Oracle(x,\alpha)

and polynomials

p( ⋅ , ⋅ , ⋅ )

q( ⋅ , ⋅ , ⋅ )

, and

r( ⋅ , ⋅ , ⋅ )

such that for any

0<\varepsilon\leq1

the following hold:

Oracle(x,\alpha)

can evaluate

P_f(x)

in time

q\left(	1
	\varepsilon

,n,size(f)\right)

;

	1
	\alpha

is bounded by

r\left(	1
	\varepsilon

,n,size(f)\right)

l{A}

outputs a model

such that

err(h)<\varepsilon

, in a number of calls to the oracle bounded by

p\left(	1
	\varepsilon

,n,size(f)\right)

Note that the confidence parameter

\delta

does not appear in the definition of learning. This is because the main purpose of

\delta

is to allow the learning algorithm a small probability of failure due to an unrepresentative sample. Since now

Oracle(x,\alpha)

always guarantees to meet the approximation criterion

Q_f(x)-\alpha\leqP_f(x)\leqQ_f(x)+\alpha

, the failure probability is no longer needed.

The statistical query model is strictly weaker than the PAC model: any efficiently SQ-learnable class is efficiently PAC learnable in the presence of classification noise, but there exist efficient PAC-learnable problems such as parity that are not efficiently SQ-learnable.

Malicious classification

In the malicious classification model^[9] an adversary generates errors to foil the learning algorithm. This setting describes situations of error burst, which may occur when for a limited time transmission equipment malfunctions repeatedly. Formally, algorithm

l{A}

calls an oracle

Oracle(x,\beta)

that returns a correctly labeled example

drawn, as usual, from distribution

l{D}

over the input space with probability

1-\beta

, but it returns with probability

\beta

an example drawn from a distribution that is not related to

l{D}

. Moreover, this maliciously chosen example may strategically selected by an adversary who has knowledge of

\beta

l{D}

, or the current progress of the learning algorithm.

Definition:Given a bound

\beta_B<

	1
	2

for

0\leq\beta<

	1
	2

, we say that

is efficiently learnable using

l{H}

in the malicious classification model, if there exist a learning algorithm

l{A}

that has access to

Oracle(x,\beta)

and a polynomial

p( ⋅ , ⋅ , ⋅ , ⋅ , ⋅ )

such that for any

0<\varepsilon\leq1

0<\delta\leq1

it outputs, in a number of calls to the oracle bounded by

p\left(	1	,
	1/2-\beta_B

	1	,
	\varepsilon

	1
	\delta

,n,size(f)\right)

, a function

h\inl{H}

that satisfies with probability at least

1-\delta

the condition

error(h)\leq\varepsilon

Errors in the inputs: nonuniform random attribute noise

In the nonuniform random attribute noise^[10] ^[11] model the algorithm is learning a Boolean function, a malicious oracle

Oracle(x,\nu)

may flip each

-th bit of example

x=(x_1,x_2,\ldots,x_n)

independently with probability

\nu_i\leq\nu

This type of error can irreparably foil the algorithm, in fact the following theorem holds:

In the nonuniform random attribute noise setting, an algorithm

l{A}

can output a function

h\inl{H}

such that

error(h)<\varepsilon

only if

\nu<2\varepsilon

Notes and References

Valiant, L. G. (August 1985). Learning Disjunction of Conjunctions. In IJCAI (pp. 560–566).
Valiant, Leslie G. "A theory of the learnable." Communications of the ACM 27.11 (1984): 1134–1142.
Laird, P. D. (1988). Learning from good and bad data. Kluwer Academic Publishers.
Kearns, Michael. "Efficient noise-tolerant learning from statistical queries ." Journal of the ACM 45.6 (1998): 983–1006.
Brunk, Clifford A., and Michael J. Pazzani. "An investigation of noise-tolerant relational concept learning algorithms." Proceedings of the 8th International Workshop on Machine Learning. 1991.
Kearns, M. J., & Vazirani, U. V. (1994). An introduction to computational learning theory, chapter 5. MIT press.
Angluin, D., & Laird, P. (1988). Learning from noisy examples. Machine Learning, 2(4), 343–370.
Kearns, M. (1998). [www.cis.upenn.edu/~mkearns/papers/sq-journal.pdf Efficient noise-tolerant learning from statistical queries]. Journal of the ACM, 45(6), 983–1006.
Kearns, M., & Li, M. (1993). [www.cis.upenn.edu/~mkearns/papers/malicious.pdf Learning in the presence of malicious errors]. SIAM Journal on Computing, 22(4), 807–837.
[Sally Goldman|Goldman, S. A.]
Sloan, R. H. (1989). Computational learning theory: New models and algorithms (Doctoral dissertation, Massachusetts Institute of Technology).