Relevance vector machine explained

In mathematics, a Relevance Vector Machine (RVM) is a machine learning technique that uses Bayesian inference to obtain parsimonious solutions for regression and probabilistic classification.^[1] The RVM has an identical functional form to the support vector machine, but provides probabilistic classification.

It is actually equivalent to a Gaussian process model with covariance function:

k( x , x' ) = \sum ^{N} _{j =1} (1)/(\alpha _{j}) \varphi( x , x _{j)\varphi(x',x} _{j)} where \varphi is the kernel function (usually Gaussian), \alpha _{j} are the variances of the prior on the weight vectorw \sim N( 0 ,\alpha ^{-1} I) , and x _{1,\ldots,x} _{N} are the input vectors of the training set.^[2]

Compared to that of support vector machines (SVM), the Bayesian formulation of the RVM avoids the set of free parameters of the SVM (that usually require cross-validation-based post-optimizations). However RVMs use an expectation maximization (EM)-like learning method and are therefore at risk of local minima. This is unlike the standard sequential minimal optimization (SMO)-based algorithms employed by SVMs, which are guaranteed to find a global optimum (of the convex problem).

The relevance vector machine was patented in the United States by Microsoft (patent expired September 4, 2019).^[3]

See also

• Kernel trick
• Platt scaling: turns an SVM into a probability model

Software

• dlib C++ Library
• The Kernel-Machine Library
• rvmbinary: R package for binary classification
• scikit-rvm
• fast-scikit-rvm, rvm tutorial

External links

• Tipping's webpage on Sparse Bayesian Models and the RVM
• A Tutorial on RVM by Tristan Fletcher
• Applied tutorial on RVM
• Comparison of RVM and SVM

Notes and References

1. Tipping . Michael E. . Sparse Bayesian Learning and the Relevance Vector Machine . 2001 . . 1 . 211 - 244 .
2. Ph.D.. Candela. Joaquin Quiñonero. 2004. Learning with Uncertainty - Gaussian Processes and Relevance Vector Machines. Technical University of Denmark . Sparse Probabilistic Linear Models and the RVM. April 22, 2016.
3. US. 6633857. Relevance vector machine. Michael E. Tipping.