Open Source Computer Vision Library https://opencv.org/
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

257 lines
13 KiB

Support Vector Machines
=======================
.. highlight:: cpp
Originally, support vector machines (SVM) was a technique for building an optimal binary (2-class) classifier. Later the technique was extended to regression and clustering problems. SVM is a partial case of kernel-based methods. It maps feature vectors into a higher-dimensional space using a kernel function and builds an optimal linear discriminating function in this space or an optimal hyper-plane that fits into the training data. In case of SVM, the kernel is not defined explicitly. Instead, a distance between any 2 points in the hyper-space needs to be defined.
The solution is optimal, which means that the margin between the separating hyper-plane and the nearest feature vectors from both classes (in case of 2-class classifier) is maximal. The feature vectors that are the closest to the hyper-plane are called *support vectors*, which means that the position of other vectors does not affect the hyper-plane (the decision function).
SVM implementation in OpenCV is based on [LibSVM]_.
.. [Burges98] C. Burges. *A tutorial on support vector machines for pattern recognition*, Knowledge Discovery and Data Mining 2(2), 1998 (available online at http://citeseer.ist.psu.edu/burges98tutorial.html)
.. [LibSVM] C.-C. Chang and C.-J. Lin. *LIBSVM: a library for support vector machines*, ACM Transactions on Intelligent Systems and Technology, 2:27:1--27:27, 2011. (http://www.csie.ntu.edu.tw/~cjlin/papers/libsvm.pdf)
11 years ago
ParamGrid
-----------
11 years ago
.. ocv:class:: ParamGrid
The structure represents the logarithmic grid range of statmodel parameters. It is used for optimizing statmodel accuracy by varying model parameters, the accuracy estimate being computed by cross-validation.
11 years ago
.. ocv:member:: double ParamGrid::minVal
Minimum value of the statmodel parameter.
11 years ago
.. ocv:member:: double ParamGrid::maxVal
Maximum value of the statmodel parameter.
11 years ago
.. ocv:member:: double ParamGrid::logStep
Logarithmic step for iterating the statmodel parameter.
The grid determines the following iteration sequence of the statmodel parameter values:
.. math::
11 years ago
(minVal, minVal*step, minVal*{step}^2, \dots, minVal*{logStep}^n),
where :math:`n` is the maximal index satisfying
.. math::
11 years ago
\texttt{minVal} * \texttt{logStep} ^n < \texttt{maxVal}
11 years ago
The grid is logarithmic, so ``logStep`` must always be greater then 1.
11 years ago
ParamGrid::ParamGrid
------------------------
The constructors.
11 years ago
.. ocv:function:: ParamGrid::ParamGrid()
11 years ago
.. ocv:function:: ParamGrid::ParamGrid( double minVal, double maxVal, double logStep )
The full constructor initializes corresponding members. The default constructor creates a dummy grid:
::
11 years ago
ParamGrid::ParamGrid()
{
11 years ago
minVal = maxVal = 0;
logStep = 1;
}
11 years ago
SVM::Params
-----------
11 years ago
.. ocv:class:: SVM::Params
SVM training parameters.
11 years ago
The structure must be initialized and passed to the training method of :ocv:class:`SVM`.
11 years ago
SVM::Params::Params
------------------------
11 years ago
The constructors
11 years ago
.. ocv:function:: SVM::Params::Params()
11 years ago
.. ocv:function:: SVM::Params::Params( int svmType, int kernelType, double degree, double gamma, double coef0, double Cvalue, double nu, double p, const Mat& classWeights, TermCriteria termCrit )
11 years ago
:param svmType: Type of a SVM formulation. Possible values are:
11 years ago
* **SVM::C_SVC** C-Support Vector Classification. ``n``-class classification (``n`` :math:`\geq` 2), allows imperfect separation of classes with penalty multiplier ``C`` for outliers.
11 years ago
* **SVM::NU_SVC** :math:`\nu`-Support Vector Classification. ``n``-class classification with possible imperfect separation. Parameter :math:`\nu` (in the range 0..1, the larger the value, the smoother the decision boundary) is used instead of ``C``.
11 years ago
* **SVM::ONE_CLASS** Distribution Estimation (One-class SVM). All the training data are from the same class, SVM builds a boundary that separates the class from the rest of the feature space.
11 years ago
* **SVM::EPS_SVR** :math:`\epsilon`-Support Vector Regression. The distance between feature vectors from the training set and the fitting hyper-plane must be less than ``p``. For outliers the penalty multiplier ``C`` is used.
11 years ago
* **SVM::NU_SVR** :math:`\nu`-Support Vector Regression. :math:`\nu` is used instead of ``p``.
See [LibSVM]_ for details.
11 years ago
:param kernelType: Type of a SVM kernel. Possible values are:
11 years ago
* **SVM::LINEAR** Linear kernel. No mapping is done, linear discrimination (or regression) is done in the original feature space. It is the fastest option. :math:`K(x_i, x_j) = x_i^T x_j`.
11 years ago
* **SVM::POLY** Polynomial kernel: :math:`K(x_i, x_j) = (\gamma x_i^T x_j + coef0)^{degree}, \gamma > 0`.
11 years ago
* **SVM::RBF** Radial basis function (RBF), a good choice in most cases. :math:`K(x_i, x_j) = e^{-\gamma ||x_i - x_j||^2}, \gamma > 0`.
11 years ago
* **SVM::SIGMOID** Sigmoid kernel: :math:`K(x_i, x_j) = \tanh(\gamma x_i^T x_j + coef0)`.
11 years ago
* **SVM::CHI2** Exponential Chi2 kernel, similar to the RBF kernel: :math:`K(x_i, x_j) = e^{-\gamma \chi^2(x_i,x_j)}, \chi^2(x_i,x_j) = (x_i-x_j)^2/(x_i+x_j), \gamma > 0`.
11 years ago
* **SVM::INTER** Histogram intersection kernel. A fast kernel. :math:`K(x_i, x_j) = min(x_i,x_j)`.
:param degree: Parameter ``degree`` of a kernel function (POLY).
:param gamma: Parameter :math:`\gamma` of a kernel function (POLY / RBF / SIGMOID / CHI2).
:param coef0: Parameter ``coef0`` of a kernel function (POLY / SIGMOID).
:param Cvalue: Parameter ``C`` of a SVM optimization problem (C_SVC / EPS_SVR / NU_SVR).
:param nu: Parameter :math:`\nu` of a SVM optimization problem (NU_SVC / ONE_CLASS / NU_SVR).
:param p: Parameter :math:`\epsilon` of a SVM optimization problem (EPS_SVR).
11 years ago
:param classWeights: Optional weights in the C_SVC problem , assigned to particular classes. They are multiplied by ``C`` so the parameter ``C`` of class ``#i`` becomes ``classWeights(i) * C``. Thus these weights affect the misclassification penalty for different classes. The larger weight, the larger penalty on misclassification of data from the corresponding class.
11 years ago
:param termCrit: Termination criteria of the iterative SVM training procedure which solves a partial case of constrained quadratic optimization problem. You can specify tolerance and/or the maximum number of iterations.
The default constructor initialize the structure with following values:
::
11 years ago
SVMParams::SVMParams() :
svmType(SVM::C_SVC), kernelType(SVM::RBF), degree(0),
gamma(1), coef0(0), C(1), nu(0), p(0), classWeights(0)
{
11 years ago
termCrit = TermCriteria( TermCriteria::MAX_ITER+TermCriteria::EPS, 1000, FLT_EPSILON );
}
A comparison of different kernels on the following 2D test case with four classes. Four C_SVC SVMs have been trained (one against rest) with auto_train. Evaluation on three different kernels (CHI2, INTER, RBF). The color depicts the class with max score. Bright means max-score > 0, dark means max-score < 0.
.. image:: pics/SVM_Comparison.png
11 years ago
SVM
-----
11 years ago
.. ocv:class:: SVM : public StatModel
Support Vector Machines.
.. note::
* (Python) An example of digit recognition using SVM can be found at opencv_source/samples/python2/digits.py
* (Python) An example of grid search digit recognition using SVM can be found at opencv_source/samples/python2/digits_adjust.py
* (Python) An example of video digit recognition using SVM can be found at opencv_source/samples/python2/digits_video.py
11 years ago
SVM::create
------------
11 years ago
Creates empty model
11 years ago
.. ocv:function:: Ptr<SVM> SVM::create(const Params& p=Params(), const Ptr<Kernel>& customKernel=Ptr<Kernel>())
11 years ago
:param p: SVM parameters
:param customKernel: the optional custom kernel to use. It must implement ``SVM::Kernel`` interface.
11 years ago
Use ``StatModel::train`` to train the model, ``StatModel::train<RTrees>(traindata, params)`` to create and train the model, ``StatModel::load<RTrees>(filename)`` to load the pre-trained model. Since SVM has several parameters, you may want to find the best parameters for your problem. It can be done with ``SVM::trainAuto``.
11 years ago
SVM::trainAuto
-----------------
Trains an SVM with optimal parameters.
11 years ago
.. ocv:function:: bool SVM::trainAuto( const Ptr<TrainData>& data, int kFold = 10, ParamGrid Cgrid = SVM::getDefaultGrid(SVM::C), ParamGrid gammaGrid = SVM::getDefaultGrid(SVM::GAMMA), ParamGrid pGrid = SVM::getDefaultGrid(SVM::P), ParamGrid nuGrid = SVM::getDefaultGrid(SVM::NU), ParamGrid coeffGrid = SVM::getDefaultGrid(SVM::COEF), ParamGrid degreeGrid = SVM::getDefaultGrid(SVM::DEGREE), bool balanced=false)
11 years ago
:param data: the training data that can be constructed using ``TrainData::create`` or ``TrainData::loadFromCSV``.
11 years ago
:param kFold: Cross-validation parameter. The training set is divided into ``kFold`` subsets. One subset is used to test the model, the others form the train set. So, the SVM algorithm is executed ``kFold`` times.
:param \*Grid: Iteration grid for the corresponding SVM parameter.
:param balanced: If ``true`` and the problem is 2-class classification then the method creates more balanced cross-validation subsets that is proportions between classes in subsets are close to such proportion in the whole train dataset.
The method trains the SVM model automatically by choosing the optimal
parameters ``C``, ``gamma``, ``p``, ``nu``, ``coef0``, ``degree`` from
11 years ago
:ocv:class:`SVMParams`. Parameters are considered optimal
when the cross-validation estimate of the test set error
is minimal.
11 years ago
If there is no need to optimize a parameter, the corresponding grid step should be set to any value less than or equal to 1. For example, to avoid optimization in ``gamma``, set ``gammaGrid.step = 0``, ``gammaGrid.minVal``, ``gamma_grid.maxVal`` as arbitrary numbers. In this case, the value ``params.gamma`` is taken for ``gamma``.
And, finally, if the optimization in a parameter is required but
11 years ago
the corresponding grid is unknown, you may call the function :ocv:func:`SVM::getDefaulltGrid`. To generate a grid, for example, for ``gamma``, call ``SVM::getDefaulltGrid(SVM::GAMMA)``.
This function works for the classification
11 years ago
(``params.svmType=SVM::C_SVC`` or ``params.svmType=SVM::NU_SVC``)
as well as for the regression
11 years ago
(``params.svmType=SVM::EPS_SVR`` or ``params.svmType=SVM::NU_SVR``). If ``params.svmType=SVM::ONE_CLASS``, no optimization is made and the usual SVM with parameters specified in ``params`` is executed.
11 years ago
SVM::getDefaulltGrid
-----------------------
Generates a grid for SVM parameters.
11 years ago
.. ocv:function:: ParamGrid SVM::getDefaulltGrid( int param_id )
:param param_id: SVM parameters IDs that must be one of the following:
11 years ago
* **SVM::C**
11 years ago
* **SVM::GAMMA**
11 years ago
* **SVM::P**
11 years ago
* **SVM::NU**
11 years ago
* **SVM::COEF**
11 years ago
* **SVM::DEGREE**
The grid is generated for the parameter with this ID.
11 years ago
The function generates a grid for the specified parameter of the SVM algorithm. The grid may be passed to the function :ocv:func:`SVM::trainAuto`.
11 years ago
SVM::getParams
-----------------
Returns the current SVM parameters.
11 years ago
.. ocv:function:: SVM::Params SVM::getParams() const
11 years ago
This function may be used to get the optimal parameters obtained while automatically training :ocv:func:`SVM::train_auto`.
11 years ago
SVM::getSupportVectors
--------------------------
11 years ago
Retrieves all the support vectors
11 years ago
.. ocv:function:: Mat SVM::getSupportVectors() const
11 years ago
The method returns all the support vector as floating-point matrix, where support vectors are stored as matrix rows.
11 years ago
SVM::getDecisionFunction
--------------------------
Retrieves the decision function
.. ocv:function:: double SVM::getDecisionFunction(int i, OutputArray alpha, OutputArray svidx) const
:param i: the index of the decision function. If the problem solved is regression, 1-class or 2-class classification, then there will be just one decision function and the index should always be 0. Otherwise, in the case of N-class classification, there will be N*(N-1)/2 decision functions.
:param alpha: the optional output vector for weights, corresponding to different support vectors. In the case of linear SVM all the alpha's will be 1's.
:param svidx: the optional output vector of indices of support vectors within the matrix of support vectors (which can be retrieved by ``SVM::getSupportVectors``). In the case of linear SVM each decision function consists of a single "compressed" support vector.
The method returns ``rho`` parameter of the decision function, a scalar subtracted from the weighted sum of kernel responses.
Prediction with SVM
--------------------
11 years ago
StatModel::predict(samples, results, flags) should be used. Pass ``flags=StatModel::RAW_OUTPUT`` to get the raw response from SVM (in the case of regression, 1-class or 2-class classification problem).