Chiebot-Mirror
/
opencv_contrib
mirror of https://github.com/opencv/opencv_contrib.git
8
0
Fork 0
Code Issues Projects Releases Wiki Activity

Merge pull request #1257 from kurnianggoro:facelandmark

Browse Source 
GSOC17 - Facemark API (#1257)

* Initial commit of facemark API

Initial structure of the facemark API and AAM header

* make training function as virtual

* Add: dataset parser

* Bug fix: clear the container before add points

* Add: AAM training - procrustes analysis

* Add AAM model

* Added training function for AAM

* Building bot fixes: remove training overload, explicit cast to float for atof

* + add dependency: imgcodecs

* Build bot fixes: add imgproc.hpp and type casting

* Building bot fix: type casting

* fixing the AAM training to match with Matlab version

fewer model parameters, change the image warp method, change the feature extraction method

* add: AAM fitting

added several functionalities for fitting

* fix warings

* Add: transformation for the initial fitting

* add sample file for aam implementation

* fix warning

* Add LFB Header

* loadTrainingData: Throw an error message if file not exist

* add: LBF prepare training data

* add: data augmentation

* change to double

* add: getMeanShape

* shuffling the dataset and parameters initialization

* add: initial structure of LBF class

* add: getDeltaShapes

Difference between the current shape and the desired shape

* add: random forest training

* generate lbf features

* global regression

* save training data

* fix the parameter initialization

* set the default parameters

* add: initial version of lbf sample

* update the current shape

* compute error

* add: prediction function

* fix some warnings

* fitting function

the result is mis-aligned, shuould be double checked

* add: fitting in the demo

* add dependencies

* Add: tutorial

* add: load model

* fixing training

* use user defined face detector

* Documents, tests, and samples

* Allow custom parameters

* Cleaning up

* Custom parameters for default detector, training, and get custom data

* AAM scales

* minor fixes , update the opencv_extra files

* change path to lbp cascade

* face: avoid memory leaks

* utilize the filestorage for the model, fixing some minor issues

* remove the liblinear dependency

* fix the aam test, avoiding to write any files

* use RNG and changes the test files
pull/1204/merge
kurnianggoro 7 years ago committed by Alexander Alekhin
parent e7955998e5
commit e85a802a90
17 changed files with 5316 additions and 2 deletions
Whitespace
Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL
Split View
Diff Options
Show Stats Download Patch File Download Diff File
  1. 6
      modules/face/include/opencv2/face.hpp
  2. 439
      modules/face/include/opencv2/face/facemark.hpp
  3. 160
      modules/face/include/opencv2/face/facemarkAAM.hpp
  4. 120
      modules/face/include/opencv2/face/facemarkLBF.hpp
  5. 318
      modules/face/samples/facemark_demo_aam.cpp
  6. 217
      modules/face/samples/facemark_demo_lbf.cpp
  7. 202
      modules/face/samples/facemark_lbf_fitting.cpp
  8. 237
      modules/face/src/facemark.cpp
  9. 1222
      modules/face/src/facemarkAAM.cpp
  10. 1446
      modules/face/src/facemarkLBF.cpp
  11. 61
      modules/face/test/test_facemark.cpp
  12. 145
      modules/face/test/test_facemark_aam.cpp
  13. 148
      modules/face/test/test_facemark_lbf.cpp
  14. 113
      modules/face/tutorials/facemark_aam/facemark_aam.markdown
  15. 273
      modules/face/tutorials/facemark_add_algorithm/facemark_add_algorithm.markdown
  16. 29
      modules/face/tutorials/facemark_tutorial.markdown
  17. 182
      modules/face/tutorials/facemark_usage/facemark_usage.markdown

6
modules/face/include/opencv2/face.hpp
Unescape View File

@ -40,7 +40,7 @@ the use of this software, even if advised of the possibility of such damage.
#define __OPENCV_FACE_HPP__
/**
@defgroup face Face Recognition
@defgroup face Face Analysis
- @ref face_changelog
- @ref tutorial_face_main
@ -374,5 +374,7 @@ protected:
}}
#include "opencv2/face/facerec.hpp"
#include "opencv2/face/facemark.hpp"
#include "opencv2/face/facemarkLBF.hpp"
#include "opencv2/face/facemarkAAM.hpp"
#endif

439
modules/face/include/opencv2/face/facemark.hpp
Unescape View File

@ -0,0 +1,439 @@
/*
By downloading, copying, installing or using the software you agree to this
license. If you do not agree to this license, do not download, install,
copy or use the software.
License Agreement
For Open Source Computer Vision Library
(3-clause BSD License)
Copyright (C) 2013, OpenCV Foundation, all rights reserved.
Third party copyrights are property of their respective owners.
Redistribution and use in source and binary forms, with or without modification,
are permitted provided that the following conditions are met:
* Redistributions of source code must retain the above copyright notice,
this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.
* Neither the names of the copyright holders nor the names of the contributors
may be used to endorse or promote products derived from this software
without specific prior written permission.
This software is provided by the copyright holders and contributors "as is" and
any express or implied warranties, including, but not limited to, the implied
warranties of merchantability and fitness for a particular purpose are
disclaimed. In no event shall copyright holders or contributors be liable for
any direct, indirect, incidental, special, exemplary, or consequential damages
(including, but not limited to, procurement of substitute goods or services;
loss of use, data, or profits; or business interruption) however caused
and on any theory of liability, whether in contract, strict liability,
or tort (including negligence or otherwise) arising in any way out of
the use of this software, even if advised of the possibility of such damage.
This file was part of GSoC Project: Facemark API for OpenCV
Final report: https://gist.github.com/kurnianggoro/74de9121e122ad0bd825176751d47ecc
Student: Laksono Kurnianggoro
Mentor: Delia Passalacqua
*/
#ifndef __OPENCV_FACELANDMARK_HPP__
#define __OPENCV_FACELANDMARK_HPP__
/**
@defgroup face Face Analysis
- @ref tutorial_table_of_content_facemark
- The Facemark API
*/
#include "opencv2/face.hpp"
#include "opencv2/objdetect.hpp"
#include "opencv2/objdetect/objdetect_c.h"
#include "opencv2/imgproc/types_c.h"
namespace cv {
namespace face {
//! @addtogroup face
//! @{
struct CV_EXPORTS_W CParams{
String cascade; //!< the face detector
double scaleFactor; //!< Parameter specifying how much the image size is reduced at each image scale.
int minNeighbors; //!< Parameter specifying how many neighbors each candidate rectangle should have to retain it.
Size minSize; //!< Minimum possible object size.
Size maxSize; //!< Maximum possible object size.
CParams(
String cascade_model,
double sf = 1.1,
int minN = 3,
Size minSz = Size(30, 30),
Size maxSz = Size()
);
};
/** @brief Default face detector
This function is mainly utilized by the implementation of a Facemark Algorithm.
End users are advised to use function Facemark::getFaces which can be manually defined
and circumvented to the algorithm by Facemark::setFaceDetector.
@param image The input image to be processed.
@param faces Output of the function which represent region of interest of the detected faces.
Each face is stored in cv::Rect container.
@param extra_params extra parameters
<B>Example of usage</B>
@code
std::vector<cv::Rect> faces;
CParams params("haarcascade_frontalface_alt.xml");
cv::face::getFaces(frame, faces, &params);
for(int j=0;j<faces.size();j++){
cv::rectangle(frame, faces[j], cv::Scalar(255,0,255));
}
cv::imshow("detection", frame);
@endcode
*/
/*other option: move this function inside Facemark as default face detector*/
CV_EXPORTS bool getFaces( InputArray image,
OutputArray faces,
void * extra_params
);
/** @brief A utility to load list of paths to training image and annotation file.
@param imageList The specified file contains paths to the training images.
@param annotationList The specified file contains paths to the training annotations.
@param images The loaded paths of training images.
@param annotations The loaded paths of annotation files.
Example of usage:
@code
String imageFiles = "images_path.txt";
String ptsFiles = "annotations_path.txt";
std::vector<String> images_train;
std::vector<String> landmarks_train;
loadDatasetList(imageFiles,ptsFiles,images_train,landmarks_train);
@endcode
*/
CV_EXPORTS_W bool loadDatasetList(String imageList,
String annotationList,
std::vector<String> & images,
std::vector<String> & annotations);
/** @brief A utility to load facial landmark dataset from a single file.
@param filename The filename of a file that contains the dataset information.
Each line contains the filename of an image followed by
pairs of x and y values of facial landmarks points separated by a space.
Example
@code
/home/user/ibug/image_003_1.jpg 336.820955 240.864510 334.238298 260.922709 335.266918 ...
/home/user/ibug/image_005_1.jpg 376.158428 230.845712 376.736984 254.924635 383.265403 ...
@endcode
@param images A vector where each element represent the filename of image in the dataset.
Images are not loaded by default to save the memory.
@param facePoints The loaded landmark points for all training data.
@param delim Delimiter between each element, the default value is a whitespace.
@param offset An offset value to adjust the loaded points.
<B>Example of usage</B>
@code
cv::String imageFiles = "../data/images_train.txt";
cv::String ptsFiles = "../data/points_train.txt";
std::vector<String> images;
std::vector<std::vector<Point2f> > facePoints;
loadTrainingData(imageFiles, ptsFiles, images, facePoints, 0.0);
@endcode
*/
CV_EXPORTS_W bool loadTrainingData( String filename , std::vector<String> & images,
OutputArray facePoints,
char delim = ' ', float offset = 0.0);
/** @brief A utility to load facial landmark information from the dataset.
@param imageList A file contains the list of image filenames in the training dataset.
@param groundTruth A file contains the list of filenames
where the landmarks points information are stored.
The content in each file should follow the standard format (see face::loadFacePoints).
@param images A vector where each element represent the filename of image in the dataset.
Images are not loaded by default to save the memory.
@param facePoints The loaded landmark points for all training data.
@param offset An offset value to adjust the loaded points.
<B>Example of usage</B>
@code
cv::String imageFiles = "../data/images_train.txt";
cv::String ptsFiles = "../data/points_train.txt";
std::vector<String> images;
std::vector<std::vector<Point2f> > facePoints;
loadTrainingData(imageFiles, ptsFiles, images, facePoints, 0.0);
@endcode
example of content in the images_train.txt
@code
/home/user/ibug/image_003_1.jpg
/home/user/ibug/image_004_1.jpg
/home/user/ibug/image_005_1.jpg
/home/user/ibug/image_006.jpg
@endcode
example of content in the points_train.txt
@code
/home/user/ibug/image_003_1.pts
/home/user/ibug/image_004_1.pts
/home/user/ibug/image_005_1.pts
/home/user/ibug/image_006.pts
@endcode
*/
CV_EXPORTS_W bool loadTrainingData( String imageList, String groundTruth,
std::vector<String> & images,
OutputArray facePoints,
float offset = 0.0);
/** @brief A utility to load facial landmark information from a given file.
@param filename The filename of file contains the facial landmarks data.
@param points The loaded facial landmark points.
@param offset An offset value to adjust the loaded points.
<B>Example of usage</B>
@code
std::vector<Point2f> points;
face::loadFacePoints("filename.txt", points, 0.0);
@endcode
The annotation file should follow the default format which is
@code
version: 1
n_points: 68
{
212.716603 499.771793
230.232816 566.290071
...
}
@endcode
where n_points is the number of points considered
and each point is represented as its position in x and y.
*/
CV_EXPORTS_W bool loadFacePoints( String filename, OutputArray points,
float offset = 0.0);
/** @brief Utility to draw the detected facial landmark points
@param image The input image to be processed.
@param points Contains the data of points which will be drawn.
@param color The color of points in BGR format represented by cv::Scalar.
<B>Example of usage</B>
@code
std::vector<Rect> faces;
std::vector<std::vector<Point2f> > landmarks;
facemark->getFaces(img, faces);
facemark->fit(img, faces, landmarks);
for(int j=0;j<rects.size();j++){
face::drawFacemarks(frame, landmarks[j], Scalar(0,0,255));
}
@endcode
*/
CV_EXPORTS_W void drawFacemarks( InputOutputArray image, InputArray points,
Scalar color = Scalar(255,0,0));
/** @brief Abstract base class for all facemark models
All facemark models in OpenCV are derived from the abstract base class Facemark, which
provides a unified access to all facemark algorithms in OpenCV.
To utilize this API in your program, please take a look at the @ref tutorial_table_of_content_facemark
### Description
Facemark is a base class which provides universal access to any specific facemark algorithm.
Therefore, the users should declare a desired algorithm before they can use it in their application.
Here is an example on how to declare facemark algorithm:
@code
// Using Facemark in your code:
Ptr<Facemark> facemark = FacemarkLBF::create();
@endcode
The typical pipeline for facemark detection is listed as follows:
- (Non-mandatory) Set a user defined face detection using Facemark::setFaceDetector.
The facemark algorithms are desgined to fit the facial points into a face.
Therefore, the face information should be provided to the facemark algorithm.
Some algorithms might provides a default face recognition function.
However, the users might prefer to use their own face detector to obtains the best possible detection result.
- (Non-mandatory) Training the model for a specific algorithm using Facemark::training.
In this case, the model should be automatically saved by the algorithm.
If the user already have a trained model, then this part can be omitted.
- Load the trained model using Facemark::loadModel.
- Perform the fitting via the Facemark::fit.
*/
class CV_EXPORTS_W Facemark : public virtual Algorithm
{
public:
virtual void read( const FileNode& fn )=0;
virtual void write( FileStorage& fs ) const=0;
/** @brief Add one training sample to the trainer.
@param image Input image.
@param landmarks The ground-truth of facial landmarks points corresponds to the image.
<B>Example of usage</B>
@code
String imageFiles = "../data/images_train.txt";
String ptsFiles = "../data/points_train.txt";
std::vector<String> images_train;
std::vector<String> landmarks_train;
// load the list of dataset: image paths and landmark file paths
loadDatasetList(imageFiles,ptsFiles,images_train,landmarks_train);
Mat image;
std::vector<Point2f> facial_points;
for(size_t i=0;i<images_train.size();i++){
image = imread(images_train[i].c_str());
loadFacePoints(landmarks_train[i],facial_points);
facemark->addTrainingSample(image, facial_points);
}
@endcode
The contents in the training files should follows the standard format.
Here are examples for the contents in these files.
example of content in the images_train.txt
@code
/home/user/ibug/image_003_1.jpg
/home/user/ibug/image_004_1.jpg
/home/user/ibug/image_005_1.jpg
/home/user/ibug/image_006.jpg
@endcode
example of content in the points_train.txt
@code
/home/user/ibug/image_003_1.pts
/home/user/ibug/image_004_1.pts
/home/user/ibug/image_005_1.pts
/home/user/ibug/image_006.pts
@endcode
*/
virtual bool addTrainingSample(InputArray image, InputArray landmarks)=0;
/** @brief Trains a Facemark algorithm using the given dataset.
Before the training process, training samples should be added to the trainer
using face::addTrainingSample function.
@param parameters Optional extra parameters (algorithm dependent).
<B>Example of usage</B>
@code
FacemarkLBF::Params params;
params.model_filename = "ibug68.model"; // filename to save the trained model
Ptr<Facemark> facemark = FacemarkLBF::create(params);
// add training samples (see Facemark::addTrainingSample)
facemark->training();
@endcode
*/
virtual void training(void* parameters=0)=0;
/** @brief A function to load the trained model before the fitting process.
@param model A string represent the filename of a trained model.
<B>Example of usage</B>
@code
facemark->loadModel("../data/lbf.model");
@endcode
*/
virtual void loadModel(String model)=0;
// virtual void saveModel(String fs)=0;
/** @brief Trains a Facemark algorithm using the given dataset.
@param image Input image.
@param faces Output of the function which represent region of interest of the detected faces.
Each face is stored in cv::Rect container.
@param landmarks The detected landmark points for each faces.
@param config Algorithm specific for running time parameters.
<B>Example of usage</B>
@code
Mat image = imread("image.jpg");
std::vector<Rect> faces;
std::vector<std::vector<Point2f> > landmarks;
facemark->fit(image, faces, landmarks);
@endcode
*/
virtual bool fit( InputArray image,\
InputArray faces,\
InputOutputArray landmarks,\
void * config = 0)=0;
/** @brief Set a user defined face detector for the Facemark algorithm.
@param f The user defined face detector function
<B>Example of usage</B>
@code
facemark->setFaceDetector(myDetector);
@endcode
Example of a user defined face detector
@code
bool myDetector( InputArray image, OutputArray ROIs ){
std::vector<Rect> & faces = *(std::vector<Rect>*) ROIs.getObj();
faces.clear();
Mat img = image.getMat();
// -------- do something --------
}
@endcode
*/
virtual bool setFaceDetector(bool(*f)(InputArray , OutputArray, void * ))=0;
/** @brief Detect faces from a given image using default or user defined face detector.
Some Algorithm might not provide a default face detector.
@param image Input image.
@param faces Output of the function which represent region of interest of the detected faces.
Each face is stored in cv::Rect container.
@param extra_params Optional extra-parameters for the face detector function.
<B>Example of usage</B>
@code
std::vector<cv::Rect> faces;
facemark->getFaces(img, faces);
for(int j=0;j<faces.size();j++){
cv::rectangle(img, faces[j], cv::Scalar(255,0,255));
}
@endcode
*/
virtual bool getFaces( InputArray image , OutputArray faces, void * extra_params=0)=0;
/** @brief Get data from an algorithm
@param items The obtained data, algorithm dependent.
<B>Example of usage</B>
@code
Ptr<FacemarkAAM> facemark = FacemarkAAM::create();
facemark->loadModel("AAM.yml");
FacemarkAAM::Data data;
facemark->getData(&data);
std::vector<Point2f> s0 = data.s0;
cout<<s0<<endl;
@endcode
*/
virtual bool getData(void * items=0)=0;
}; /* Facemark*/
//! @}
} /* namespace face */
} /* namespace cv */
#endif //__OPENCV_FACELANDMARK_HPP__

160
modules/face/include/opencv2/face/facemarkAAM.hpp
Unescape View File

@ -0,0 +1,160 @@
/*
By downloading, copying, installing or using the software you agree to this
license. If you do not agree to this license, do not download, install,
copy or use the software.
License Agreement
For Open Source Computer Vision Library
(3-clause BSD License)
Copyright (C) 2013, OpenCV Foundation, all rights reserved.
Third party copyrights are property of their respective owners.
Redistribution and use in source and binary forms, with or without modification,
are permitted provided that the following conditions are met:
* Redistributions of source code must retain the above copyright notice,
this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.
* Neither the names of the copyright holders nor the names of the contributors
may be used to endorse or promote products derived from this software
without specific prior written permission.
This software is provided by the copyright holders and contributors "as is" and
any express or implied warranties, including, but not limited to, the implied
warranties of merchantability and fitness for a particular purpose are
disclaimed. In no event shall copyright holders or contributors be liable for
any direct, indirect, incidental, special, exemplary, or consequential damages
(including, but not limited to, procurement of substitute goods or services;
loss of use, data, or profits; or business interruption) however caused
and on any theory of liability, whether in contract, strict liability,
or tort (including negligence or otherwise) arising in any way out of
the use of this software, even if advised of the possibility of such damage.
This file was part of GSoC Project: Facemark API for OpenCV
Final report: https://gist.github.com/kurnianggoro/74de9121e122ad0bd825176751d47ecc
Student: Laksono Kurnianggoro
Mentor: Delia Passalacqua
*/
#ifndef __OPENCV_FACEMARK_AAM_HPP__
#define __OPENCV_FACEMARK_AAM_HPP__
#include "opencv2/face/facemark.hpp"
namespace cv {
namespace face {
//! @addtogroup face
//! @{
class CV_EXPORTS_W FacemarkAAM : public Facemark
{
public:
struct CV_EXPORTS Params
{
/**
* \brief Constructor
*/
Params();
/**
* \brief Read parameters from file, currently unused
*/
void read(const FileNode& /*fn*/);
/**
* \brief Read parameters from file, currently unused
*/
void write(FileStorage& /*fs*/) const;
std::string model_filename;
int m;
int n;
int n_iter;
bool verbose;
bool save_model;
int max_m, max_n, texture_max_m;
std::vector<float>scales;
};
/**
* \brief Optional parameter for fitting process.
*/
struct CV_EXPORTS Config
{
Config( Mat rot = Mat::eye(2,2,CV_32F),
Point2f trans = Point2f(0.0,0.0),
float scaling = 1.0,
int scale_id=0
);
Mat R;
Point2f t;
float scale;
int model_scale_idx;
};
/**
* \brief Data container for the facemark::getData function
*/
struct CV_EXPORTS Data
{
std::vector<Point2f> s0;
};
/**
* \brief The model of AAM Algorithm
*/
struct CV_EXPORTS Model
{
int npts; //!< unused delete
int max_n; //!< unused delete
std::vector<float>scales;
//!< defines the scales considered to build the model
/*warping*/
std::vector<Vec3i> triangles;
//!< each element contains 3 values, represent index of facemarks that construct one triangle (obtained using delaunay triangulation)
struct Texture{
int max_m; //!< unused delete
Rect resolution;
//!< resolution of the current scale
Mat A;
//!< gray values from all face region in the dataset, projected in PCA space
Mat A0;
//!< average of gray values from all face region in the dataset
Mat AA;
//!< gray values from all erorded face region in the dataset, projected in PCA space
Mat AA0;
//!< average of gray values from all erorded face region in the dataset
std::vector<std::vector<Point> > textureIdx;
//!< index for warping of each delaunay triangle region constructed by 3 facemarks
std::vector<Point2f> base_shape;
//!< basic shape, normalized to be fit in an image with current detection resolution
std::vector<int> ind1;
//!< index of pixels for mapping process to obtains the grays values of face region
std::vector<int> ind2;
//!< index of pixels for mapping process to obtains the grays values of eroded face region
};
std::vector<Texture> textures;
//!< a container to holds the texture data for each scale of fitting
/*shape*/
std::vector<Point2f> s0;
//!< the basic shape obtained from training dataset
Mat S,Q;
//!< the encoded shapes from training data
};
//!< initializer
static Ptr<FacemarkAAM> create(const FacemarkAAM::Params &parameters = FacemarkAAM::Params() );
virtual ~FacemarkAAM() {}
}; /* AAM */
//! @}
} /* namespace face */
} /* namespace cv */
#endif

120
modules/face/include/opencv2/face/facemarkLBF.hpp
Unescape View File

@ -0,0 +1,120 @@
/*
By downloading, copying, installing or using the software you agree to this
license. If you do not agree to this license, do not download, install,
copy or use the software.
License Agreement
For Open Source Computer Vision Library
(3-clause BSD License)
Copyright (C) 2013, OpenCV Foundation, all rights reserved.
Third party copyrights are property of their respective owners.
Redistribution and use in source and binary forms, with or without modification,
are permitted provided that the following conditions are met:
* Redistributions of source code must retain the above copyright notice,
this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.
* Neither the names of the copyright holders nor the names of the contributors
may be used to endorse or promote products derived from this software
without specific prior written permission.
This software is provided by the copyright holders and contributors "as is" and
any express or implied warranties, including, but not limited to, the implied
warranties of merchantability and fitness for a particular purpose are
disclaimed. In no event shall copyright holders or contributors be liable for
any direct, indirect, incidental, special, exemplary, or consequential damages
(including, but not limited to, procurement of substitute goods or services;
loss of use, data, or profits; or business interruption) however caused
and on any theory of liability, whether in contract, strict liability,
or tort (including negligence or otherwise) arising in any way out of
the use of this software, even if advised of the possibility of such damage.
This file was part of GSoC Project: Facemark API for OpenCV
Final report: https://gist.github.com/kurnianggoro/74de9121e122ad0bd825176751d47ecc
Student: Laksono Kurnianggoro
Mentor: Delia Passalacqua
*/
#ifndef __OPENCV_FACEMARK_LBF_HPP__
#define __OPENCV_FACEMARK_LBF_HPP__
#include "opencv2/face/facemark.hpp"
namespace cv {
namespace face {
//! @addtogroup face
//! @{
class CV_EXPORTS_W FacemarkLBF : public Facemark
{
public:
struct CV_EXPORTS Params
{
/**
* \brief Constructor
*/
Params();
double shape_offset;
//!< offset for the loaded face landmark points
String cascade_face;
//!< filename of the face detector model
bool verbose;
//!< show the training print-out
int n_landmarks;
//!< number of landmark points
int initShape_n;
//!< multiplier for augment the training data
int stages_n;
//!< number of refinement stages
int tree_n;
//!< number of tree in the model for each landmark point refinement
int tree_depth;
//!< the depth of decision tree, defines the size of feature
double bagging_overlap;
//!< overlap ratio for training the LBF feature
std::string model_filename;
//!< filename where the trained model will be saved
bool save_model; //!< flag to save the trained model or not
unsigned int seed; //!< seed for shuffling the training data
std::vector<int> feats_m;
std::vector<double> radius_m;
std::vector<int> pupils[2];
//!< index of facemark points on pupils of left and right eye
Rect detectROI;
void read(const FileNode& /*fn*/);
void write(FileStorage& /*fs*/) const;
};
class BBox {
public:
BBox();
~BBox();
BBox(double x, double y, double w, double h);
cv::Mat project(const cv::Mat &shape) const;
cv::Mat reproject(const cv::Mat &shape) const;
double x, y;
double x_center, y_center;
double x_scale, y_scale;
double width, height;
};
static Ptr<FacemarkLBF> create(const FacemarkLBF::Params &parameters = FacemarkLBF::Params() );
virtual ~FacemarkLBF(){};
}; /* LBF */
//! @}
} /* namespace face */
}/* namespace cv */
#endif

318
modules/face/samples/facemark_demo_aam.cpp
Unescape View File

@ -0,0 +1,318 @@
/*
By downloading, copying, installing or using the software you agree to this
license. If you do not agree to this license, do not download, install,
copy or use the software.
License Agreement
For Open Source Computer Vision Library
(3-clause BSD License)
Copyright (C) 2013, OpenCV Foundation, all rights reserved.
Third party copyrights are property of their respective owners.
Redistribution and use in source and binary forms, with or without modification,
are permitted provided that the following conditions are met:
* Redistributions of source code must retain the above copyright notice,
this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.
* Neither the names of the copyright holders nor the names of the contributors
may be used to endorse or promote products derived from this software
without specific prior written permission.
This software is provided by the copyright holders and contributors "as is" and
any express or implied warranties, including, but not limited to, the implied
warranties of merchantability and fitness for a particular purpose are
disclaimed. In no event shall copyright holders or contributors be liable for
any direct, indirect, incidental, special, exemplary, or consequential damages
(including, but not limited to, procurement of substitute goods or services;
loss of use, data, or profits; or business interruption) however caused
and on any theory of liability, whether in contract, strict liability,
or tort (including negligence or otherwise) arising in any way out of
the use of this software, even if advised of the possibility of such damage.
This file was part of GSoC Project: Facemark API for OpenCV
Final report: https://gist.github.com/kurnianggoro/74de9121e122ad0bd825176751d47ecc
Student: Laksono Kurnianggoro
Mentor: Delia Passalacqua
*/
/*----------------------------------------------
* Usage:
* facemark_demo_aam <face_cascade_model> <eyes_cascade_model> <training_images> <annotation_files> [test_files]
*
* Example:
* facemark_demo_aam ../face_cascade.xml ../eyes_cascade.xml ../images_train.txt ../points_train.txt ../test.txt
*
* Notes:
* the user should provides the list of training images_train
* accompanied by their corresponding landmarks location in separated files.
* example of contents for images_train.txt:
* ../trainset/image_0001.png
* ../trainset/image_0002.png
* example of contents for points_train.txt:
* ../trainset/image_0001.pts
* ../trainset/image_0002.pts
* where the image_xxxx.pts contains the position of each face landmark.
* example of the contents:
* version: 1
* n_points: 68
* {
* 115.167660 220.807529
* 116.164839 245.721357
* 120.208690 270.389841
* ...
* }
* example of the dataset is available at https://ibug.doc.ic.ac.uk/download/annotations/lfpw.zip
*--------------------------------------------------*/
#include <stdio.h>
#include <fstream>
#include <sstream>
#include "opencv2/core.hpp"
#include "opencv2/highgui.hpp"
#include "opencv2/imgproc.hpp"
#include "opencv2/face.hpp"
#include <iostream>
#include <string>
#include <ctime>
using namespace std;
using namespace cv;
using namespace cv::face;
bool myDetector( InputArray image, OutputArray ROIs, CascadeClassifier face_cascade);
bool getInitialFitting(Mat image, Rect face, std::vector<Point2f> s0,
CascadeClassifier eyes_cascade, Mat & R, Point2f & Trans, float & scale);
bool parseArguments(int argc, char** argv, CommandLineParser & , String & cascade,
String & model, String & images, String & annotations, String & testImages
);
int main(int argc, char** argv )
{
CommandLineParser parser(argc, argv,"");
String cascade_path,eyes_cascade_path,images_path, annotations_path, test_images_path;
if(!parseArguments(argc, argv, parser,cascade_path,eyes_cascade_path,images_path, annotations_path, test_images_path))
return -1;
//! [instance_creation]
/*create the facemark instance*/
FacemarkAAM::Params params;
params.scales.push_back(2.0);
params.scales.push_back(4.0);
params.model_filename = "AAM.yaml";
Ptr<FacemarkAAM> facemark = FacemarkAAM::create(params);
//! [instance_creation]
//! [load_dataset]
/*Loads the dataset*/
std::vector<String> images_train;
std::vector<String> landmarks_train;
loadDatasetList(images_path,annotations_path,images_train,landmarks_train);
//! [load_dataset]
//! [add_samples]
Mat image;
std::vector<Point2f> facial_points;
for(size_t i=0;i<images_train.size();i++){
image = imread(images_train[i].c_str());
loadFacePoints(landmarks_train[i],facial_points);
facemark->addTrainingSample(image, facial_points);
}
//! [add_samples]
//! [training]
/* trained model will be saved to AAM.yml */
facemark->training();
//! [training]
//! [load_test_images]
/*test using some images*/
String testFiles(images_path), testPts(annotations_path);
if(!test_images_path.empty()){
testFiles = test_images_path;
testPts = test_images_path; //unused
}
std::vector<String> images;
std::vector<String> facePoints;
loadDatasetList(testFiles, testPts, images, facePoints);
//! [load_test_images]
//! [trainsformation_variables]
float scale ;
Point2f T;
Mat R;
//! [trainsformation_variables]
//! [base_shape]
FacemarkAAM::Data data;
facemark->getData(&data);
std::vector<Point2f> s0 = data.s0;
//! [base_shape]
//! [fitting]
/*fitting process*/
std::vector<Rect> faces;
//! [load_cascade_models]
CascadeClassifier face_cascade(cascade_path);
CascadeClassifier eyes_cascade(eyes_cascade_path);
//! [load_cascade_models]
for(int i=0;i<(int)images.size();i++){
printf("image #%i ", i);
//! [detect_face]
image = imread(images[i]);
myDetector(image, faces, face_cascade);
//! [detect_face]
if(faces.size()>0){
//! [get_initialization]
std::vector<FacemarkAAM::Config> conf;
std::vector<Rect> faces_eyes;
for(unsigned j=0;j<faces.size();j++){
if(getInitialFitting(image,faces[j],s0,eyes_cascade, R,T,scale)){
conf.push_back(FacemarkAAM::Config(R,T,scale,(int)params.scales.size()-1));
faces_eyes.push_back(faces[j]);
}
}
//! [get_initialization]
//! [fitting_process]
if(conf.size()>0){
printf(" - face with eyes found %i ", (int)conf.size());
std::vector<std::vector<Point2f> > landmarks;
double newtime = (double)getTickCount();
facemark->fit(image, faces_eyes, landmarks, (void*)&conf);
double fittime = ((getTickCount() - newtime)/getTickFrequency());
for(unsigned j=0;j<landmarks.size();j++){
drawFacemarks(image, landmarks[j],Scalar(0,255,0));
}
printf("%f ms\n",fittime*1000);
imshow("fitting", image);
waitKey(0);
}else{
printf("initialization cannot be computed - skipping\n");
}
//! [fitting_process]
}
} //for
//! [fitting]
}
bool myDetector( InputArray image, OutputArray ROIs, CascadeClassifier face_cascade){
Mat gray;
std::vector<Rect> & faces = *(std::vector<Rect>*) ROIs.getObj();
faces.clear();
if(image.channels()>1){
cvtColor(image.getMat(),gray,CV_BGR2GRAY);
}else{
gray = image.getMat().clone();
}
equalizeHist( gray, gray );
face_cascade.detectMultiScale( gray, faces, 1.2, 2, CV_HAAR_SCALE_IMAGE, Size(30, 30) );
return true;
}
bool getInitialFitting(Mat image, Rect face, std::vector<Point2f> s0 ,CascadeClassifier eyes_cascade, Mat & R, Point2f & Trans, float & scale){
std::vector<Point2f> mybase;
std::vector<Point2f> T;
std::vector<Point2f> base = Mat(Mat(s0)+Scalar(image.cols/2,image.rows/2)).reshape(2);
std::vector<Point2f> base_shape,base_shape2 ;
Point2f e1 = Point2f((float)((base[39].x+base[36].x)/2.0),(float)((base[39].y+base[36].y)/2.0)); //eye1
Point2f e2 = Point2f((float)((base[45].x+base[42].x)/2.0),(float)((base[45].y+base[42].y)/2.0)); //eye2
if(face.width==0 || face.height==0) return false;
std::vector<Point2f> eye;
bool found=false;
Mat faceROI = image( face);
std::vector<Rect> eyes;
//-- In each face, detect eyes
eyes_cascade.detectMultiScale( faceROI, eyes, 1.1, 2, 0 |CV_HAAR_SCALE_IMAGE, Size(20, 20) );
if(eyes.size()==2){
found = true;
int j=0;
Point2f c1( (float)(face.x + eyes[j].x + eyes[j].width*0.5), (float)(face.y + eyes[j].y + eyes[j].height*0.5));
j=1;
Point2f c2( (float)(face.x + eyes[j].x + eyes[j].width*0.5), (float)(face.y + eyes[j].y + eyes[j].height*0.5));
Point2f pivot;
double a0,a1;
if(c1.x<c2.x){
pivot = c1;
a0 = atan2(c2.y-c1.y, c2.x-c1.x);
}else{
pivot = c2;
a0 = atan2(c1.y-c2.y, c1.x-c2.x);
}
scale = (float)(norm(Mat(c1)-Mat(c2))/norm(Mat(e1)-Mat(e2)));
mybase= Mat(Mat(s0)*scale).reshape(2);
Point2f ey1 = Point2f((float)((mybase[39].x+mybase[36].x)/2.0),(float)((mybase[39].y+mybase[36].y)/2.0));
Point2f ey2 = Point2f((float)((mybase[45].x+mybase[42].x)/2.0),(float)((mybase[45].y+mybase[42].y)/2.0));
#define TO_DEGREE 180.0/3.14159265
a1 = atan2(ey2.y-ey1.y, ey2.x-ey1.x);
Mat rot = getRotationMatrix2D(Point2f(0,0), (a1-a0)*TO_DEGREE, 1.0);
rot(Rect(0,0,2,2)).convertTo(R, CV_32F);
base_shape = Mat(Mat(R*scale*Mat(Mat(s0).reshape(1)).t()).t()).reshape(2);
ey1 = Point2f((float)((base_shape[39].x+base_shape[36].x)/2.0),(float)((base_shape[39].y+base_shape[36].y)/2.0));
ey2 = Point2f((float)((base_shape[45].x+base_shape[42].x)/2.0),(float)((base_shape[45].y+base_shape[42].y)/2.0));
T.push_back(Point2f(pivot.x-ey1.x,pivot.y-ey1.y));
Trans = Point2f(pivot.x-ey1.x,pivot.y-ey1.y);
return true;
}else{
Trans = Point2f( (float)(face.x + face.width*0.5),(float)(face.y + face.height*0.5));
}
return found;
}
bool parseArguments(int argc, char** argv, CommandLineParser & parser,
String & cascade,
String & model,
String & images,
String & annotations,
String & test_images
){
const String keys =
"{ @f face-cascade | | (required) path to the cascade model file for the face detector }"
"{ @e eyes-cascade | | (required) path to the cascade model file for the eyes detector }"
"{ @i images | | (required) path of a text file contains the list of paths to all training images}"
"{ @a annotations | | (required) Path of a text file contains the list of paths to all annotations files}"
"{ @t test-images | | Path of a text file contains the list of paths to the test images}"
"{ help h usage ? | | facemark_demo_aam -face-cascade -eyes-cascade -images -annotations [-t]\n"
" example: facemark_demo_aam ../face_cascade.xml ../eyes_cascade.xml ../images_train.txt ../points_train.txt ../test.txt}"
;
parser = CommandLineParser(argc, argv,keys);
parser.about("hello");
if (parser.has("help")){
parser.printMessage();
return false;
}
cascade = String(parser.get<String>("face-cascade"));
model = String(parser.get<string>("eyes-cascade"));
images = String(parser.get<string>("images"));
annotations = String(parser.get<string>("annotations"));
test_images = String(parser.get<string>("test-images"));
if(cascade.empty() || model.empty() || images.empty() || annotations.empty()){
std::cerr << "one or more required arguments are not found" << '\n';
cout<<"face-cascade : "<<cascade.c_str()<<endl;
cout<<"eyes-cascade : "<<model.c_str()<<endl;
cout<<"images : "<<images.c_str()<<endl;
cout<<"annotations : "<<annotations.c_str()<<endl;
parser.printMessage();
return false;
}
return true;
}

217
modules/face/samples/facemark_demo_lbf.cpp
Unescape View File

@ -0,0 +1,217 @@
/*
By downloading, copying, installing or using the software you agree to this
license. If you do not agree to this license, do not download, install,
copy or use the software.
License Agreement
For Open Source Computer Vision Library
(3-clause BSD License)
Copyright (C) 2013, OpenCV Foundation, all rights reserved.
Third party copyrights are property of their respective owners.
Redistribution and use in source and binary forms, with or without modification,
are permitted provided that the following conditions are met:
* Redistributions of source code must retain the above copyright notice,
this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.
* Neither the names of the copyright holders nor the names of the contributors
may be used to endorse or promote products derived from this software
without specific prior written permission.
This software is provided by the copyright holders and contributors "as is" and
any express or implied warranties, including, but not limited to, the implied
warranties of merchantability and fitness for a particular purpose are
disclaimed. In no event shall copyright holders or contributors be liable for
any direct, indirect, incidental, special, exemplary, or consequential damages
(including, but not limited to, procurement of substitute goods or services;
loss of use, data, or profits; or business interruption) however caused
and on any theory of liability, whether in contract, strict liability,
or tort (including negligence or otherwise) arising in any way out of
the use of this software, even if advised of the possibility of such damage.
This file was part of GSoC Project: Facemark API for OpenCV
Final report: https://gist.github.com/kurnianggoro/74de9121e122ad0bd825176751d47ecc
Student: Laksono Kurnianggoro
Mentor: Delia Passalacqua
*/
/*----------------------------------------------
* Usage:
* facemark_demo_lbf <face_cascade_model> <saved_model_filename> <training_images> <annotation_files> [test_files]
*
* Example:
* facemark_demo_lbf ../face_cascade.xml ../LBF.model ../images_train.txt ../points_train.txt ../test.txt
*
* Notes:
* the user should provides the list of training images_train
* accompanied by their corresponding landmarks location in separated files.
* example of contents for images_train.txt:
* ../trainset/image_0001.png
* ../trainset/image_0002.png
* example of contents for points_train.txt:
* ../trainset/image_0001.pts
* ../trainset/image_0002.pts
* where the image_xxxx.pts contains the position of each face landmark.
* example of the contents:
* version: 1
* n_points: 68
* {
* 115.167660 220.807529
* 116.164839 245.721357
* 120.208690 270.389841
* ...
* }
* example of the dataset is available at https://ibug.doc.ic.ac.uk/download/annotations/ibug.zip
*--------------------------------------------------*/
#include <stdio.h>
#include <fstream>
#include <sstream>
#include <iostream>
#include "opencv2/core.hpp"
#include "opencv2/highgui.hpp"
#include "opencv2/imgproc.hpp"
#include "opencv2/face.hpp"
using namespace std;
using namespace cv;
using namespace cv::face;
CascadeClassifier face_cascade;
bool myDetector( InputArray image, OutputArray roi, void * config=0 );
bool parseArguments(int argc, char** argv, CommandLineParser & , String & cascade,
String & model, String & images, String & annotations, String & testImages
);
int main(int argc, char** argv)
{
CommandLineParser parser(argc, argv,"");
String cascade_path,model_path,images_path, annotations_path, test_images_path;
if(!parseArguments(argc, argv, parser,cascade_path,model_path,images_path, annotations_path, test_images_path))
return -1;
/*create the facemark instance*/
FacemarkLBF::Params params;
params.model_filename = model_path;
params.cascade_face = cascade_path;
Ptr<Facemark> facemark = FacemarkLBF::create(params);
face_cascade.load(params.cascade_face.c_str());
facemark->setFaceDetector(myDetector);
/*Loads the dataset*/
std::vector<String> images_train;
std::vector<String> landmarks_train;
loadDatasetList(images_path,annotations_path,images_train,landmarks_train);
Mat image;
std::vector<Point2f> facial_points;
for(size_t i=0;i<images_train.size();i++){
printf("%i/%i :: %s\n", (int)(i+1), (int)images_train.size(),images_train[i].c_str());
image = imread(images_train[i].c_str());
loadFacePoints(landmarks_train[i],facial_points);
facemark->addTrainingSample(image, facial_points);
}
/*train the Algorithm*/
facemark->training();
/*test using some images*/
String testFiles(images_path), testPts(annotations_path);
if(!test_images_path.empty()){
testFiles = test_images_path;
testPts = test_images_path; //unused
}
std::vector<String> images;
std::vector<String> facePoints;
loadDatasetList(testFiles, testPts, images, facePoints);
std::vector<Rect> rects;
CascadeClassifier cc(params.cascade_face.c_str());
for(size_t i=0;i<images.size();i++){
std::vector<std::vector<Point2f> > landmarks;
cout<<images[i];
Mat img = imread(images[i]);
facemark->getFaces(img, rects);
facemark->fit(img, rects, landmarks);
for(size_t j=0;j<rects.size();j++){
drawFacemarks(img, landmarks[j], Scalar(0,0,255));
rectangle(img, rects[j], Scalar(255,0,255));
}
if(rects.size()>0){
cout<<endl;
imshow("result", img);
waitKey(0);
}else{
cout<<"face not found"<<endl;
}
}
}
bool myDetector( InputArray image, OutputArray roi, void * config ){
Mat gray;
std::vector<Rect> & faces = *(std::vector<Rect>*) roi.getObj();
faces.clear();
if(config!=0){
//do nothing
}
if(image.channels()>1){
cvtColor(image,gray,CV_BGR2GRAY);
}else{
gray = image.getMat().clone();
}
equalizeHist( gray, gray );
face_cascade.detectMultiScale( gray, faces, 1.4, 2, CV_HAAR_SCALE_IMAGE, Size(30, 30) );
return true;
}
bool parseArguments(int argc, char** argv, CommandLineParser & parser,
String & cascade,
String & model,
String & images,
String & annotations,
String & test_images
){
const String keys =
"{ @c cascade | | (required) path to the face cascade xml file fo the face detector }"
"{ @i images | | (required) path of a text file contains the list of paths to all training images}"
"{ @a annotations | | (required) Path of a text file contains the list of paths to all annotations files}"
"{ @m model | | (required) path to save the trained model }"
"{ t test-images | | Path of a text file contains the list of paths to the test images}"
"{ help h usage ? | | facemark_demo_lbf -cascade -images -annotations -model [-t] \n"
" example: facemark_demo_lbf ../face_cascade.xml ../images_train.txt ../points_train.txt ../lbf.model}"
;
parser = CommandLineParser(argc, argv,keys);
parser.about("hello");
if (parser.has("help")){
parser.printMessage();
return false;
}
cascade = String(parser.get<String>("cascade"));
model = String(parser.get<string>("model"));
images = String(parser.get<string>("images"));
annotations = String(parser.get<string>("annotations"));
test_images = String(parser.get<string>("t"));
cout<<"cascade : "<<cascade.c_str()<<endl;
cout<<"model : "<<model.c_str()<<endl;
cout<<"images : "<<images.c_str()<<endl;
cout<<"annotations : "<<annotations.c_str()<<endl;
if(cascade.empty() || model.empty() || images.empty() || annotations.empty()){
std::cerr << "one or more required arguments are not found" << '\n';
parser.printMessage();
return false;
}
return true;
}

202
modules/face/samples/facemark_lbf_fitting.cpp
Unescape View File

@ -0,0 +1,202 @@
/*
By downloading, copying, installing or using the software you agree to this
license. If you do not agree to this license, do not download, install,
copy or use the software.
License Agreement
For Open Source Computer Vision Library
(3-clause BSD License)
Copyright (C) 2013, OpenCV Foundation, all rights reserved.
Third party copyrights are property of their respective owners.
Redistribution and use in source and binary forms, with or without modification,
are permitted provided that the following conditions are met:
* Redistributions of source code must retain the above copyright notice,
this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.
* Neither the names of the copyright holders nor the names of the contributors
may be used to endorse or promote products derived from this software
without specific prior written permission.
This software is provided by the copyright holders and contributors "as is" and
any express or implied warranties, including, but not limited to, the implied
warranties of merchantability and fitness for a particular purpose are
disclaimed. In no event shall copyright holders or contributors be liable for
any direct, indirect, incidental, special, exemplary, or consequential damages
(including, but not limited to, procurement of substitute goods or services;
loss of use, data, or profits; or business interruption) however caused
and on any theory of liability, whether in contract, strict liability,
or tort (including negligence or otherwise) arising in any way out of
the use of this software, even if advised of the possibility of such damage.
This file was part of GSoC Project: Facemark API for OpenCV
Final report: https://gist.github.com/kurnianggoro/74de9121e122ad0bd825176751d47ecc
Student: Laksono Kurnianggoro
Mentor: Delia Passalacqua
*/
/*----------------------------------------------
* Usage:
* facemark_lbf_fitting <face_cascade_model> <lbf_model> <video_name>
*
* example:
* facemark_lbf_fitting ../face_cascade.xml ../LBF.model ../video.mp4
*
* note: do not forget to provide the LBF_MODEL and DETECTOR_MODEL
* the model are available at opencv_contrib/modules/face/data/
*--------------------------------------------------*/
#include <stdio.h>
#include <ctime>
#include <iostream>
#include "opencv2/core.hpp"
#include "opencv2/highgui.hpp"
#include "opencv2/imgproc.hpp"
#include "opencv2/face.hpp"
using namespace std;
using namespace cv;
using namespace cv::face;
CascadeClassifier face_cascade;
bool myDetector( InputArray image, OutputArray ROIs, void * config = 0);
bool parseArguments(int argc, char** argv, CommandLineParser & parser,
String & cascade, String & model,String & video);
int main(int argc, char** argv ){
CommandLineParser parser(argc, argv,"");
String cascade_path,model_path,images_path, video_path;
if(!parseArguments(argc, argv, parser,cascade_path,model_path,video_path))
return -1;
face_cascade.load(cascade_path);
FacemarkLBF::Params params;
params.model_filename = model_path;
params.cascade_face = cascade_path;
Ptr<Facemark> facemark = FacemarkLBF::create(params);
facemark->setFaceDetector(myDetector);
facemark->loadModel(params.model_filename.c_str());
VideoCapture capture(video_path);
Mat frame;
if( !capture.isOpened() ){
printf("Error when reading vide\n");
return 0;
}
Mat img;
String text;
char buff[255];
double fittime;
int nfaces;
std::vector<Rect> rects,rects_scaled;
std::vector<std::vector<Point2f> > landmarks;
CascadeClassifier cc(params.cascade_face.c_str());
namedWindow( "w", 1);
for( ; ; )
{
capture >> frame;
if(frame.empty())
break;
double __time__ = (double)getTickCount();
float scale = (float)(400.0/frame.cols);
resize(frame, img, Size((int)(frame.cols*scale), (int)(frame.rows*scale)));
facemark->getFaces(img, rects);
rects_scaled.clear();
for(int j=0;j<(int)rects.size();j++){
rects_scaled.push_back(Rect(
(int)(rects[j].x/scale),
(int)(rects[j].y/scale),
(int)(rects[j].width/scale),
(int)(rects[j].height/scale)));
}
rects = rects_scaled;
fittime=0;
nfaces = (int)rects.size();
if(rects.size()>0){
double newtime = (double)getTickCount();
facemark->fit(frame, rects, landmarks);
fittime = ((getTickCount() - newtime)/getTickFrequency());
for(int j=0;j<(int)rects.size();j++){
landmarks[j] = Mat(Mat(landmarks[j]));
drawFacemarks(frame, landmarks[j], Scalar(0,0,255));
}
}
double fps = (getTickFrequency()/(getTickCount() - __time__));
sprintf(buff, "faces: %i %03.2f fps, fit:%03.0f ms",nfaces,fps,fittime*1000);
text = buff;
putText(frame, text, Point(20,40), FONT_HERSHEY_PLAIN , 2.0,Scalar::all(255), 2, 8);
imshow("w", frame);
waitKey(1); // waits to display frame
}
waitKey(0); // key press to close window
}
bool myDetector( InputArray image, OutputArray ROIs, void * config ){
Mat gray;
std::vector<Rect> & faces = *(std::vector<Rect>*) ROIs.getObj();
faces.clear();
if(config!=0){
//do nothing
}
if(image.channels()>1){
cvtColor(image.getMat(),gray,CV_BGR2GRAY);
}else{
gray = image.getMat().clone();
}
equalizeHist( gray, gray );
face_cascade.detectMultiScale( gray, faces, 1.4, 2, CV_HAAR_SCALE_IMAGE, Size(30, 30) );
return true;
}
bool parseArguments(int argc, char** argv, CommandLineParser & parser,
String & cascade,
String & model,
String & video
){
const String keys =
"{ @c cascade | | (required) path to the cascade model file for the face detector }"
"{ @m model | | (required) path to the trained model }"
"{ @v video | | (required) path input video}"
"{ help h usage ? | | facemark_lbf_fitting -cascade -model -video [-t]\n"
" example: facemark_lbf_fitting ../face_cascade.xml ../LBF.model ../video.mp4}"
;
parser = CommandLineParser(argc, argv,keys);
parser.about("hello");
if (parser.has("help")){
parser.printMessage();
return false;
}
cascade = String(parser.get<String>("cascade"));
model = String(parser.get<string>("model"));
video = String(parser.get<string>("video"));
if(cascade.empty() || model.empty() || video.empty() ){
std::cerr << "one or more required arguments are not found" << '\n';
cout<<"cascade : "<<cascade.c_str()<<endl;
cout<<"model : "<<model.c_str()<<endl;
cout<<"video : "<<video.c_str()<<endl;
parser.printMessage();
return false;
}
return true;
}

237
modules/face/src/facemark.cpp
Unescape View File

@ -0,0 +1,237 @@
/*
By downloading, copying, installing or using the software you agree to this
license. If you do not agree to this license, do not download, install,
copy or use the software.
License Agreement
For Open Source Computer Vision Library
(3-clause BSD License)
Copyright (C) 2013, OpenCV Foundation, all rights reserved.
Third party copyrights are property of their respective owners.
Redistribution and use in source and binary forms, with or without modification,
are permitted provided that the following conditions are met:
* Redistributions of source code must retain the above copyright notice,
this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.
* Neither the names of the copyright holders nor the names of the contributors
may be used to endorse or promote products derived from this software
without specific prior written permission.
This software is provided by the copyright holders and contributors "as is" and
any express or implied warranties, including, but not limited to, the implied
warranties of merchantability and fitness for a particular purpose are
disclaimed. In no event shall copyright holders or contributors be liable for
any direct, indirect, incidental, special, exemplary, or consequential damages
(including, but not limited to, procurement of substitute goods or services;
loss of use, data, or profits; or business interruption) however caused
and on any theory of liability, whether in contract, strict liability,
or tort (including negligence or otherwise) arising in any way out of
the use of this software, even if advised of the possibility of such damage.
This file was part of GSoC Project: Facemark API for OpenCV
Final report: https://gist.github.com/kurnianggoro/74de9121e122ad0bd825176751d47ecc
Student: Laksono Kurnianggoro
Mentor: Delia Passalacqua
*/
#include "opencv2/face.hpp"
#include "opencv2/core.hpp"
#include "precomp.hpp"
/*dataset parser*/
#include <fstream>
#include <sstream>
#include <string>
#include <stdlib.h> /* atoi */
namespace cv {
namespace face {
CParams::CParams(String s, double sf, int minN, Size minSz, Size maxSz){
cascade = s;
scaleFactor = sf;
minNeighbors = minN;
minSize = minSz;
maxSize = maxSz;
}
bool getFaces(InputArray image, OutputArray faces, void * parameters){
Mat gray;
std::vector<Rect> roi;
if(parameters!=0){
CParams * params = (CParams *)parameters;
cvtColor( image.getMat(), gray, CV_BGR2GRAY );
equalizeHist( gray, gray );
CascadeClassifier face_cascade;
if( !face_cascade.load( params->cascade ) ){ printf("--(!)Error loading face_cascade\n"); return false; };
face_cascade.detectMultiScale( gray, roi, params->scaleFactor, params->minNeighbors, 0|CV_HAAR_SCALE_IMAGE, params->minSize, params->maxSize);
Mat(roi).copyTo(faces);
return true;
}else{
return false;
}
}
bool loadDatasetList(String imageList, String groundTruth, std::vector<String> & images, std::vector<String> & landmarks){
std::string line;
/*clear the output containers*/
images.clear();
landmarks.clear();
/*open the files*/
std::ifstream infile;
infile.open(imageList.c_str(), std::ios::in);
std::ifstream ss_gt;
ss_gt.open(groundTruth.c_str(), std::ios::in);
if ((!infile) || !(ss_gt)) {
printf("No valid input file was given, please check the given filename.\n");
return false;
}
/*load the images path*/
while (getline (infile, line)){
images.push_back(line);
}
/*load the points*/
while (getline (ss_gt, line)){
landmarks.push_back(line);
}
return true;
}
bool loadTrainingData(String filename, std::vector<String> & images, OutputArray _facePoints, char delim, float offset){
std::string line;
std::string item;
std::vector<Point2f> pts;
std::vector<float> raw;
std::vector<std::vector<Point2f> > & facePoints =
*(std::vector<std::vector<Point2f> >*) _facePoints.getObj();
std::ifstream infile;
infile.open(filename.c_str(), std::ios::in);
if (!infile) {
std::string error_message = "No valid input file was given, please check the given filename.";
CV_Error(CV_StsBadArg, error_message);
}
/*clear the output containers*/
images.clear();
facePoints.clear();
/*the main loading process*/
while (getline (infile, line)){
std::istringstream ss(line); // string stream for the current line
/*pop the image path*/
getline (ss, item, delim);
images.push_back(item);
/*load all numbers*/
raw.clear();
while (getline (ss, item, delim)){
raw.push_back((float)atof(item.c_str()));
}
/*convert to opencv points*/
pts.clear();
for(unsigned i = 0;i< raw.size();i+=2){
pts.push_back(Point2f(raw[i]+offset,raw[i+1]+offset));
}
facePoints.push_back(pts);
} // main loading process
return true;
}
bool loadTrainingData(String imageList, String groundTruth, std::vector<String> & images, OutputArray _facePoints, float offset){
std::string line;
std::vector<Point2f> facePts;
std::vector<std::vector<Point2f> > & facePoints =
*(std::vector<std::vector<Point2f> >*) _facePoints.getObj();
/*clear the output containers*/
images.clear();
facePoints.clear();
/*load the images path*/
std::ifstream infile;
infile.open(imageList.c_str(), std::ios::in);
if (!infile) {
std::string error_message = "No valid input file was given, please check the given filename.";
CV_Error(CV_StsBadArg, error_message);
}
while (getline (infile, line)){
images.push_back(line);
}
/*load the points*/
std::ifstream ss_gt(groundTruth.c_str());
while (getline (ss_gt, line)){
facePts.clear();
loadFacePoints(line, facePts, offset);
facePoints.push_back(facePts);
}
return true;
}
bool loadFacePoints(String filename, OutputArray points, float offset){
std::vector<Point2f> & pts = *(std::vector<Point2f> *)points.getObj();
std::string line, item;
std::ifstream infile(filename.c_str());
/*pop the version*/
std::getline(infile, line);
CV_Assert(line.compare(0,7,"version")==0);
/*pop the number of points*/
std::getline(infile, line);
CV_Assert(line.compare(0,8,"n_points")==0);
/*get the number of points*/
std::string item_npts;
int npts;
std::istringstream linestream(line);
linestream>>item_npts>>npts;
/*pop out '{' character*/
std::getline(infile, line);
/*main process*/
int cnt = 0;
std::string x, y;
pts.clear();
while (std::getline(infile, line) && cnt<npts )
{
cnt+=1;
std::istringstream ss(line);
ss>>x>>y;
pts.push_back(Point2f((float)atof(x.c_str())+offset,(float)atof(y.c_str())+offset));
}
return true;
}
void drawFacemarks(InputOutputArray image, InputArray points, Scalar color){
Mat img = image.getMat();
std::vector<Point2f> pts = *(std::vector<Point2f>*)points.getObj();
for(size_t i=0;i<pts.size();i++){
circle(img, pts[i],3, color,-1);
}
} //drawPoints
} /* namespace face */
} /* namespace cv */

1222
modules/face/src/facemarkAAM.cpp
View File

File diff suppressed because it is too large Load Diff

1446
modules/face/src/facemarkLBF.cpp
View File

File diff suppressed because it is too large Load Diff

61
modules/face/test/test_facemark.cpp
Unescape View File

@ -0,0 +1,61 @@
/*
By downloading, copying, installing or using the software you agree to this
license. If you do not agree to this license, do not download, install,
copy or use the software.
License Agreement
For Open Source Computer Vision Library
(3-clause BSD License)
Copyright (C) 2013, OpenCV Foundation, all rights reserved.
Third party copyrights are property of their respective owners.
Redistribution and use in source and binary forms, with or without modification,
are permitted provided that the following conditions are met:
* Redistributions of source code must retain the above copyright notice,
this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.
* Neither the names of the copyright holders nor the names of the contributors
may be used to endorse or promote products derived from this software
without specific prior written permission.
This software is provided by the copyright holders and contributors "as is" and
any express or implied warranties, including, but not limited to, the implied
warranties of merchantability and fitness for a particular purpose are
disclaimed. In no event shall copyright holders or contributors be liable for
any direct, indirect, incidental, special, exemplary, or consequential damages
(including, but not limited to, procurement of substitute goods or services;
loss of use, data, or profits; or business interruption) however caused
and on any theory of liability, whether in contract, strict liability,
or tort (including negligence or otherwise) arising in any way out of
the use of this software, even if advised of the possibility of such damage.
This file was part of GSoC Project: Facemark API for OpenCV
Final report: https://gist.github.com/kurnianggoro/74de9121e122ad0bd825176751d47ecc
Student: Laksono Kurnianggoro
Mentor: Delia Passalacqua
*/
#include "test_precomp.hpp"
#include "opencv2/imgcodecs.hpp"
#include "opencv2/face.hpp"
#include <vector>
#include <string>
using namespace std;
using namespace cv;
using namespace cv::face;
TEST(CV_Face_Facemark, test_utilities) {
string image_file = cvtest::findDataFile("face/david1.jpg", true);
string annotation_file = cvtest::findDataFile("face/david1.pts", true);
string cascade_filename =
cvtest::findDataFile("cascadeandhog/cascades/lbpcascade_frontalface.xml", true);
std::vector<Point2f> facial_points;
EXPECT_NO_THROW(loadFacePoints(annotation_file,facial_points));
Mat img = imread(image_file);
EXPECT_NO_THROW(drawFacemarks(img, facial_points, Scalar(0,0,255)));
CParams params(cascade_filename);
std::vector<Rect> faces;
EXPECT_TRUE(getFaces(img, faces, &params));
}

145
modules/face/test/test_facemark_aam.cpp
Unescape View File

@ -0,0 +1,145 @@
/*
By downloading, copying, installing or using the software you agree to this
license. If you do not agree to this license, do not download, install,
copy or use the software.
License Agreement
For Open Source Computer Vision Library
(3-clause BSD License)
Copyright (C) 2013, OpenCV Foundation, all rights reserved.
Third party copyrights are property of their respective owners.
Redistribution and use in source and binary forms, with or without modification,
are permitted provided that the following conditions are met:
* Redistributions of source code must retain the above copyright notice,
this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.
* Neither the names of the copyright holders nor the names of the contributors
may be used to endorse or promote products derived from this software
without specific prior written permission.
This software is provided by the copyright holders and contributors "as is" and
any express or implied warranties, including, but not limited to, the implied
warranties of merchantability and fitness for a particular purpose are
disclaimed. In no event shall copyright holders or contributors be liable for
any direct, indirect, incidental, special, exemplary, or consequential damages
(including, but not limited to, procurement of substitute goods or services;
loss of use, data, or profits; or business interruption) however caused
and on any theory of liability, whether in contract, strict liability,
or tort (including negligence or otherwise) arising in any way out of
the use of this software, even if advised of the possibility of such damage.
This file was part of GSoC Project: Facemark API for OpenCV
Final report: https://gist.github.com/kurnianggoro/74de9121e122ad0bd825176751d47ecc
Student: Laksono Kurnianggoro
Mentor: Delia Passalacqua
*/
/*Usage:
download the opencv_extra from https://github.com/opencv/opencv_extra
and then execute the following commands:
export OPENCV_TEST_DATA_PATH=/home/opencv/opencv_extra/testdata
<build_folder>/bin/opencv_test_face
*/
#include "test_precomp.hpp"
#include "opencv2/imgcodecs.hpp"
#include "opencv2/face.hpp"
#include <vector>
#include <string>
using namespace std;
using namespace cv;
using namespace cv::face;
CascadeClassifier face_detector;
static bool customDetector( InputArray image, OutputArray ROIs, void * config = 0 ){
Mat gray;
std::vector<Rect> & faces = *(std::vector<Rect>*) ROIs.getObj();
faces.clear();
if(config!=0){
//do nothing
}
if(image.channels()>1){
cvtColor(image.getMat(),gray,CV_BGR2GRAY);
}else{
gray = image.getMat().clone();
}
equalizeHist( gray, gray );
face_detector.detectMultiScale( gray, faces, 1.4, 2, CV_HAAR_SCALE_IMAGE, Size(30, 30) );
return true;
}
TEST(CV_Face_FacemarkAAM, can_create_default) {
FacemarkAAM::Params params;
Ptr<Facemark> facemark;
EXPECT_NO_THROW(facemark = FacemarkAAM::create(params));
EXPECT_FALSE(facemark.empty());
}
TEST(CV_Face_FacemarkAAM, can_set_custom_detector) {
string cascade_filename =
cvtest::findDataFile("cascadeandhog/cascades/lbpcascade_frontalface.xml", true);
EXPECT_TRUE(face_detector.load(cascade_filename));
Ptr<Facemark> facemark = FacemarkAAM::create();
EXPECT_TRUE(facemark->setFaceDetector(customDetector));
}
TEST(CV_Face_FacemarkAAM, test_workflow) {
string i1 = cvtest::findDataFile("face/david1.jpg", true);
string p1 = cvtest::findDataFile("face/david1.pts", true);
string i2 = cvtest::findDataFile("face/david2.jpg", true);
string p2 = cvtest::findDataFile("face/david2.pts", true);
std::vector<string> images_train;
images_train.push_back(i1);
images_train.push_back(i2);
std::vector<String> points_train;
points_train.push_back(p1);
points_train.push_back(p2);
string cascade_filename =
cvtest::findDataFile("cascadeandhog/cascades/lbpcascade_frontalface.xml", true);
FacemarkAAM::Params params;
params.n = 1;
params.m = 1;
params.verbose = false;
params.save_model = false;
Ptr<Facemark> facemark = FacemarkAAM::create(params);
Mat image;
std::vector<Point2f> landmarks;
for(size_t i=0;i<images_train.size();i++){
image = imread(images_train[i].c_str());
EXPECT_TRUE(loadFacePoints(points_train[i].c_str(),landmarks));
EXPECT_TRUE(landmarks.size()>0);
EXPECT_TRUE(facemark->addTrainingSample(image, landmarks));
}
EXPECT_NO_THROW(facemark->training());
/*------------ Fitting Part ---------------*/
facemark->setFaceDetector(customDetector);
string image_filename = cvtest::findDataFile("face/david1.jpg", true);
image = imread(image_filename.c_str());
EXPECT_TRUE(!image.empty());
std::vector<Rect> rects;
std::vector<std::vector<Point2f> > facial_points;
EXPECT_TRUE(facemark->getFaces(image, rects));
EXPECT_TRUE(rects.size()>0);
EXPECT_TRUE(facemark->fit(image, rects, facial_points));
EXPECT_TRUE(facial_points[0].size()>0);
/*------------ Test getData ---------------*/
FacemarkAAM::Data data;
EXPECT_TRUE(facemark->getData(&data));
EXPECT_TRUE(data.s0.size()>0);
}

148
modules/face/test/test_facemark_lbf.cpp
Unescape View File

@ -0,0 +1,148 @@
/*
By downloading, copying, installing or using the software you agree to this
license. If you do not agree to this license, do not download, install,
copy or use the software.
License Agreement
For Open Source Computer Vision Library
(3-clause BSD License)
Copyright (C) 2013, OpenCV Foundation, all rights reserved.
Third party copyrights are property of their respective owners.
Redistribution and use in source and binary forms, with or without modification,
are permitted provided that the following conditions are met:
* Redistributions of source code must retain the above copyright notice,
this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.
* Neither the names of the copyright holders nor the names of the contributors
may be used to endorse or promote products derived from this software
without specific prior written permission.
This software is provided by the copyright holders and contributors "as is" and
any express or implied warranties, including, but not limited to, the implied
warranties of merchantability and fitness for a particular purpose are
disclaimed. In no event shall copyright holders or contributors be liable for
any direct, indirect, incidental, special, exemplary, or consequential damages
(including, but not limited to, procurement of substitute goods or services;
loss of use, data, or profits; or business interruption) however caused
and on any theory of liability, whether in contract, strict liability,
or tort (including negligence or otherwise) arising in any way out of
the use of this software, even if advised of the possibility of such damage.
This file was part of GSoC Project: Facemark API for OpenCV
Final report: https://gist.github.com/kurnianggoro/74de9121e122ad0bd825176751d47ecc
Student: Laksono Kurnianggoro
Mentor: Delia Passalacqua
*/
/*Usage:
download the opencv_extra from https://github.com/opencv/opencv_extra
and then execute the following commands:
export OPENCV_TEST_DATA_PATH=/home/opencv/opencv_extra/testdata
<build_folder>/bin/opencv_test_face
*/
#include "test_precomp.hpp"
#include "opencv2/imgcodecs.hpp"
#include "opencv2/face.hpp"
#include <vector>
#include <string>
using namespace std;
using namespace cv;
using namespace cv::face;
CascadeClassifier cascade_detector;
static bool myCustomDetector( InputArray image, OutputArray ROIs, void * config = 0 ){
Mat gray;
std::vector<Rect> & faces = *(std::vector<Rect>*) ROIs.getObj();
faces.clear();
if(config!=0){
//do nothing
}
if(image.channels()>1){
cvtColor(image.getMat(),gray,CV_BGR2GRAY);
}else{
gray = image.getMat().clone();
}
equalizeHist( gray, gray );
cascade_detector.detectMultiScale( gray, faces, 1.4, 2, CV_HAAR_SCALE_IMAGE, Size(30, 30) );
return true;
}
TEST(CV_Face_FacemarkLBF, can_create_default) {
FacemarkLBF::Params params;
params.n_landmarks = 68;
Ptr<Facemark> facemark;
EXPECT_NO_THROW(facemark = FacemarkLBF::create(params));
EXPECT_FALSE(facemark.empty());
}
TEST(CV_Face_FacemarkLBF, can_set_custom_detector) {
string cascade_filename =
cvtest::findDataFile("cascadeandhog/cascades/lbpcascade_frontalface.xml", true);
EXPECT_TRUE(cascade_detector.load(cascade_filename));
Ptr<Facemark> facemark = FacemarkLBF::create();
EXPECT_TRUE(facemark->setFaceDetector(myCustomDetector));
}
TEST(CV_Face_FacemarkLBF, test_workflow) {
string i1 = cvtest::findDataFile("face/david1.jpg", true);
string p1 = cvtest::findDataFile("face/david1.pts", true);
string i2 = cvtest::findDataFile("face/david2.jpg", true);
string p2 = cvtest::findDataFile("face/david2.pts", true);
std::vector<string> images_train;
images_train.push_back(i1);
images_train.push_back(i2);
std::vector<String> points_train;
points_train.push_back(p1);
points_train.push_back(p2);
string cascade_filename =
cvtest::findDataFile("cascadeandhog/cascades/lbpcascade_frontalface.xml", true);
FacemarkLBF::Params params;
params.cascade_face = cascade_filename;
params.verbose = false;
params.save_model = false;
Ptr<Facemark> facemark = FacemarkLBF::create(params);
Mat image;
std::vector<Point2f> landmarks;
for(size_t i=0;i<images_train.size();i++){
image = imread(images_train[i].c_str());
EXPECT_TRUE(loadFacePoints(points_train[i].c_str(),landmarks));
EXPECT_TRUE(landmarks.size()>0);
EXPECT_TRUE(facemark->addTrainingSample(image, landmarks));
}
EXPECT_NO_THROW(facemark->training());
/*------------ Fitting Part ---------------*/
cascade_detector.load(cascade_filename);
facemark->setFaceDetector(myCustomDetector);
string image_filename = cvtest::findDataFile("face/david1.jpg", true);
image = imread(image_filename.c_str());
EXPECT_TRUE(!image.empty());
std::vector<Rect> rects;
std::vector<std::vector<Point2f> > facial_points;
EXPECT_TRUE(facemark->getFaces(image, rects));
EXPECT_TRUE(rects.size()>0);
EXPECT_TRUE(facemark->fit(image, rects, facial_points));
EXPECT_TRUE(facial_points[0].size()>0);
}
TEST(CV_Face_FacemarkLBF, get_data) {
Ptr<Facemark> facemark = FacemarkLBF::create();
EXPECT_TRUE(facemark->getData());
}

113
modules/face/tutorials/facemark_aam/facemark_aam.markdown
Unescape View File

@ -0,0 +1,113 @@
Using the FacemarkAAM {#tutorial_facemark_aam}
==========================================================
Goals
----
In this tutorial you will learn how to:
- creating the instance of FacemarkAAM
- training the AAM model
- Fitting using FacemarkAAM
Preparation
--------
Before you continue with this tutorial, you should download the dataset of facial landmarks detection.
We suggest you to download the LFPW dataset which can be retrieved at <https://ibug.doc.ic.ac.uk/download/annotations/lfpw.zip>.
Make sure that the annotation format is supported by the API, the contents in annotation file should look like the following snippet:
@code
version: 1
n_points: 68
{
212.716603 499.771793
230.232816 566.290071
...
}
@endcode
The next thing to do is to make 2 text files containing the list of image files and annotation files respectively. Make sure that the order or image and annotation in both files are matched. Furthermore, it is advised to use absolute path instead of relative path.
Example to make the file list in Linux machine
@code
ls $PWD/trainset/*.jpg > images_train.txt
ls $PWD/trainset/*.pts > annotation_train.txt
@endcode
example of content in the images_train.txt
@code
/home/user/lfpw/trainset/100032540_1.jpg
/home/user/lfpw/trainset/100040721_1.jpg
/home/user/lfpw/trainset/100040721_2.jpg
/home/user/lfpw/trainset/1002681492_1.jpg
@endcode
example of content in the annotation_train.txt
@code
/home/user/lfpw/trainset/100032540_1.pts
/home/user/lfpw/trainset/100040721_1.pts
/home/user/lfpw/trainset/100040721_2.pts
/home/user/lfpw/trainset/1002681492_1.pts
@endcode
Optionally, you can create the similar files for the testset.
In this tutorial, the pre-trained model will not be provided due to its large file size (~500MB). By following this tutorial, you will be able to train obtain your own trained model within few minutes.
Working with the AAM Algorithm
--------
The full working code is available in the face/samples/facemark_demo_aam.cpp file. In this tutorial, the explanation of some important parts are covered.
-# <B>Creating the instance of AAM algorithm</B>
@snippet face/samples/facemark_demo_aam.cpp instance_creation
Firstly, an instance of parameter for the AAM algorithm is created. In this case, we will modify the default list of the scaling factor. By default, the scaling factor used is 1.0 (no scaling). Here we add two more scaling factor which will make the instance trains two more model at scale 2 and 4 (2 time smaller and 4 time smaller, faster faster fitting time). However, you should make sure that this scaling factor is not too big since it will make the image scaled into a very small one. Thus it will lost all of its important information for the landmark detection purpose.
Alternatively, you can override the default scaling in similar way to this example:
@code
std::vector<float>scales;
scales.push_back(1.5);
scales.push_back(2.4);
FacemarkAAM::Params params;
params.scales = scales;
@endcode
-# <B>Loading the dataset</B>
@snippet face/samples/facemark_demo_aam.cpp load_dataset
List of the dataset are loaded into the program. We will put the samples from dataset one by one in the next step.
-# <B>Adding the samples to the trainer</B>
@snippet face/samples/facemark_demo_aam.cpp add_samples
The image from the dataset list are loaded one by one as well as its corresponding annotation data. Then the pair of sample is added to the trainer.
-# <B>Training process</B>
@snippet face/samples/facemark_demo_aam.cpp training
The training process is called using a single line of code. Make sure that all the required training samples are already added to the trainer.
-# <B>Preparation for fitting</B>
First of all, you need to load the list of test files.
@snippet face/samples/facemark_demo_aam.cpp load_test_images
Since the AAM needs initialization parameters (rotation, translation, and scaling), you need to declare the required variable to store these information which will be obtained using a custom function. Since the implementation of getInitialFitting() function in this example is not optimal, you can create your own function.
The initialization is obtained by comparing the base shape of the trained model with the current face image. In this case, the rotation is obtained by comparing the angle of line formed by two eyes in the input face image with the same line in the base shape. Meanwhile, the scaling is obtained by comparing the length of line between eyes in the input image compared to the base shape.
-# <B>Fitting process</B>
The fitting process is started by detecting the face in a given image.
@snippet face/samples/facemark_demo_aam.cpp detect_face
If at least one face is found, then the next step is computing the initialization parameters. In this case, since the getInitialFitting() function is not optimal, it may not find pair of eyes from a given face. Therefore, we will filter out the face without initialization parameters and in this case, each element in the `conf` vector represent the initialization parameter for each filtered face.
@snippet face/samples/facemark_demo_aam.cpp get_initialization
For the fitting parameter stored in the `conf` vector, the last parameter represent the ID of scaling factor that will be used in the fitting process. In this example the fitting will use the biggest scaling factor (4) which is expected to have the fastest computation time compared to the other scales. If the ID if bigger than the available trained scale in the model, the the model with the biggest scale ID is used.
The fitting process is quite simple, you just need to put the corresponding image, vector of `cv::Rect` representing the ROIs of all faces in the given image, container of the landmark points represented by `landmarks` variable, and the configuration variables.
@snippet face/samples/facemark_demo_aam.cpp fitting_process
After the fitting process is finished, you can visualize the result using the `drawFacemarks` function.

273
modules/face/tutorials/facemark_add_algorithm/facemark_add_algorithm.markdown
Unescape View File

@ -0,0 +1,273 @@
Adding a new algorithm to the Facemark API {#tutorial_facemark_add_algorithm}
==========================================================
Goals
----
In this tutorial you will learn how to:
- integrate a new algorithm of facial landmark detector into the Facemark API
- compile a specific contrib module
- using extra parameters in a function
Explanation
-----------
- **Add the class header**
The class header for a new algorithm should be added to a new file in include/opencv2/face.
Here is the template that you can use to integrate a new algorithm, change the FacemarkNEW to a representative name of the new algorithm and save it using a representative filename accordingly.
@code{.cpp}
class CV_EXPORTS_W FacemarkNEW : public Facemark {
public:
struct CV_EXPORTS Config {
Config();
/*read only parameters - just for example*/
double detect_thresh; //!< detection confidence threshold
double sigma; //!< another parameter
void read(const FileNode& /*fn*/);
void write(FileStorage& /*fs*/) const;
};
/*Builder and destructor*/
static Ptr<FacemarkNEW> create(const FacemarkNEW::Config &conf = FacemarkNEW::Config() );
virtual ~FacemarkNEW(){};
};
@endcode
- **Add the implementation code**
Create a new file in the source folder with name representing the new algorithm.
Here is the template that you can use.
@code{.cpp}
#include "opencv2/face.hpp"
#include "precomp.hpp"
namespace cv
{
FacemarkNEW::Config::Config(){
detect_thresh = 0.5;
sigma=0.2;
}
void FacemarkNEW::Config::read( const cv::FileNode& fn ){
*this = FacemarkNEW::Config();
if (!fn["detect_thresh"].empty())
fn["detect_thresh"] >> detect_thresh;
if (!fn["sigma"].empty())
fn["sigma"] >> sigma;
}
void FacemarkNEW::Config::write( cv::FileStorage& fs ) const{
fs << "detect_thresh" << detect_thresh;
fs << "sigma" << sigma;
}
/*implementation of the algorithm is in this class*/
class FacemarkNEWImpl : public FacemarkNEW {
public:
FacemarkNEWImpl( const FacemarkNEW::Config &conf = FacemarkNEW::Config() );
void read( const FileNode& /*fn*/ );
void write( FileStorage& /*fs*/ ) const;
void loadModel(String filename);
bool setFaceDetector(bool(*f)(InputArray , OutputArray, void * extra_params));
bool getFaces( InputArray image , OutputArray faces, void * extra_params);
Config config;
protected:
bool addTrainingSample(InputArray image, InputArray landmarks);
void training();
bool fit(InputArray image, InputArray faces, InputOutputArray landmarks, void * runtime_params);
Config config; // configurations
/*proxy to the user defined face detector function*/
bool(*faceDetector)(InputArray , OutputArray, void * );
}; // class
Ptr<FacemarkNEW> FacemarkNEW::create(const FacemarkNEW::Config &conf){
return Ptr<FacemarkNEWImpl>(new FacemarkNEWImpl(conf));
}
FacemarkNEWImpl::FacemarkNEWImpl( const FacemarkNEW::Config &conf ) :
config( conf )
{
// other initialization
}
bool FacemarkNEWImpl::addTrainingSample(InputArray image, InputArray landmarks){
// pre-process and save the new training sample
return true;
}
void FacemarkNEWImpl::training(){
printf("training\n");
}
bool FacemarkNEWImpl::fit(
InputArray image,
InputArray faces,
InputOutputArray landmarks,
void * runtime_params)
{
if(runtime_params!=0){
// do something based on the extra parameters
}
printf("fitting\n");
return 0;
}
void FacemarkNEWImpl::read( const cv::FileNode& fn ){
config.read( fn );
}
void FacemarkNEWImpl::write( cv::FileStorage& fs ) const {
config.write( fs );
}
void FacemarkNEWImpl::loadModel(String filename){
// load the model
}
bool FacemarkNEWImpl::setFaceDetector(bool(*f)(InputArray , OutputArray, void * extra_params )){
faceDetector = f;
isSetDetector = true;
return true;
}
bool FacemarkNEWImpl::getFaces( InputArray image , OutputArray roi, void * extra_params){
if(!isSetDetector){
return false;
}
if(extra_params!=0){
//extract the extra parameters
}
std::vector<Rect> & faces = *(std::vector<Rect>*)roi.getObj();
faces.clear();
faceDetector(image.getMat(), faces, extra_params);
return true;
}
}
@endcode
- **Compiling the code**
Clear the build folder and then rebuild the entire library.
Note that you can deactivate the compilation of other contrib modules by adding "-D BUILD_opencv_<MODULE_NAME>=OFF" flag to the cmake.
After that you can execute make command in "<build_folder>/modules/face" to speed up the compiling process.
Best Practice
-----------
- **Handling the extra parameters**
To handle the extra parameters, a new struct should be created to holds all the required parameters.
Here is an example of of a parameters container
@code
struct CV_EXPORTS Params
{
Params( Mat rot = Mat::eye(2,2,CV_32F),
Point2f trans = Point2f(0.0,0.0),
float scaling = 1.0
);
Mat R;
Point2f t;
float scale;
};
@endcode
Here is a snippet to extract the extra parameters:
@code
if(runtime_params!=0){
Telo* conf = (Telo*)params;
Params* params
std::vector<Params> params = *(std::vector<Params>*)runtime_params;
for(size_t i=0; i<params.size();i++){
fit(img, landmarks[i], params[i].R,params[i].t, params[i].scale);
}
}else{
// do something
}
@endcode
And here is an example to pass the extra parameter into fit function
@code
FacemarkAAM::Params * params = new FacemarkAAM::Params(R,T,scale);
facemark->fit(image, faces, landmarks, params)
@endcode
In order to understand this scheme, here is a simple example that you can try to compile and see how it works.
@code
struct Params{
int x,y;
Params(int _x, int _y);
};
Params::Params(int _x,int _y){
x = _x;
y = _y;
}
void test(int a, void * params=0){
printf("a:%i\n", a);
if(params!=0){
Params* params = (Params*)params;
printf("extra parameters:%i %i\n", params->x, params->y);
}
}
int main(){
Params* params = new Params(7,22);
test(99, params);
return 0;
}
@endcode
- **Minimize the dependency**
It is highly recomended to keep the code as small as possible when compiled. For this purpose, the developers are ecouraged to avoid the needs of heavy dependency such as `imgcodecs` and `highgui`.
- **Documentation and examples**
Please update the documentation whenever needed and put example code for the new algorithm.
- **Test codes**
An algorithm should be accompanied with its corresponding test code to ensure that the algorithm is compatible with various types of environment (Linux, Windows64, Windows32, Android, etc). There are several basic test that should be performed as demonstrated in the test/test_facemark_lbf.cpp file including cration of its instance, add training data, perform the training process, load a trained model, and perform the fitting to obtain facial landmarks.
- **Data organization**
It is advised to divide the data for a new algorithm into 3 parts :
@code
class CV_EXPORTS_W FacemarkNEW : public Facemark {
public:
struct CV_EXPORTS Params
{
// variables utilized as extra parameters
}
struct CV_EXPORTS Config
{
// variables used to configure the algorithm
}
struct CV_EXPORTS Model
{
// variables to store the information of model
}
static Ptr<FacemarkNEW> create(const FacemarkNEW::Config &conf = FacemarkNEW::Config() );
virtual ~FacemarkNEW(){};
}
@endcode

29
modules/face/tutorials/facemark_tutorial.markdown
Unescape View File

@ -0,0 +1,29 @@
Tutorial on Facial Landmark Detector API {#tutorial_table_of_content_facemark}
==========================================================
The facial landmark detector API is useful to detect facial landmarks from an input image.
- @subpage tutorial_facemark_add_algorithm
*Compatibility:* \> OpenCV 3.0
*Author:* Laksono Kurnianggoro
Adding a new algorithm in to the API.
- @subpage tutorial_facemark_usage
*Compatibility:* \> OpenCV 3.0
*Author:* Laksono Kurnianggoro
Tutorial on how to use the API.
- @subpage tutorial_facemark_aam
*Compatibility:* \> OpenCV 3.0
*Author:* Laksono Kurnianggoro
Tutorial on how to use the FacemarkAAM algorithm.

182
modules/face/tutorials/facemark_usage/facemark_usage.markdown
Unescape View File

@ -0,0 +1,182 @@
Using the Facemark API {#tutorial_facemark_usage}
==========================================================
Goals
----
In this tutorial will helps you to
- Create a Facemark object.
- Set a user defined face detector for the facemark algorithm
- Train the algorithm.
- Use the trained model to detect the facial landmarks from a given image.
Preparation
---------
Before you continue with this tutorial, you should download the dataset of facial landmarks detection.
We suggest you to download the helen dataset which can be retrieved at <http://www.ifp.illinois.edu/~vuongle2/helen/> (Caution! The algorithm requires around 9GB of RAM to train on this dataset).
Make sure that the annotation format is supported by the API, the contents in annotation file should look like the following snippet:
@code
version: 1
n_points: 68
{
212.716603 499.771793
230.232816 566.290071
...
}
@endcode
The next thing to do is to make 2 text files containing the list of image files and annotation files respectively. Make sure that the order or image and annotation in both files are matched. Furthermore, it is advised to use absolute path instead of relative path.
Example to make the file list in Linux machine
@code
ls $PWD/trainset/*.jpg > images_train.txt
ls $PWD/trainset/*.pts > annotation_train.txt
@endcode
example of content in the images_train.txt
@code
/home/user/helen/trainset/100032540_1.jpg
/home/user/helen/trainset/100040721_1.jpg
/home/user/helen/trainset/100040721_2.jpg
/home/user/helen/trainset/1002681492_1.jpg
@endcode
example of content in the annotation_train.txt
@code
/home/user/helen/trainset/100032540_1.pts
/home/user/helen/trainset/100040721_1.pts
/home/user/helen/trainset/100040721_2.pts
/home/user/helen/trainset/1002681492_1.pts
@endcode
Creating the facemark object
---------
@code
/*create the facemark instance*/
FacemarkLBF::Params params;
params.model_filename = "helen.model"; // the trained model will be saved using this filename
Ptr<Facemark> facemark = FacemarkLBF::create(params);
@endcode
Set a custom face detector function
---------
Firstly, you need to create your own face detector function, you might also need to create a `struct` to save the custom parameter. Alternatively, you can just make these parameter hard coded within the `myDetector` function.
@code
struct Conf {
cv::String model_path;
double scaleFactor;
Conf(cv::String s, double d){
model_path = s;
scaleFactor = d;
};
};
bool myDetector( InputArray image, OutputArray roi, void * config ){
Mat gray;
std::vector<Rect> & faces = *(std::vector<Rect>*) roi.getObj();
faces.clear();
if(config!=0){
Conf* conf = (Conf*)config;
if(image.channels()>1){
cvtColor(image,gray,CV_BGR2GRAY);
}else{
gray = image.getMat().clone();
}
equalizeHist( gray, gray );
CascadeClassifier face_cascade(conf->model_path);
face_cascade.detectMultiScale( gray, faces, conf->scaleFactor, 2, CV_HAAR_SCALE_IMAGE, Size(30, 30) );
return true;
}else{
return false;
}
}
@endcode
The following snippet demonstrates how to set the custom detector to the facemark object and use it to detect the faces. Keep in mind that some facemark object might use the face detector during the training process.
@code
Conf* config = new Conf("../data/lbpcascade_frontalface.xml",1.4);
facemark->setFaceDetector(myDetector);
@endcode
Here is the snippet for detecting face using the user defined face detector function.
@code
Mat img = imread("../data/himym3.jpg");
std::vector<cv::Rect> faces;
facemark->getFaces(img, faces, config);
for(int j=0;j<faces.size();j++){
cv::rectangle(img, faces[j], cv::Scalar(255,0,255));
}
imshow("result", img);
waitKey(0);
@endcode
Training a facemark object
----
- First of all, you need to set the training parameters
@code
params.n_landmarks = 68; // number of landmark points
params.initShape_n = 10; // number of multiplier for make data augmentation
params.stages_n=5; // amount of refinement stages
params.tree_n=6; // number of tree in the model for each landmark point
params.tree_depth=5; //he depth of decision tree
facemark = FacemarkLBF::create(params);
@endcode
- And then, you need to load the file list from the dataset that you have prepared.
@code
std::vector<String> images_train;
std::vector<String> landmarks_train;
loadDatasetList("images_train.txt","annotation_train.txt",images_train,landmarks_train);
@endcode
- The next step is to add training samples into the facemark object.
@code
Mat image;
std::vector<Point2f> facial_points;
for(size_t i=0;i<images_train.size();i++){
image = imread(images_train[i].c_str());
loadFacePoints(landmarks_train[i],facial_points);
facemark->addTrainingSample(image, facial_points);
}
@endcode
- execute the training process
@code
/*train the Algorithm*/
facemark->training();
@endcode
Use the trained model to detect the facial landmarks from a given image.
-----
- First of all, load the trained model. You can also download the pre-trained model in this link <https://raw.githubusercontent.com/kurnianggoro/GSOC2017/master/data/lbfmodel.yaml>
@code
facemark->loadModel(params.model_filename);
@endcode
- Detect the faces
@code
facemark->getFaces(img, faces, config);
@endcode
- Perform the fitting process
@code
std::vector<std::vector<Point2f> > landmarks;
facemark->fit(img, faces, landmarks);
@endcode
- Display the result
@code
for(int j=0;j<faces.size();j++){
face::drawFacemarks(img, landmarks[j], Scalar(0,0,255));
}
imshow("result", img);
waitKey(0);
@endcode
Write Preview
Loading…
Cancel
Save