Machine Learning-Based Fast Angular Prediction Mode Decision Technique in Video Coding

In this paper we propose a machine learning-based fast intra-prediction mode decision algorithm, using random forest that is an ensemble model of randomized decision trees. The random forest is used to estimate an intra-prediction mode from a prediction unit and to reduce encoding time significantly by avoiding the intensive Rate-Distortion optimization of a number of intra-prediction modes. To this aim, we develop a randomized tree model including parameterized split functions at nodes to learn directional block-based features. The feature uses only four pixels reflecting a directional property of a block, and, thus the evaluation is fast and efficient. To integrate the proposed technique into the conventional video coding standard frameworks, the intra-prediction mode derived from the proposed technique, called an inferred mode (IM), is used to shrink the pool of the candidate modes before carrying out the Rate-Distortion (R-D) optimization. The proposed technique is implemented into the High Efficiency Video Coding Test Model (HM) reference software of the state-of-the-art video coding standard and Joint Exploration Model (JEM) reference software, by integrating the random forest trained off-line into the codecs. Experimental results demonstrate that the proposed technique achieves significant encoding time reduction with only slight coding loss as compared the reference software models.