Table of Contents Author Guidelines Submit a Manuscript
Applied Computational Intelligence and Soft Computing
Volume 2015 (2015), Article ID 193868, 12 pages
http://dx.doi.org/10.1155/2015/193868
Research Article

On the Performance Improvement of Devanagari Handwritten Character Recognition

1IET, DAVV, Khandwa Road, Indore 452017, India
2IIT, Khandwa Road, Indore 452017, India

Received 3 August 2014; Revised 29 December 2014; Accepted 16 January 2015

Academic Editor: Ying-Tung Hsiao

Copyright © 2015 Pratibha Singh et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

The paper is about the application of mini minibatch stochastic gradient descent (SGD) based learning applied to Multilayer Perceptron in the domain of isolated Devanagari handwritten character/numeral recognition. This technique reduces the variance in the estimate of the gradient and often makes better use of the hierarchical memory organization in modern computers. -weight decay is added on minibatch SGD to avoid overfitting. The experiments are conducted firstly on the direct pixel intensity values as features. After that, the experiments are performed on the proposed flexible zone based gradient feature extraction algorithm. The results are promising on most of the standard dataset of Devanagari characters/numerals.

1. Introduction

The need to recognize the handwritten text is challenging problem not only from the perspective of behavioural biometrics but also in the context of pattern recognition. Writing is the most natural mode of collecting, storing, and transmitting the information. It is a widely used communication tool among human being and forms the input for simulation of reading by a machine. The intensive research effort in the field of character recognition (CR) was due to challenges on simulation of human reading and also because of its potential applications, for example, postal automation, bank cheque analysis and processing, conversion of handwritten text into Braille, hand drawn pictogram or formula recognition, and so forth. Pattern recognition is a computationally intensive and time-consuming task due to vast amount of image data and large number of computational steps involved. The great demand for fast classification of letters by the post office requires a fast automated recognition system. Traditionally, the conventional approach always demands a very high speed computer or a parallel computer system to perform a satisfactory and fast recognition. We cannot meet these demands using simple digital computer. Digital computers are good at handling problems which are explicitly formulated, but handwritten character recognition is not such a problem. With the advent of neurocomputing technology, the great research effort has been devoted to formulate the pattern recognition tasks in an efficient manner. Present study investigates the direction for the improvement of performance in Devanagari CR system.

There are 18 official languages accepted in the present Indian constitution. Twelve different scripts are used for writing these languages. Many of the Indian documents are supposed to be written in three languages, namely, English, Hindi, and the state official language as per the three language formula [1]. Hindi is the popularly used language of India and is the third most popular language in the world, which is written and encoded using Devanagari script. Not only Hindi but also the other languages such as Marathi, Sanskrit, and Konkani are encoded into Devanagari script. Basic characters in Devanagri script consists of 13 vowels and 36 consonants [2] as shown in Figure 1. Writing style is from left to write. There is no concept of upper and lower case. Vowels following consonants take a modified shape and are known as modified characters. Shape of modified characters varies depending on whether the vowel modifier is placed to the left, right, top, or bottom of the consonants as shown in Table 1. In Devanagari script, there is a practice of using more than twelve different forms each of 36 consonants [3], giving rise to its shape variation. The existence of modifiers and the compound characters, shown in Figure 2, makes the character recognition more difficult for Devanagari script.

Table 1: Modifiers in Devanagari script.
Figure 1: Basic characters of Devanagari script.
Figure 2: Compound characters of Devanagari script.

A key reason for the absence of sustained research efforts in Devanagari Optical Character Recognition (OCR) is mainly because of the paucity of data resources. Ground-truthed data for words and characters, on-line dictionaries, corpora of text documents, reliable standardized statistical analysis, and evaluation tools are currently lacking. So, the creation of such data resources will undoubtedly provide a much needed fillip to researchers working on Devanagari OCR. The major research for isolated Devanagari characters is done by Computer Vision and Pattern Recognition (CVPR) Unit of Indian Statistical Institute (ISI), Kolkata. Fuzzy model based recognition scheme was proposed by Hanmandlu and Murthy [4] for isolated Devanagari numerals. Classifier combination is now widely applied on Devanagari CR system for increasing recognition accuracy [59]. Classifier combination techniques using shadow features is proposed by Arora et al. [3, 10]. MLP and HMM combination schemes have been proposed by Bhattacharya et al. [6].

Pattern recognition applications use the algorithm of machine learning. Machine learning in supervised classification domain mainly involves two steps: training and testing. One of the way to improve the performance of machine learning algorithm is to use a low bias algorithm and to train the algorithm with a large data typically known as big data. But learning with large dataset comes with its own computational problems and requires a million aggregations over each step. So, massive computation cost in the most popular algorithms such as gradient descent requires alternative solution. The present study focuses on this issue for the larger dataset of Devanagri. Stochastic gradient descent, online learning, and minibatch learning are some of the alternatives to deal with this issue. The rest of the paper is organized as follows: Section 2 talks about the methodology used, Section 3 gives the description of classifier model used, Section 4 describes the evaluation results on the Devanagari datasets, and final concluding remarks are presented in Section 5.

2. Methodology Used

The design cycle for the recognition of characters follows all the steps of standard pattern recognition technique in supervised learning mode. For the purpose of training, datasets developed by various research groups are used. To the best of our knowledge, only isolated characters and numeral dataset of Devanagari script is available as test bed. So, these are experimented in this study. The design cycle used in this study is shown in Figure 3.

Figure 3: Design cycle of handwriting recognition system.

The images obtained from benchmarking dataset exist in two groups, namely, training and testing. The images are subjected to various preprocessing steps described in Table 2. The output after preprocessing module is not suitable for the classifier training because of the high dimensionality. Feature extraction/selection is an important step for the dimensionality reduction. In this study, we have used a gradient based directional features. The features are obtained from the 9 different parts of the image sample. The global and local histogram based zone boundary concept [11] as shown in Figure 4, is used before feature extraction.

Table 2: Proposed feature extraction algorithm.
Figure 4: (a) Standard zoning; (b) global zoning; (c) and (d) local zoning.
2.1. The Feature Extraction Module

The features are accumulated zonewise using two different alternatives of zoning. For example,(i)standard zoning: the entire bounding box of the image is divided into zones, and gradients are accumulated for each zone;(ii)elastic zoning: the concept of elastic zoning is based on equalizing the density of each zone. We here define global or local zoning concept. In global zoning, the zone separating line is decided on the basis of equal density division, horizontally and vertically, whereas in local method the image is divided horizontally based on density equalization in each zone and then the vertical boundary is decided on the local division of density.

2.1.1. The Gradient Features

The gradient feature decomposition was originally proposed for online character recognition. This method is applicable to machine printed/handwritten and binary/grayscale, as well as low resolution images. Conventionally, the gradient is calculated on each pixel of the image. In our analysis, we have applied “Sobel” edge detection algorithm to calculate gradient vector at each image pixel of preprocessed image. The gradient vector can be quantized into eight directions using two methods, namely, angle vector quantization and vector decomposition using parallelogram rule. In the first method, the magnitude of gradient in each image pixel is assigned to its nearest directional plane and in the second method the gradient vector is decomposed into two nearest directional planes using parallelogram vector division rule. The parallelogram quantization method gives less quantization error, so we have taken this method for quantizing gradient vector.

The calculated gradient of the image is decomposed into four, eight, or sixteen directional planes. For our analysis, we have taken eight directional planes. Figure 5(c) shows the gradient vector decomposition into their nearest vector plane. We have accumulated the magnitude of gradient in eight discrete directions for each of the subsection of original image. The components of gradient vector are given by the following equations:

Figure 5: (a) Sobel mask for vertical gradient; (b) Sobel mask for horizontal gradient; (c) quantization using parallelogram rule.

3. The Classifier Model

Multilayer Perceptron is used as classifier. The architecture of Multilayer Perceptron (MLP) consists of input layer, output layer, and hidden layer. Single hidden layer perceptron gives universal approximation in many pattern recognition applications. The output vector for a single layer perceptron is given by where , are the bias vectors at the hidden and output layers, , are the weight matrices at the respective nodes, and , are the activation functions. For a classification problem, if is the training vector, where , a -dimensional training vector, and . For the prediction function given in (2), the zero-one loss function is given by where is the indicator function given by where is the set of all parameters of the given model. The objective of the training is to minimize the loss function. But, the zero-one loss function is not differentiable; therefore, negative log likelihood of loss function minimization is used as the objective of the training Weights are updated using gradient of the error surface defined by loss function. Gradient is estimated from the training data. In this study, stochastic gradient descent based learning (Table 3) approach is applied to MLP. In ordinary gradient descent algorithm, repeatedly small steps are made in downward direction on an error surface, which is the mean square error. Mean square error is a function of weights. Stochastic gradient descent (SGD) works according to the same principles as ordinary gradient descent, but it proceeds more quickly by estimating the gradient from just a few examples at a time instead of the entire training set. In its purest form, estimation of the gradient is made from just a single example at a time. In both gradient descent (GD) and stochastic gradient descent (SGD), we update a set of weights in an iterative manner to minimize an error function. In normal GD (Table 4), all the samples of the training set have to be processed before updating the weight for a particular iteration, while, in SGD, only single training sample from whole training set is used to do the update for a weight in a particular iteration. Thus, for big data, if the number of training samples is very large, then using gradient descent may take too long because in every iteration, when we are updating the values of the parameters, we are running through the complete training set. On the other hand, using SGD will be faster because you use only one training sample and it starts improving itself right away from the first sample. SGD often converges much faster compared to GD, but the error function is not as well minimized as in the case of GD. Often, in most cases, the close approximation that we get in SGD for the parameter values is enough because they reach the optimal values and keep oscillating there. Stochastic gradient descent has a convergence rate which is independent of the size of our dataset and is thus adapted when we have a huge or even infinite dataset. But, there are two downsides to it:

Table 3: Stochastic gradient descent.
Table 4: Gradient descent.

(i)its slow convergence rate: it is not beneficial to get a faster convergence rate on the training set as this will not be translated into a better convergence rate on the test set [12];(ii)its sensitivity to the two parameters, the learning rate, and the decrease constant.For deep learning, a variant that we recommend is a further modification in stochastic gradient descent using the so-called “minibatches.” Minibatch SGD (MSGD), explained in Table 5, works identically to SGD, except that we use more than one training example to make each estimate of the gradient. This technique reduces variance in the estimate of the gradient and often makes better use of the hierarchical memory organization in modern computers.

Table 5: Minibatch-SGD.

There is a trade-off in the choice of the minibatch size . With large , time is wasted in reducing the variance of the gradient estimator, that time would be better spent on additional gradient steps. An optimal is model, dataset, and hardware dependent and can be anywhere from 1 to several hundreds. In machine learning, when we train our model from data, we are trying to prepare it to do well on new examples, not the ones it has already seen. The MSGD training loop does not have the capability for this generalization and have a tendency for overfitting. The way to combat overfitting is through regularization and early stopping using validation. There are several techniques for regularization; the most commonly used method is regularization which is explained in the next section.

3.1. Weight Decay

Weight decay is a subset of regularization methods. The penalty term in weight decay, by definition, penalizes large weights. Other regularization methods may involve not only the weights but also various derivatives of the output function. The weight decay penalty term causes the weights to converge to smaller absolute values than they otherwise would. Large weights can hurt generalization in two different ways. Excessively large weights leading to hidden units can cause the output function to be too rough, possibly with near discontinuities. Excessively large weights leading to output units can cause wild outputs far beyond the range of the data if the output activation function is not bounded to the same range as the data. To put it another way, large weights can cause excessive variance of the output.

3.2. and Regularization

and regularization involve adding an extra term to the loss function, which penalizes certain parameter configurations. Formally, if our negative log likelihood loss function is , then the regularized loss will be given by This is written as follow for the present study: where is the norm of is a hyperparameter which controls the relative importance of the regularization parameter. Commonly used values for are 1 and 2, hence the nomenclature. If , then the regularizer is also called “weight decay.” In principle, adding a regularization term to the loss will encourage smooth network mappings in a neural network (by penalizing large values of the parameters, which decreases the amount of nonlinearity that the network models). More intuitively, the two terms (NLL and ) correspond to modeling the data well (NLL), having “simple” or “smooth” solutions. Thus, minimizing the sum of both will, in theory, correspond to finding the right trade-off between the fit to the training data and the “generality” of the solution that is found.

4. Experimental Results

The experiments are performed on various datasets as described in the next subsections. The images from the dataset (training and testing) are preprocessed. The feature extraction method as discussed in Table 2 is applied and Feed-Forward Neural Network based classifier is used which is incorporated with -Regularization and mini-batch. For implementing the Feed-Forward Neural Network using minibatch approach, the code developed by Palm [13] is used in MATLAB. The neural network configuration as specified in column 4 of Table 9 is either a three layer or a four layer network. The first value of the configuration specifies the number of features, last value specifies the number of classes and the rest (between input and output) specifies the units present in the hidden layer. The experiments are performed on Intel Core i-3 processor with 6 GB RAM. In next subsection, the information about various datasets is given.

4.1. The MNIST Dataset

The MNIST [14] dataset consists of handwritten digit images and it is divided in 60,000 examples for the training set and 10,000 examples for testing. All digit images have been size-normalized and centered in a fixed size image of 28 28 pixels. In the original dataset, each pixel of the image is represented by a value between 0 and 255, where 0 is black, 255 is white, and anything in between is a different shade of grey. An image is represented as 1-dimensional array of 784 (28 28) float values between 0 and 1 (0 stands for black and 1 for white). The labels are numbers between 0 and 9 indicating which digit the image represents. When using the dataset, we usually divide it in minibatches.

4.2. The CPAR-2012 Dataset

This dataset is available since the year 2012 [15] to the research community and is developed by Intelligent System Group Noida (Centre for Pattern Analysis and Recognition, CPAR). This is the largest dataset available for the handwritten isolated patterns. It consists of 35000 images of numerals and 78400 images of characters. The data is collected from diverse population strata of 2000 writers from various states of India having different religions. The numeral dataset is having 11 classes in this dataset because the pattern corresponding to number “9” can be written in two alternate ways as shown in Table 7. The character dataset is having 1000 training images and 600 test images of 49 classes each. Table 6 describes the number of pattern for each numeral class of CPAR-12 used in this study.

Table 6: Number of image samples in CPAR-12 numeral dataset.
Table 7: Image samples of CPAR-2012 handwritten numerals [15].
4.3. CVPR-ISI Dataset

This dataset is available for the global research community since the year 2009 and is developed by CVPR unit of ISI Kolkata. The Devanagari numeral database includes samples collected from mail pieces and job application forms through specially designed form for data collection. The dataset consists of 22,556 images (as shown in Table 8) stored in “tif ” format, collected from 1049 writers.

Table 8: Sample images from CVPR Dataset [16].
Table 9: Error rates for various datasets using direct pixel values.

In this study, experiments are performed with single hidden layer (having 200 or 100 hidden units) NN and two hidden layer NN (having 200 and 100 hidden units resp.). Input layer is made up of the number of units equal to the size of feature vector generated. Network is trained for 200 epoch with a mini batch size of 100. The learning rate used in the experiments is kept constant to be equal to 1. The value of regularization weight decay is maintained at . The activation function used in hidden layer is hyperbolic tangent and in the output layer it is logistic. The momentum set for experiments is equal to 0.7. The error rates obtained for the various dataset using direct pixel values as features are tabulated in Table 9. The results are obtained for the size normalized image patterns with the features as direct pixel intensities. The network configuration varies with respect to the number of hidden layers. The time for recognition includes the training time along with the classification.

Table 10 illustrates the performance of various feature extraction algorithms described in Table 2 on SGD based learning of MLP. All the sample images are partitioned into 3 3 subimages, but the criterion for zoning is different. Edge based directional features are extracted from each of the subimage in 8 discrete directions defining 72D feature vector in each case. Three layer architecture of MLP is used with 200 nodes in hidden layer. The rest of the NN-settings are kept as same as in the previous experiment on direct pixel values. The number of data samples used in this case for training and testing is same as that mentioned in Table 9 (column 2 and column 3).

Table 10: Error rates for various datasets using proposed feature extraction method.
4.4. Performance Comparison for Numeral Dataset CPAR-2012

In the experiments performed by Kumar and Ravulakollu [17], the average recognition rate reported was 95.18% with rejection of some of the samples, on a single classifier and 97.87% with rejection of some of samples on an ensemble of classifier is applied to the patterns. For the same dataset, the proposed learning strategy gives the recognition rate of 98.07% without any rejection. For the CPAR-2012 numeral dataset, the accuracy the proposed features with SGD learning of MLP is improved by 0.2%. Here, the point to be emphasized is that our results are better than the previous results because of two reasons: we have not used any rejection of samples and we have not used model of classifier ensemble. The recognition results reported here in this work are for the single classifier.

4.5. Performance Comparison for Character Dataset CPAR-2012

Kumar and Ravulakollu [17] applied the method of classifier ensemble on the character dataset and obtained the recognition rate of 84.03% for a rejection of 5.3%. For the same dataset, the proposed learning method yields the recognition rate of 82.79% with direct pixel values and 85.11% with feature extraction method without any rejection of patterns. For CPAR-2012 character dataset, the accuracy improvement of 1.08% is observed with the proposed feature set along with the SGD learning on MLP. Our results are better from the point of view of the reasoning given in the previous section.

4.6. Performance Comparison for Numeral Dataset ISI-CVPR

For ISI Devanagari numerals performance comparison is made with the three previously reported results. The result reported by Das et al. [18] is better, but it is tested for small subset of ISI data, whereas our method is tested on complete ISI dataset. Result reported by Bhattacharya and Chaudhuri [16] is tested on complete dataset, but the accuracy is obtained with 0.24% rejection of patterns and features used in the study belong to complex algorithms of feature extraction. Also, the classification algorithm used by them is a multistage MLP based classifier and there is a rejection of patterns used by the last stage in multistage MLP. For the single -NN (-nearest neighbor), the benchmark established by them is of accuracy level 97.26%. So, for ISI data considering the complexity of the algorithm, our results are better. The proposed method is better compared with the method developed by Bhattacharya and Chaudhuri [16] in terms for speed of recognition.

5. Conclusion

The minibatch stochastic gradient is used to accelerate the speed of recognition for large dataset. The recognition results are obtained using direct pixel intensities as feature on MSGD. Secondly, the results are obtained for edge based directional features on MSGD. From the results it is clear that if Pre-processing and feature extraction is used along with the mini batch algorithm, the error rate reduces by a 1–3% over direct pixel intensity features. The proposed method gives better/same recognition accuracy with most of the standard benchmarks available for Devanagari characters. The recognition time cannot be compared from the previously reported results as the time was not considered as criterion in previous research. The proposed method is faster over normal gradient descent based learning and it gives good accuracy on even direct pixel intensities. The performance improvement of the proposed methods is given in terms of accuracy as tabulated in column 6 and 7 of Table 11. In this table the first column is giving the information about the dataset and the reference associated with the previous result. For showing the effectiveness of our proposed strategy we have taken the same dataset of Devanagari characters. Column 2 is summarizing the approach used by the previously reported results, column 3 provides features used by them, column 4 gives the classification method used by the previous research and column 5 of Table 11 provides the accuracy of previously reported results by other researchers.

Table 11: Comparison of performance of proposed method with the previously reported results.

Figure 6 is showing the accuracy improvement of the proposed results over the existing methods.

Figure 6: Performance of the proposed method with the previously reported result.

Conflict of Interests

The authors declare that there is no conflict of interests regarding the publication of this paper.

Acknowledgments

The authors would like to thank Intelligent System Group Noida and CVPR Unit of ISI Kolkata for providing the dataset of Devanagari characters/numerals.

References

  1. Department of Official Language, 1968, http://www.rajbhasha.nic.in/GOLPContent.aspx?t=endolresolution.
  2. N. Sharma, U. Pal, F. Kimura, and S. Pal, “Recognition of off-line handwritten devnagari characters using quadratic classifier,” in Computer Vision, Graphics and Image Processing : Proceedings of the 5th Indian Conference, ICVGIP 2006, Madurai, India, December 13-16, 2006, vol. 4338 of Lecture Notes in Computer Science, pp. 805–816, Springer, Berlin, Germany, 2006. View at Publisher · View at Google Scholar
  3. S. Arora, D. Bhattacharjee, M. Nasipuri, D. K. Basu, and M. Kundu, “Recognition of non-compound handwritten Devanagari characters using a combination of MLP and minimum edit distance,” International Journal of Computer Science and Security, vol. 4, no. 1, pp. 1–14, 2010. View at Google Scholar
  4. M. Hanmandlu and O. V. R. Murthy, “Fuzzy model based recognition of handwritten numerals,” Pattern Recognition, vol. 40, no. 6, pp. 1840–1854, 2007. View at Publisher · View at Google Scholar · View at Scopus
  5. S. Shelke and S. Apte, “A novel multi-feature multi-classifier scheme for unconstrained handwritten devanagari character recognition,” in Proceedings of the 12th International Conference on Frontiers in Handwriting Recognition (ICFHR '10), pp. 215–219, kolkata, India, November 2010. View at Publisher · View at Google Scholar · View at Scopus
  6. U. Bhattacharya, S. K. Parui, B. Shaw, and K. Bhattacharya, “Neural combination of ANN and HMM for handwritten Devnagari numeral recognition,” in Proceedings of the 10th International Workshop on Frontiers in Handwriting Recognition, pp. 613–618, La Baule, France, 2006.
  7. T. Jindal and U. Bhattacharya, “Recognition of offline handwritten numerals using an ensemble of MLPs combined by adaboost,” in Proceedings of the 4th International Workshop on Multilingual OCR (MOCR '13), vol. 18, Washington, DC , USA, August 2013. View at Publisher · View at Google Scholar
  8. S. Kumar, “A three tier scheme for devanagari hand-printed character recognition,” in Proceedings of the World Congress on Nature and Biologically Inspired Computing (NABIC '09), pp. 1016–1021, Coimbatore, India, December 2009. View at Publisher · View at Google Scholar · View at Scopus
  9. R. Kumar and K. K. Ravulakollu, “On the performance of Devnagari handwritten character recognition,” World Applied Sciences Journal, vol. 31, no. 6, pp. 1012–1019, 2014. View at Google Scholar · View at Scopus
  10. S. Arora, D. Bhattacharjee, M. Nasipuri, D. K. Basu, and M. Kundu, “Combining multiple feature extraction techniques for Handwritten Devnagari Character recognition,” in Proceedings of the 3rd International Conference on Industrial and Information Systems (ICIIS '08), pp. 1–6, IEEE, Kharagpur, India, December 2008. View at Publisher · View at Google Scholar · View at Scopus
  11. P. Singh, A. Verma, and N. S. Chaudhari, “Performance analysis of flexible zone based features to classify Hindi numerals,” in Proceedings of the 3rd International Conference on Electronics Computer Technology (ICECT '11), pp. 292–296, Kanyakumari, India, April 2011. View at Publisher · View at Google Scholar · View at Scopus
  12. L. Bottou and O. Bousquet, “The tradeoffs of large scale learning,” in Optimization for Machine Learning, S. Sra, N. Sebastian, and W. Ste, Eds., pp. 351–368, MIT Press, Cambridge, Mass, USA, 2011. View at Google Scholar
  13. R. B. Palm, Prediction as a candidate for learning deep hierarchical models of data [M.S. thesis], 2012.
  14. Y. LeCun, C. Cortes, and C. J. C. Burges, The MNIST Dataset of Handwritten Digits, 1998, http://yann.lecun.com/exdb/mnist/.
  15. R. Kumar, A. Kumar, and P. Ahmed, A Benchmark Dataset for Devanagari Document Recognition Research, WSEAS Press, Lemesos, Cyprus, 2013.
  16. U. Bhattacharya and B. B. Chaudhuri, “Handwritten numeral databases of Indian scripts and multistage recognition of mixed numerals,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 31, no. 3, pp. 444–457, 2009. View at Publisher · View at Google Scholar · View at Scopus
  17. R. Kumar and K. K. Ravulakollu, “Handwritten devnagari digit recognition: benchmarking on new dataset,” Journal of Theoretical and Applied Information Technology, vol. 60, no. 3, pp. 543–555, 2014. View at Google Scholar · View at Scopus
  18. N. Das, J. M. Reddy, R. Sarkar et al., “A statistical-topological feature combination for recognition of handwritten numerals,” Applied Soft Computing Journal, vol. 12, no. 8, pp. 2486–2495, 2012. View at Publisher · View at Google Scholar · View at Scopus