Jump to content

MNIST database

From Wikipedia, the free encyclopedia
This is an old revision of this page, as edited by Cosmia Nebula (talk | contribs) at 09:31, 25 March 2025 (→Special Database: 19). The present address (URL) is a permanent link to this revision, which may differ significantly from the current revision.
MNIST sample images
Sample images from MNIST test dataset

The MNIST database (Modified National Institute of Standards and Technology database[1]) is a large database of handwritten digits that is commonly used for training various image processing systems.[2][3] The database is also widely used for training and testing in the field of machine learning.[4][5] It was created by "re-mixing" the samples from NIST's original datasets.[6] The creators felt that since NIST's training dataset was taken from American Census Bureau employees, while the testing dataset was taken from American high school students, it was not well-suited for machine learning experiments.[7] Furthermore, the black and white images from NIST were normalized to fit into a 28x28 pixel bounding box and anti-aliased, which introduced grayscale levels.[7]

The MNIST database contains 60,000 training images and 10,000 testing images.[8] Half of the training set and half of the test set were taken from NIST's training dataset, while the other half of the training set and the other half of the test set were taken from NIST's testing dataset.[9] The original creators of the database keep a list of some of the methods tested on it.[7] In their original paper, they use a support-vector machine to get an error rate of 0.8%.[10]

The original MNIST dataset contains at least 4 wrong labels.[11]

History

USPS database

In 1988, a dataset of digits from the US Postal Service was constructed. It contained 16×16 grayscale images digitized from handwritten zip codes that appeared on U.S. mail passing through the Buffalo, New York post office. The training set had 7291 images, and test set had 2007, making a total of 9298. Both training and test set contained ambiguous, unclassifiable, and misclassified data. The dataset was used to train and benchmark the 1989 LeNet.[12][13]

The task is rather difficult. On the test set, two humans made errors at an average rate of 2.5%.[14]

Special Database

An example HSF. This is from NIST Special Database 19, with filename f1002_33.png.

Previously, NIST released several "Special Databases". Of particular importance to MNIST are NIST Test Data 1, or SD-1, released in 1990-05, Special Database 3, or SD-3, released in 1992-02, and Special Database 7, or SD-7, released in 1992-04. They were released on CD-ROMs.[6] They were obtained by asking people to write on "Handwriting Sample Forms" (HSFs), then digitizing the HSFs, then segmenting out the alphanumerical characters. Each HSF contains multiple entry fields, wherein people were asked to write. Each writer wrote a single HSF.

SD-1 and SD-3 were constructed from the same set of HSFs by 2100 employees of the United States Census Bureau stationed throughout the US. SD-1 contained the segmented data entry fields, but not the segmented alphanumerics. SD-3 contained binary 128×128 images digitized from alphanumerics. It consisted of 223,125 digits, 44,951 upper-case letters, and 45,313 lower case letters.

SD-7 was the test set, and it contained 58,646 128×128 binary images written by 500 high school students in Bethesda, Maryland. Each image is accompanied by the identity of its writer. SD-7 was released without labels on CD-ROMs, and the labels were later released on floppy drives. It did not contain the HSFs.

SD-3 was much cleaner and easier to recognize than images in SD-7.[7] It was found that machine learning systems trained and validated on SD-3 suffered significant drops in performance on SD-7.[15] At the First Census OCR Systems Conference, there was a competition, where teams were given SD-3 as the training set and would submit their systems for classifying SD-7.

SD-19 was published in 1995, as a compilation of SD-1, SD-3, SD-7 and some further data. It contained 814,255 binary images of alphanumerics and binary images of 4169 HSFs, including those 500 HSFs that were used to generate SD-7. It was updated in 2016.[6]

MNIST

The original dataset from MNIST contained 128x128 binary images. Each was size-normalized to fit in a 20x20 pixel box while preserving their aspect ratio, and anti-aliased to grayscale. Then it was put into a 28x28 image by translating it until the center of mass of the pixels is in the center of the image. The details of how the downsampling proceeded was reconstructed.[16]

The training set and the test set both originally had 60k samples, but 50k of the test set samples were usually discarded, and only the samples indexed 24476 to 34475 would be used, giving just 10k samples in the test set.[17]

Further versions

In 2019, the full 60k test set from MNIST was restored to construct the QMNIST, which has 60k images in the training set and 60k in the test set.[18][16]

Extended MNIST (EMNIST) is a newer dataset developed and released by NIST to be the (final) successor to MNIST, released in 2017.[19][20] MNIST included images only of handwritten digits. EMNIST includes all the images from NIST Special Database 19 (SD 19) released in 1995, .[21][22] The images in EMNIST were converted into the same 28x28 pixel format, by the same process, as were the MNIST images. Accordingly, tools which work with MNIST would likely work unmodified with EMNIST.

Fashion MNIST was created in 2017 as a more challenging replacement for MNIST. The dataset consists of 70,000 28x28 grayscale images of fashion products from 10 categories.[23]

Performance

Some researchers have achieved "near-human performance" on the MNIST database, using a committee of neural networks; in the same paper, the authors achieve performance double that of humans on other recognition tasks.[24] The highest error rate listed[7] on the original website of the database is 12 percent, which is achieved using a simple linear classifier with no preprocessing.[10]

In 2004, a best-case error rate of 0.42 percent was achieved on the database by researchers using a new classifier called the LIRA, which is a neural classifier with three neuron layers based on Rosenblatt's perceptron principles.[25]

Some researchers have tested artificial intelligence systems using the database put under random distortions. The systems in these cases are usually neural networks and the distortions used tend to be either affine distortions or elastic distortions.[7] Sometimes, these systems can be very successful; one such system achieved an error rate on the database of 0.39 percent.[26]

In 2011, an error rate of 0.27 percent, improving on the previous best result, was reported by researchers using a similar system of neural networks.[27] In 2013, an approach based on regularization of neural networks using DropConnect has been claimed to achieve a 0.21 percent error rate.[28] In 2016, the single convolutional neural network best performance was 0.25 percent error rate.[29] As of August 2018, the best performance of a single convolutional neural network trained on MNIST training data using no data augmentation is 0.25 percent error rate.[29][30] Also, the Parallel Computing Center (Khmelnytskyi, Ukraine) obtained an ensemble of only 5 convolutional neural networks which performs on MNIST at 0.21 percent error rate.[31][32]

Classifiers

This is a table of some of the machine learning methods used on the dataset and their error rates, by type of classifier:

Type Classifier Distortion Preprocessing Error rate (%)
Linear classifier Pairwise linear classifier None Deskewing 7.6[10]
K-Nearest Neighbors K-NN with rigid transformations None None 0.96[33]
K-Nearest Neighbors K-NN with non-linear deformation (P2DHMDM) None Shiftable edges 0.52[34]
Boosted Stumps Product of stumps on Haar features None Haar features 0.87[35]
Non-linear classifier 40 PCA + quadratic classifier None None 3.3[10]
Random Forest Fast Unified Random Forests for Survival, Regression, and Classification (RF-SRC)[36] None Simple statistical pixel importance 2.8[37]
Support-vector machine (SVM) Virtual SVM, deg-9 poly, 2-pixel jittered None Deskewing 0.56[38]
Neural network 2-layer 784-800-10 None None 1.6[39]
Neural network 2-layer 784-800-10 Elastic distortions None 0.7[39]
Deep neural network (DNN) 6-layer 784-2500-2000-1500-1000-500-10 Elastic distortions None 0.35[40]
Convolutional neural network (CNN) 6-layer 784-40-80-500-1000-2000-10 None Expansion of the training data 0.31[41]
Convolutional neural network 6-layer 784-50-100-500-1000-10-10 None Expansion of the training data 0.27[42]
Convolutional neural network (CNN) 13-layer 64-128(5x)-256(3x)-512-2048-256-256-10 None None 0.25[29]
Convolutional neural network Committee of 35 CNNs, 1-20-P-40-P-150-10 Elastic distortions Width normalizations 0.23[24]
Convolutional neural network Committee of 5 CNNs, 6-layer 784-50-100-500-1000-10-10 None Expansion of the training data 0.21[31][32]
Convolutional neural network Committee of 20 CNNS with Squeeze-and-Excitation Networks[43] None Data augmentation 0.17[44]
Convolutional neural network Ensemble of 3 CNNs with varying kernel sizes None Data augmentation consisting of rotation and translation 0.09[45]

See also

References

  1. ^ "The MNIST Database of handwritten digits". Yann LeCun, Courant Institute, NYU Corinna Cortes, Google Labs, New York Christopher J.C. Burges, Microsoft Research, Redmond.
  2. ^ "Support vector machines speed pattern recognition - Vision Systems Design". Vision Systems Design. September 2004. Retrieved 17 August 2013.
  3. ^ Gangaputra, Sachin. "Handwritten digit database". Retrieved 17 August 2013.
  4. ^ Qiao, Yu (2007). "The MNIST Database of handwritten digits". Retrieved 18 August 2013.
  5. ^ Platt, John C. (1999). "Using analytic QP and sparseness to speed training of support vector machines" (PDF). Advances in Neural Information Processing Systems: 557–563. Archived from the original (PDF) on 4 March 2016. Retrieved 18 August 2013.
  6. ^ a b c Grother, Patrick J. "NIST Special Database 19 - Handprinted Forms and Characters Database" (PDF). National Institute of Standards and Technology.
  7. ^ a b c d e f LeCun, Yann; Cortez, Corinna; Burges, Christopher C.J. "The MNIST Handwritten Digit Database". Yann LeCun's Website yann.lecun.com. Retrieved 30 April 2020.
  8. ^ Kussul, Ernst; Baidyk, Tatiana (2004). "Improved method of handwritten digit recognition tested on MNIST database". Image and Vision Computing. 22 (12): 971–981. doi:10.1016/j.imavis.2004.03.008.
  9. ^ Zhang, Bin; Srihari, Sargur N. (2004). "Fast k-Nearest Neighbor Classification Using Cluster-Based Trees" (PDF). IEEE Transactions on Pattern Analysis and Machine Intelligence. 26 (4): 525–528. doi:10.1109/TPAMI.2004.1265868. PMID 15382657. S2CID 6883417. Retrieved 20 April 2020.
  10. ^ a b c d LeCun, Yann; Léon Bottou; Yoshua Bengio; Patrick Haffner (1998). "Gradient-Based Learning Applied to Document Recognition" (PDF). Proceedings of the IEEE. 86 (11): 2278–2324. doi:10.1109/5.726791. S2CID 14542261. Retrieved 18 August 2013.
  11. ^ Muller, Nicolas M.; Markert, Karla (July 2019). Identifying Mislabeled Instances in Classification Datasets. 2019 International Joint Conference on Neural Networks (IJCNN). IEEE. pp. 1–8. arXiv:1912.05283. doi:10.1109/IJCNN.2019.8851920. ISBN 978-1-7281-1985-4.
  12. ^ Denker, John; Gardner, W.; Graf, Hans; Henderson, Donnie; Howard, R.; Hubbard, W.; Jackel, L. D.; Baird, Henry; Guyon, Isabelle (1988). "Neural Network Recognizer for Hand-Written Zip Code Digits". Advances in Neural Information Processing Systems. 1. Morgan-Kaufmann.
  13. ^ LeCun, Y.; Boser, B.; Denker, J. S.; Henderson, D.; Howard, R. E.; Hubbard, W.; Jackel, L. D. (December 1989). "Backpropagation Applied to Handwritten Zip Code Recognition". Neural Computation. 1 (4): 541–551. doi:10.1162/neco.1989.1.4.541. ISSN 0899-7667. S2CID 41312633.
  14. ^ Simard, Patrice; LeCun, Yann; Denker, John (1992). "Efficient Pattern Recognition Using a New Transformation Distance". Advances in Neural Information Processing Systems. 5. Morgan-Kaufmann.
  15. ^ Bottou, Léon; Cortes, Corinna; Denker, John S.; Drucker, Harris; Guyon, Isabelle; Jackel, L. D.; LeCun, Y.; Muller, U. A.; Sackinger, E.; Simard, P.; Vapnik, V. (1994). "Comparison of classifier methods: A case study in handwritten digit recognition". Proceedings of the 12th IAPR International Conference on Pattern Recognition (Cat. No.94CH3440-5). Vol. 2. Jerusalem, Israel. pp. 77–82. doi:10.1109/ICPR.1994.576879. ISBN 0-8186-6270-0.{{cite book}}: CS1 maint: location missing publisher (link)
  16. ^ a b Yadav, Chhavi; Bottou, Leon (2019). "Cold Case: The Lost MNIST Digits". Advances in Neural Information Processing Systems. 32. arXiv:1905.10498. Article has a detailed history and a reconstruction of the discarded testing set.
  17. ^ Decoste, Dennis; Schölkopf, Bernhard (2002). "Training invariant support vector machines". Machine Learning. 46 (1/3): 161–190. doi:10.1023/A:1012454411458.
  18. ^ facebookresearch/qmnist, Meta Research, 2024-09-23, retrieved 2024-10-25
  19. ^ Cohen, G.; Afshar, S.; Tapson, J.; van Schaik, A. (2017). "EMNIST: an extension of MNIST to handwritten letters". arXiv:1702.05373 [cs.CV].
  20. ^ NIST (4 April 2017). "The EMNIST Dataset". NIST. Retrieved 11 April 2022.
  21. ^ NIST (27 August 2010). "NIST Special Database 19". NIST. Retrieved 11 April 2022.
  22. ^ Grother, Patrick J., and K. K. Hanaoka. "NIST special database 19." Handprinted forms and characters database, National Institute of Standards and Technology 10 (1995): 69.
  23. ^ Xiao, Han; Rasul, Kashif; Vollgraf, Roland (2017-09-15). "Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms". arXiv:1708.07747 [cs.LG].
  24. ^ a b Cires¸an, Dan; Ueli Meier; Jürgen Schmidhuber (2012). "Multi-column deep neural networks for image classification" (PDF). 2012 IEEE Conference on Computer Vision and Pattern Recognition. pp. 3642–3649. arXiv:1202.2745. CiteSeerX 10.1.1.300.3283. doi:10.1109/CVPR.2012.6248110. ISBN 978-1-4673-1228-8. S2CID 2161592.
  25. ^ Kussul, Ernst; Tatiana Baidyk (2004). "Improved method of handwritten digit recognition tested on MNIST database" (PDF). Image and Vision Computing. 22 (12): 971–981. doi:10.1016/j.imavis.2004.03.008. Archived from the original (PDF) on 21 September 2013. Retrieved 20 September 2013.
  26. ^ Ranzato, Marc'Aurelio; Christopher Poultney; Sumit Chopra; Yann LeCun (2006). "Efficient Learning of Sparse Representations with an Energy-Based Model" (PDF). Advances in Neural Information Processing Systems. 19: 1137–1144. Retrieved 20 September 2013.
  27. ^ Ciresan, Dan Claudiu; Ueli Meier; Luca Maria Gambardella; Jürgen Schmidhuber (2011). "Convolutional neural network committees for handwritten character classification" (PDF). 2011 International Conference on Document Analysis and Recognition (ICDAR). pp. 1135–1139. CiteSeerX 10.1.1.465.2138. doi:10.1109/ICDAR.2011.229. ISBN 978-1-4577-1350-7. S2CID 10122297. Archived from the original (PDF) on 22 February 2016. Retrieved 20 September 2013.
  28. ^ Wan, Li; Matthew Zeiler; Sixin Zhang; Yann LeCun; Rob Fergus (2013). Regularization of Neural Network using DropConnect. International Conference on Machine Learning(ICML).
  29. ^ a b c SimpleNet (2016). "Lets Keep it simple, Using simple architectures to outperform deeper and more complex architectures". arXiv:1608.06037. Retrieved 3 December 2020.
  30. ^ SimpNet (2018). "Towards Principled Design of Deep Convolutional Networks: Introducing SimpNet". Github. arXiv:1802.06205. Retrieved 3 December 2020.
  31. ^ a b Romanuke, Vadim. "Parallel Computing Center (Khmelnytskyi, Ukraine) represents an ensemble of 5 convolutional neural networks which performs on MNIST at 0.21 percent error rate". Retrieved 24 November 2016.
  32. ^ a b Romanuke, Vadim (2016). "Training data expansion and boosting of convolutional neural networks for reducing the MNIST dataset error rate". Research Bulletin of NTUU "Kyiv Polytechnic Institute". 6 (6): 29–34. doi:10.20535/1810-0546.2016.6.84115.
  33. ^ Lindblad, Joakim; Nataša Sladoje (January 2014). "Linear time distances between fuzzy sets with applications to pattern matching and classification". IEEE Transactions on Image Processing. 23 (1): 126–136. Bibcode:2014ITIP...23..126L. doi:10.1109/TIP.2013.2286904. PMID 24158476. S2CID 1908950.
  34. ^ Keysers, Daniel; Thomas Deselaers; Christian Gollan; Hermann Ney (August 2007). "Deformation models for image recognition". IEEE Transactions on Pattern Analysis and Machine Intelligence. 29 (8): 1422–1435. CiteSeerX 10.1.1.106.3963. doi:10.1109/TPAMI.2007.1153. PMID 17568145. S2CID 2528485.
  35. ^ Kégl, Balázs; Róbert Busa-Fekete (2009). "Boosting products of base classifiers" (PDF). Proceedings of the 26th Annual International Conference on Machine Learning. pp. 497–504. doi:10.1145/1553374.1553439. ISBN 9781605585161. S2CID 8460779. Retrieved 27 August 2013.
  36. ^ "RandomForestSRC: Fast Unified Random Forests for Survival, Regression, and Classification (RF-SRC)". 21 January 2020.
  37. ^ "Mehrad Mahmoudian / MNIST with RandomForest".
  38. ^ Decoste, Dennis; Schölkopf, Bernhard (2002). "Training Invariant Support Vector Machines". Machine Learning. 46 (1–3): 161–190. doi:10.1023/A:1012454411458. ISSN 0885-6125. OCLC 703649027.
  39. ^ a b Patrice Y. Simard; Dave Steinkraus; John C. Platt (2003). "Best Practices for Convolutional Neural Networks Applied to Visual Document Analysis". Proceedings of the Seventh International Conference on Document Analysis and Recognition. Vol. 1. Institute of Electrical and Electronics Engineers. p. 958. doi:10.1109/ICDAR.2003.1227801. ISBN 978-0-7695-1960-9. S2CID 4659176.
  40. ^ Ciresan, Claudiu Dan; Ueli Meier; Luca Maria Gambardella; Juergen Schmidhuber (December 2010). "Deep Big Simple Neural Nets Excel on Handwritten Digit Recognition". Neural Computation. 22 (12): 3207–20. arXiv:1003.0358. doi:10.1162/NECO_a_00052. PMID 20858131. S2CID 1918673.
  41. ^ Romanuke, Vadim. "The single convolutional neural network best performance in 18 epochs on the expanded training data at Parallel Computing Center, Khmelnytskyi, Ukraine". Retrieved 16 November 2016.
  42. ^ Romanuke, Vadim. "Parallel Computing Center (Khmelnytskyi, Ukraine) gives a single convolutional neural network performing on MNIST at 0.27 percent error rate". Retrieved 24 November 2016.
  43. ^ Hu, Jie; Shen, Li; Albanie, Samuel; Sun, Gang; Wu, Enhua (2019). "Squeeze-and-Excitation Networks". IEEE Transactions on Pattern Analysis and Machine Intelligence. 42 (8): 2011–2023. arXiv:1709.01507. doi:10.1109/TPAMI.2019.2913372. PMID 31034408. S2CID 140309863.
  44. ^ "GitHub - Matuzas77/MNIST-0.17: MNIST classifier with average 0.17% error". GitHub. 25 February 2020.
  45. ^ An, Sanghyeon; Lee, Minjun; Park, Sanglee; Yang, Heerin; So, Jungmin (2020-10-04). "An Ensemble of Simple Convolutional Neural Network Models for MNIST Digit Recognition". arXiv:2008.10400 [cs.CV].

Further reading

Retrieved from "https://en.wikipedia.org/w/index.php?title=MNIST_database&oldid=1282256756"