This project aims to develop a new spatial-HMM for automatically classifying and annotating images. Our model is a 2D generalization of HMM and differs from the Hidden Markov-mesh based approach. We regard visual features extracted from images as observations, and the semantic concepts are hidden states which govern the underlying probabilistic characteristics of observations. With the aim to model the context of semantic features, we employ a second-order neighborhood system. The spatial relationships of features are encoded in the transitions between hidden states in our model. Hence we term our approach Spatial Hidden Markov Model or SHMM. Given a matrix of feature vectors for all blocks in an image, the most appropriate semantic labels determined by our models are used for annotation. Experiments were carried on both histological images and natural images. Our experiments showed that our model is superior to HMM in both recognition and annotation accuracy.