The production of high-quality food products needs an efficient method to detect defects in food, this is particularly true in the production of apples. Hyperspectral image processing is a popular technique to carry out this detection. However, the stem and calyx of the apple provoke frequent detection errors. We analyze the spectrum of our apple data set, propose an algorithm that uses the average of the principal components of two regions of the spectrum to identify the defects, and couple this detection routine with a two-band ratio that discriminates the calyx and stem. Our study considers the spectral range between 403 nm and 998 nm. Our results include the detection of scab, bruise, crack, and cut with and without stem and calyx. We describe all the necessary parameters provided by our spectral analysis. Our algorithm has an overall accuracy of 95%. We conclude that our algorithm effectively detects defects in the presence of stem and calyx.