Scatter plot is one of the well‐known charts and is frequently embedded in different types of documents such as articles, books, and dissertations. However, the information given in the scatter plots can’t be directly noticed by visually impaired individuals, because they are usually in an image format, and so they are not naturally readable by machines. To solve this problem, this paper proposes a system that can extract visual properties from scatter plot images using deep learning and image processing techniques. It is the first study that automatically classifies scatter plots in terms of two aspects: degree of correlation (strong or weak) and types of correlation (positive, negative, or neutral). In the experimental studies, alternative convolutional neural network (CNN) architectures were compared on both synthetic and real‐world datasets in terms of accuracy, including Residual Networks (ResNet), Alex Networks (AlexNet), and Visual Geometry Group (VGG) Networks. The experimental results showed that the proposed system successfully (93.90%) classified scatter plot images to help visually impaired users understand the information given in the graph.