Challenges facing pen tip detection for applications supporting the visually impaired

Abstract

Paper-based forms (such as tax forms) present a significant barrier for the visually impaired, whom traditionally require the assistance of a third party to fill in such forms. While there have been efforts in creating systems providing the visually impaired with independence in form-filling, the critical problem of detecting a user's pen effectively has remained unsolved. Given that little research exists in pen tip detection, we examine a range of algorithms that could be used for the task -- borrowing from other fields, such as fingertip tracking, as well as adapting shape detection techniques to finding pen tips. We choose algorithms that could ultimately be implemented on consumer mobile devices and compare their performance in terms of accuracy and speed in both a simplified environment as well as on a real form containing text, lines and fields. In doing so, we demonstrate that existing algorithms are inadequate and the problem is not yet solved, despite being one of importance for a significant proportion of the population.