Abstract: Letters, foundational units of alphabetic writing systems, are quintessential to human culture. The ability to read, indispensable to perform in today’s society, necessitates a reorganization of visual cortex for fast letter recognition, but the developmental course of this process has not yet been characterized. Here, we show the emergence of visual sensitivity to letters across five electroencephalography measurements from kindergarten and throughout elementary school and relate this development to emerging reading skills. We examined the visual N1, the electrophysiological correlate of ventral occipito-temporal cortex activation in 65 children at varying familial risk for dyslexia. N1 letter sensitivity emerged in first grade, when letter sound knowledge gains were most pronounced and decayed shortly after when letter knowledge is consolidated, showing an inverted U-shaped development. This trajectory can be interpreted within an interactive framework that underscores the influence of top-down predictions. While the N1 amplitudes to letters correlated with letter sound knowledge at the beginning of learning, no association between the early N1 letter response and later reading skills was found. In summary, the current findings provide an important reference point for our neuroscientific understanding of learning trajectories and the process of visual specialization during skill learning.