People with grapheme-color synesthesia perceive enriched experiences of colors in response to graphemes (letters, digits). In this study, we examined whether these synesthetes show a generic associative memory advantage for stimuli that do not elicit a synesthetic color. We used a novel between group design (14 young synesthetes, 14 young, and 14 older adults) with a self-paced visual associative learning paradigm and subsequent retrieval (immediate and delayed). Non-synesthesia inducing, achromatic fractal pair-associates were manipulated in visual similarity (high and low) and corresponded to high and low memory load conditions. The main finding was a learning and retrieval advantage of synesthetes relative to older, but not to younger, adults. Furthermore, the significance testing was supported with effect size measures and power calculations. Differences between synesthetes and older adults were found during dissimilar pair (high memory load) learning and retrieval at immediate and delayed stages. Moreover, we found a medium size difference between synesthetes and young adults for similar pair (low memory load) learning. Differences between young and older adults were also observed during associative learning and retrieval, but were of medium effect size coupled with low power. The results show a subtle associative memory advantage in synesthetes for non-synesthesia inducing stimuli, which can be detected against older adults. They also indicate that perceptual mechanisms (enhanced in synesthesia, declining as part of the aging process) can translate into a generic associative memory advantage, and may contribute to associative deficits accompanying healthy aging.