Olney has historical associations with the poet William Cowper (1731 - 1800) and John Newton (1725 - 1807). William Cowper lived in Orchard House, which is now the Cowper and Newton Museum, between 1768 and 1786. John Newton became curate of Olney's Parish Church between 1764 and 1800. They are both commemorated in the Cowper and Newton Museum in the Market Place.

Together they wrote the 380 hymns that make up "The Olney Hymn Book". Best known are: 'How sweet the name of Jesus sounds', 'Glorious things of these are spoken' and 'Amazing Grace' by John Newton and '0 for a closer walk with God', 'Hark my soul it is the Lord' and 'God moves in a mysterious way', by William Cowper. John Newton, after trading in slaves between Africa and America, was converted to the Christian faith (inspiring him to write the famous hymn 'Amazing Grace') and was ordained a priest. His remarkable ministry bought large crowds to Olney to hear him preach from the pulpit that is now preserved in the Church.