Billy struck on the stroke of half time following a period of sustained Bury pressure after the home team had made the early running.
Indeed, Diamonds should have taken the lead after only three minutes but somehow Glynn Garner got down to push away a header from danger-man Onandi Lowe.
Lowe proved a real menace in the air and headed over at least half a dozen occasions as the Bury goal had a charmed life.
In the second half Rushden piled forward and it seemed only a matter of time before they found an equaliser.
However, Bury held firm and might have doubled their advantage in the closing stages when substitute David Nugent's close-range attempts were superbly saved by Billy Turley.
Referee Thorpe played six minutes of injury time and despite a number of close shaves Bury held firm and Garner kept his first clean sheet in the League this season.
Tweet
