Rickie Weeks doubled just inside the first base line with one out before going to third on a wild pitch by reliever Adam Warren (1-2).
Reynolds, who briefly played for New York last year, held his right index finger in the air after his hard chopper bounced past diving third baseman Yangervis Solarte at third to drive in Weeks two batters later. Reynolds then was mobbed by his teammates near first base.
The Yankees were one strike away from defeat in the top of the ninth when Mark Teixeira reached out for a low and away, full-count pitch from Rodriguez for a homer to right. It was the first run allowed all season by the veteran closer.
Rodriguez (1-0) did pick up the win. The Brewers' typically stingy bullpen gave up two runs to New York, including Solarte's run-scoring single in the seventh that drew New York within a run.
This time, the Brewers' bats picked up the relievers.
It was the kind of scrappy victory manager Ron Roenicke was looking for with third baseman Aramis Ramirez sidelined by a hamstring injury and outfielder Ryan Braun already on the disabled list with an oblique injury.
Until the seventh, the Brewers appeared to be in control. They scraped out three runs in the sixth, highlighted by pinch-hitter Jonathan Lucroy's single and Logan Schafer's RBI double, to take a 5-3 lead. Weeks, who finished 3 for 5, drove in the third run that inning with a fielder's choice.
David Phelps had a so-so outing in his second start of the season after making nine appearances out of the bullpen for the Yankees. Starting in place of the injured Michael Pineda, he allowed eight hits and walked three in five-plus innings, allowing the leadoff runner to reach base five times.
But that performance might be good enough, for now, for a Yankees rotation battered by injuries. With Ivan Nova (elbow) already out for the season, the Yankees announced before the game that struggling left-hander CC Sabathia would join Pineda (upper back) on the disabled list with a right knee injury.
The lineup gave Phelps support early while taking advantage in a three-run first of starter Matt Garza, who allowed three walks and two hits in the inning, including Kelly Johnson's two-run double. Garza settled down to allow six hits, four walks and three runs in five innings.
Prior to the game, the Brewers gave Yankees shortstop Derek Jeter a gold-plated bat and a $10,000 check for his Turn 2 Foundation, which promotes healthy lifestyles among youth. The gifts were presented by Dick Groch, a special assistant to general manager Doug Melvin. Groch was the Yankees scout in 1992 who recommended the team draft Jeter in the first round that year.
NOTES: Yankees C John Ryan Murphy groundout to second was overturned to an infield single in the eighth after manager Joe Girardi requested a replay review, which took 53 seconds. ... Carlos Gomez was hit in the left forearm by a pitch from Phelps in the third, causing him to writhe in pain a few seconds. He stayed in the game. ... The Yankees come home for the Subway Series against the Mets, sending Huroki Kuroda (2-3) to the hill. The Brewers have Monday off, with Marco Estrada (2-1) scheduled to start Tuesday against the Pirates.