Simon Grayson's men left Gigg Lane with just a point this afternoon despite taking an early three goal lead.
Sean Morrison's opener and a brace from Rhodes had put Town firmly in the ascendancy with 30 minutes of the game gone.
However, Steven Schmacher hit back immediately for the home side with a deflected effort before Lenell John-Lewis and Ashley Eastham completed the comeback for the Shakers in the second 45.
Simon Grayson made three changes to the side that drew at Stevenage in mid-week. Antony Kay and Scott Arfield were handed starts in midfield at the expense of Danny Cadamarteri and Oscar Gobern.
Unsurprisingly, npower League 1's top goal scorer Jordan Rhodes was reintroduced back into the line-up following his mid-week international exploits with Scotland.
Town, playing in their change strip of red and black, started the game on the front foot and forced the Bury defence into a number of early blocks when Joey Gudjonsson and central midfield partner Antony Kay tried from outside the 18 yard box.
The home side's 'keeper Trevor Carson was called upon for the first time in the ninth minute when Jordan Rhodes headed Jack Hunt's cross weakly at the stopper. He was beaten in the 13th minute though when Jamie McCombe met Danny Ward's free-kick with a header, but the ball flashed agonizingly wide of his left upright.
Antony Kay, who was making his first start for the Terriers since the 30th December, almost caught Carson on his heels with 16 minutes played when he neatly controlled Scott Arfield's lay-off before firing just over on the volley with his left.
GOAL - The visitors deservedly opened the scoring in the 22nd minute when Sean Morrison poked in his first goal for the Club. Morrison's centre back partner Jamie McCombe caused problems for Carson from Ward's corner when he challenged him in the air and when the ball fell free, the powerful centre-half broke from his marker and stabbed in at the back stick.
GOAL - Fresh from international duty, Jordan Rhodes headed in Town's second following a superb counter attacking move in the 25th minute. Rhodes started the counter after collecting Alex Smithies' long throw out before passing inside to Arfield. The Scotsman neatly slid the ball wide to Ward who collected his second assist of the afternoon finding Rhodes perfectly with his right to head home.
GOAL - Three minutes later and the Scottish international was at it again - smashing the ball past a hapless Trevor Carson. The home side were chasing shadows at times and following good work from Kay, Arfield, Gudjonsson and Novak, Jordan Rhodes wriggled free of his marker on the edge of Bury's box before rifling home his second of the afternoon with his left.
GOAL - The home side hit back immediately in the 31st minute when Steven Schumacher's deflected 30 yard free-kick wrong footed Smithies. Joey Gudjonsson provided the Shakers with the opportunity after fouling Phil Picken and fortunately for the home side, the Captains' free-kick took a deflection to beat Smithies.
BOOKING - Jack Hunt received the first caution of the game in the 35th minute after arguing with referee Graham after he decided to penalise the right-back for a foul on David Worrall. From the resulting set-piece former Leeds United striker Mike Grella turned the ball just wide of Smithies' right upright.
HALF-TIME - SHAKERS 1 TERRIERS 3
SUBS - Calum Woods was introduced at half-time for Jack Hunt, who had collected a yellow card in the first half. For the home team David Amoo replaced Mark Carrington, who looked to be struggling in the first 45 with an injury.
The home side started the second half well and forced Smithies into a superb low save when substitute David Amoo chased down his block and fired low from close range. Smithies was alert to the danger and showed great athleticism to turn the ball away - Jamie McCombe completed the clearance with Mike Grella hunting him down.
The Terriers broke the home side's constant pressure with two well hit efforts. Firstly, Icelandic midfielder Joey Gudjonsson tried his luck from range and then, following good work from Novak, Ward fired straight at Carson.
As the clock ticked past the hour mark, the Terriers were well and truly in control of the game and should have put it to bed when Ward capitalised on a breakdown in communication between Mark Hughes and Worral. The winger drove in on goal and from 18 yards rattled Carson's cross bar with a fantastic shot.
SUB - With just over twenty minutes remaining Richie Barker introduced Lateef Elford-Alliyu at the expense of debutant Mike Grella.
GOAL - Bury closed the deficit to one in the 70th minute when Lenell John-Lewis scuffed in Elford-Alliyu's cross. The substitute showed good pace to get to the by-line before picking out Bury's lone strike man who miss hit his effort, but still saw it lob over Smithies and into the back of the net.
BOOKING - Ashley Eastham received Bury's first card of the afternoon when he cynically blocked Lee Novak.
SUB - Danny Ward was replaced by Oscar Gobern with just under 15 minutes remaining. As a result Town went to three central midfielders Kay, Joey and Gobern with Arfield and Novak flanking Rhodes.
SUB - With eight minutes remaining Bury's goal scorer John-Lewis was replaced by Andy Bishop.
Bury were within inches of completing a superb comeback with 84 minutes on the clock when Eastham headed Sweeney's back post cross agonisingly wide.
SUB - Grayson made his final change in the 88th minute of the game - the hard working Lee Novak was replaced by Gary Roberts.
GOAL - Bury deservedly completed their comeback in the first minute of injury time when Ashley Eastham this time connected perfectly with Peter Sweeney's cross to score at the back post.
FULL TIME - BURY 3 HUDDERSFIELD TOWN 3
Town's team (4-4-2):
Alex Smithies; Gary Naysmith (c), Jamie McCombe, Sean Morrison, Jack Hunt (Calum Woods 45); Danny Ward (Oscar Gobern 78), Antony Kay, Joey Gudjonsson, Scott Arfield; Jordan Rhodes, Lee Novak (Gary Roberts 88)
Unused substitutes: Ian Bennett, Alan Lee
Bookings Town: Jack Hunt, 35
Bookings Bury: Ashley Eastham 78,