It all looked good for the Terriers at the Banks's Stadium when Theo Robinson's eighth goal of the season put Town ahead early in the second half, but two goals from the home side condemned Town to their second consecutive away defeat.
Robinson (pictured) netted his goal just two minutes after the restart following a scrappy first-half, but a penalty from striker Darren Byfield for a disputed handball in Town's area preceded a late strike from former Town midfielder Dwayne Mattis that sealed all three points for Chris Hutchings's side.
Town started the game in confident fashion and almost crafted a very good chance as early as the 5th minute when Skarz almost picked out an unmarked Jordan Rhodes in the area, but the Saddlers had the first shot of the match a minute later when striker Darren Byfield blasted wide of Smithies' right-hand post from long range.
Walsall's giant centre-back Clayton McDonald - who stands at 6' 6" - then caused the Town defence a few hesitant moments as he tried to nod home a free-kick from Weston on the right side, but the referee immediately gave a free-kick for a foul on Peter Clarke.
Alex Smithies had a rare lapse of concentration in the 12th minute, spilling a Nicholls corner delivered from Town's right side under little pressure; the ball fell behind the 'keeper, but Kay was on hand to clear to safety off the line.
Walsall were causing Town a few anxious moments through set-pieces - they were taking every opportunity to launch the ball into the area on the tiny Banks's Stadium pitch. One raking ball from Weston in the 22nd minute found the head of striker Deeney and as the ball dropped on the penalty spot, former Town midfielder Dwayne Mattis scuffed a left-foot volley that struck the outside of the post as it dribbled wide.
Striker Deeney then hit a 25-yard shot over Smithies bar on the stroke of the 24th minute after receiving a neat pass from right winger Nicholls.
Town created their best chance to date in the 31st minute after some great wing play from Anthony Pilkington. After receiving a pass to feet from Skarz, he tormented full-back Weston down the left before racing to the by-line and crossing; the ball looped towards the unmarked Robinson at the back post, but from eight yards out he placed his header straight at the goalkeeper.
Suddenly the Terriers were on top and just two minutes later another chance came. Peltier turned the ball down the right channel for Robinson and his early cross to the near post found Rhodes. Unfortunately, he placed his volley into the fans behind the goal while under heavy pressure from a defender.
Smithies was required to push away a Richards free-kick in the 37th minute, but it was an easy save for the England under-19 international. Soon after normal service was resumed when Robinson ran at the home defence, but he smashed a wild 25-yard drive high and wide with Drinkwater screaming for a pass to his right.
BOOKING - Walsall striker Troy Deeney entered the referee's notebook in the 39th minute for a dive inside the Town penalty area - he went down under a challenge from Andy Butler, but the referee had no hesitation in reprimanding him.
With a minute of the half remaining Town won a corner due to the persistence of Theo Robinson; Drinkwater found Clarke at the near post who looped a volley to the back post, where Skarz headed the ball back across goal - Walsall cleared the ball to the edge of the box, where Goodwin sent a powerful shot just over the crossbar.
HALF-TIME: WALSALL 0 HUDDERSFIELD TOWN 0
A speedy break for Town in the opening minute almost led to an early opportunity - Drinkwater found Rhodes' arcing run to the right flank but as he tried to square to the unmarked Robinson in the area, he under-hit his cross.
GOAL - It didn't matter a minute later as Theo Robinson netted his eighth goal of the season. A great pass from Jordan Rhodes found the speedy striker, who had sprung the offside trap despite the protests of many Walsall players and officials; he entered the area before calmly rounding goalkeeper Gilmartin and beating the last covering defender with his shot in front of the travelling Town faithful.
Town created a chance to double their lead in the 56th minute when Walsall didn't pick up Rhodes as Peter Clarke looped the ball over the top, but he snatched at his shot from a narrow angle and only succeeded in ramming the ball into the stand.
Richards and Byfield then combined well for the home side down Town's right flank but as the latter tried to square the ball across the face of goal, Kay positioned himself well and cleared at the near post.
Rhodes then showed good strength at the other end to hold off McDonald in the 62nd minute - he played the ball into the area for Robinson who in turn found Drinkwater with a low pass, but he fired his snapshot over the bar.
Walsall were then awarded a 64th minute penalty for an unfortunate handball by Joe Skarz. Lee Peltier made a mess of a clearance down the right side, allowing Byfield to cross deep to the back post; winger Nicholls placed a firm header back across goal that struck the arm of the left-back according to the assistant referee.
GOAL - Up stepped Darren Byfield to send Alex Smithies the wrong way and fire the ball into the right-hand side of the goal.
SUB - Lee Clark was planning his first substitution before the equaliser and carried it though afterwards - Gary Roberts replaced Anthony Pilkington on the left in the 67th minute.
Town fought back well and enjoyed their best passing move of the game in the 71st minute when Goodwin and Kay combined to find Robinson with a pass to feet. He threaded a through ball to strike partner Rhodes inside the area, but he had two shots blocked well by the Walsall centre backs.
SUB - Robbie Simpson replaced Jordan Rhodes in Town's second change on 74 minutes.
BOOKING - Jim Goodwin was shown a yellow card with ten minutes left, with the referee indicating it was for an accumulation of fouls.
SUB - Walsall introduced big Sam Parkin in place of Darren Byfield a minute later.
BOOKING - Joe Skarz followed Goodwin into the book for a pull on Alex Nicholls' shirt down the touchline in the 84th minute - it was the left-back's third yellow of the campaign.
GOAL - Walsall then scored a second goal in the 87th minute to apparently secure the win through midfielder Dwayne Mattis. A ball down the left channel out of defence found striker Deeney, who turned it back to winger Richards; his excellent deep cross found the head of substitute Parkin who cushioned the ball down to the edge of the box for the former Town man to drive past the unsighted Smithies and into the far corner.
Town almost netted a late equaliser when Goodwin's deflected shot fell to Robbie Simpson, but 'keeper Gilmartin did excellently to block the striker's powerful left-footed effort.
SUB - Walsall swapped Alex Nicholls with former Town loanee Steve Jones in the final minute.
Town threw skipper Peter Clarke forward to join Simpson and Robinson in added time, but it was to no avail as Walsall held out for the win.
FULL-TIME: WALSALL 2 HUDDERSFIELD TOWN 1
Town's Team (4-4-2):
Alex Smithies; Joe Skarz, Andy Butler, Peter Clarke (c), Lee Peltier; Anthony Pilkington (Gary Roberts 67), Antony Kay, Jim Goodwin, Danny Drinkwater; Jordan Rhodes (Robbie Simpson 74), Theo Robinson
Unused Substitutes: Matt Glennon, Nathan Clarke, Lee Novak, James Berrett, Lionel Ainsworth
Town Bookings: Jim Goodwin (80), Joe Skarz (84)
Walsall Booking: Troy Deeney (39)