Despite on-loan Salford City striker Rory Gaffney grabbing his first goal in a Saddlers shirt, goals from Eoin Doyle and Diallang Jaiyesimi proved to be enough to secure all three points for the promotion-chasing Robins.
The result meant that Walsall registered their sixth defeat in a row and edged closer to the foot of the League Two table, but Clarke believes his side simply have to keep going as they go in search of some much-needed points.
Speaking to the assembled media, Darrell said; "I am upset, I think my team has given me everything today. We deserved something out of the game and I am upset. I’m a passionate guy, it’s killing me.
"I’m a hard working football manager that doesn’t win for himself, I try and win for the fans, the board and the players. We’re falling short so I am going to be upset and angry, but we have to keep going."
Mansfield-born Clarke, who saw his current team lose to his hometown side last weekend, went on to criticise his team for giving away yet more 'sloppy' goals as they have now gone seven games without keeping a clean-sheet.
The manager added; "Should have, could have, didn't. It’s as simple as that isn’t it. It’s a real sloppy couple of goals straight at the start of the game. In the ascendancy most of the half, make them change shape.
"Second half, yet again, sloppy goal. We get back into the game, the goalkeeper pulls over a worldie at the end, just the way it’s going at the minute for us. I keep saying it every week, but we have to go again.
"We got a little bit panicky, a bit rushed in our approach. We changed things around to freshen it up. To come to Swindon, who are obviously flying at the minute, and not get something out of that game today, I felt for my lads.
"But at the end of the day we can’t give the goals we’re conceding. We’ve been saying that over the course of the last six defeats and today is another one of those days.
"Very frustrated and disappointed. Like I keep saying each week, I get bored of saying it but I have to face it, we have to regroup and go again."