Despite a complete squad overhaul in the post-relegation summer, the Saddlers' form hasn't improved and they arrive at this weekend's clash with the Stags in the midst of a dismal run of form.
Defeats to Salford City, Leyton Orient, Cheltenahm Town, and Oldham Athletic have seen tensions rise as the Saddlers faithful struggle to stick behind a team which appears to be repeating the same mistakes each and every week.
And the gaffer, who takes on his hometown club this weekend, insisted there will be no excuses and admitted he expects to receive criticism as his team continues to under-perform.
When asked whether it was his toughest time as a manager, Clarke admitted it was but is determined to turn things around, with or without the help of the current crop.
Speaking via the Express & Star, Darrell said; "It is. But you don't get out of these spells by feeling sorry for yourself. You have to take your flack and stick. I'm a rubbish manager at the minute, that's what I am.
"What you have to do is turn yourself into a better manager. I expect massive questions to be asked of me, I understand it. There's no excuses from me or my players.
"All I'm going to do is keep working hard to be a better manager for Walsall. Give up? There's nothing to give up on. I've been in the job and we've had twelve weeks of competitive football.
"Are we doing rubbish, yes we are. I'm not giving up until we get the club back where we should be, end of."
Another element to the Saddlers' recent slump in form has been the increase in individual errors, with several of the recent goals coming after an error, adding yet more fuel to the fire of frustration which is spreading among the fans.
Mansfield-born Clarke has pulled few punches over recent weeks going as far as to question his players' desire and ability, but insists the squad remains behind his vision for the club, even if a few may need to toughen up if they are to make it in the game.
A clearly frustrated Clarke continued; "Don't give me this old 'player power', it's a load of rubbish. My players are fully behind what we're trying to do here, make no bones about that.
"They have to be better, I'm not going to cuddle them, they know that. I'm not a cuddling manager. If they're going to struggle in the modern world with confidence, they go on social media, their Twitter, they read newspaper articles.
"The criticism is there and if they can't handle the criticism out there and from a manager when they're not doing well, then they shouldn't be footballers should they. Go and stack shelves at ASDA.
"I'm not having it, there's no player's feeling 'oh it's not fair', they need to be working doubly hard to make sure we turn it around. If they don't like that, there's the door, get out the door.
"Soft players don't win you promotions, soft players don't get their careers where they need to be. If my players need me to be nice and soft and say everything is going to be OK, I think they need to get another manager in.
"It's not going to happen, they need to do better. Staff need to do better, I've got to do a lot better, everybody at the football club has got to do a lot better. If they can't handle that, they'll be on their way, trust me."