I used to live in Norseman which is a couple of hours north of Esperance. Never got to ride in Esperance though.
I remember that AMB did a article on it once, riding up that hill or mountain (forget the name) in the national park. I walked up it and at the time thought it was pretty darn steep and beyond my ability - maybe they just rode down? It's like a big rock anyway.
So check out the National Park, near that Hellfire Beach or something.
If you end up going through Norseman it's not bad exploring the tracks that go from the lookout, mostly just old mining land, I used to ride around there all the time.