I have wasted so many traps on pets because they always fail. Is there some strategy as to when you use the trap or which pets you should try to trap?

I believe it's mostly luck, unfortunately.

EDIT: Like Flute said, I try to lower the health before capturing. I have no idea if this actually makes any difference though? xD

Some people lower the health of the pet before/during capture. Kinda helps, but the drawback is that if you're not careful about how your attacks cost, you could kill the pet. xD

It's hard to muster up gold for traps at the beginning, though personally, I always prepare myself with a sizable amount of traps.
When I was a beginner, I allotted myself 3 traps per outing. If I used up all my traps, the pet wasn't worth it, and I just killed it. (Of course I celebrated if I caught it within the 3 traps I had lol)

As you progress through the game, you'll be able to make more money for more traps. The 3 trap limit that I allotted to myself slowly grew to 5, to 6, to 9, then later 21. I just throw traps at the pet til they get caught or flee; could take 3, could take 5, and could run away and go all to waste. x:

I'd say give yourself a set amount of traps per pet, like how I did. If it takes more than ___ traps, then it isn't worth it, sort of mentality. At the same time, keep progressing with the game to accumulate more gold.

Originally lowering health did not help, but as of one of the updates it does, just be careful and be sure you want the pet.

@Katasaur here is the update http://www.sylestia.com/forums/?t=811

Oooh cool thanks Amaya! Glad to know my efforts aren't in vain. xD