I think I caught two rare sylestian's in the Spring Zone, both had over half their health and I didn't get any experience points and I didn't defeat any enemies... But they aren't in my stable! Where are they?

When you use a trap on a sylesti, there's 3 things that can happen:
-trap fails, pet remains, you can try to trap it again
-trap fails and the pet runs away, battle ends with no pet/ experience/ anything
-trap succeeds, pet is added to your stables with a link to it in the battle log

Most likely for your case, the pet ran away. Next time when the battle ends, look at your battle log. If there's a link to the pet there, it means your trap succeeded, and the pet is in your stables.