First and most importantly, inbreeding has no negative effect on pets whatsoever.
You cannot breed pets with their parents, siblings, half-siblings, or children; however, they /can/ be bred with their grandparents, cousins, or grandchildren. Inbreeding increases when they're bred with those relations (or any more distant relations, too).
It's commonly found in trait breeding projects. For example:
https://www.sylestia.com/view/lineage/?pet_id=3272477
This pet is the result of one of my projects. I started with only three regular essences, two carries each, and bred up to four visible traits. As a result, every child of this project is inbred.
I can't tell you exactly how the percentage is calculated, just that it goes up when two related pets are bred with each other. But it doesn't hurt the pets in any way.