There is no evidence to suggest that any part of the olive tree (Olea europaea) is poisonous to animals.
Some other, unrelated tree species with olive in their common names or "olea" - Latin for olive - in their botanical or common names, are toxic and may be mistaken for olive trees.