"a stimulating liquor comprised of spirits of any kind, with sugar, water, and bitters included"
After all, bitters are just an infusion of plant material. What is tea but an infusion of plant material?
On the other hand, bitters generally include many different plants and seems to be quite concentrated, while tea is just one plant and isn't as concentrated.
I guess asking whether tea is bitters is kind of like asking if hot dogs or tacos are sandwiches...
(Also that some plants will have poisons that might be extracted in one method but not another, so you always want to research first.)
Obviously oil-based bitters aren't going to be compatible with cocktails because they won't mix in most cases.
But you can definitely make water-based bitters, if you want to avoid the flavors that alcohol extracts. But I think you'd then still add alcohol afterwards simply for preservation purposes, since you want a bottle of bitters to last for months/years and not need to bother with refrigeration.