I've had similar struggles in the past. My solution at the time was a bib file quasi-managed by hand using JabRef. I would also periodically do some automatic updating using a script I wrote[0]. The script uses the NASA Astrophysics Data System to look for updates to papers (they cross-match arXiv preprints with the eventual publications) and updates the bibtex accordingly. It mostly worked, but there were some bugs (the last 10% takes 90% of the time).

I've since resorted to more hand-checking of references when I'm publishing a paper. Rreference management is tedious, but I've found that it is overall is a minor fraction of the time spent working on a paper. So it doesn't seem worth spending too much effort automating.

[0] https://github.com/privong/PaperTools/blob/main/bib_update.p...