I’d tend to agree that using the browser history doesn’t feel like the right solution for this kind of undo/redo functionality. In my mind, undo/redo actions are too fine-granular, and pollute the history, which can be very inconvenient.
Another potential approach could be to offer a custom-built widget for the undo/redo stack in the UI – in the simplest case, for example, a dropdown. It could also be persisted somewhere, e.g. in local storage, so that it isn’t lost when you navigate away.