I think you probably could use the "History.js" tool, which provides a "statechange" event that you can listen for. The tool fires that event either when the forward/backward buttons are pressed or when you use pushState/replaceState.
https://github.com/browserstate/history.js/
Of course, now you've traded for a different problem, which is that you have the event being fired for both types of actions, and you often want to distinguish between them and that's more difficult. So you have to hack your own solution to distinguishing them. Most people seem to set some global flag whenever they are about to call pushState/replaceState, and then check for this flag in the event handler. If the flag is set, then the event came from your own code, if not, it came from the browser buttons.