I will note down Nicks comment above as it ended up being the answer.
In order to control Glimpse behaviour for a particular browser the, the option is to write your own custom runtime policy as mentioned in https://github.com/Glimpse/Glimpse/issues/272.
In my case, one of the master files was still having the call Html.GlimpseClient() which obviously injected the client regardless of what happens with the runtime policy.