Your server(s) are with OVH, which certainly does not mean lag-free in any sense of the matter.

1GBps or not, OVH's network is filled with Torrent clients/seedboxes and the quality of the traffic is extremely low. So whilst fast, response times (ie. ping) is still fairly high. This is even more amplified from you only having a basic server with them, the QoS is automatically lowered / de-prioritised.

I'm not saying you offer a bad service or anything, just don't get hung up on the 1GBps shit, because that's only connectivity with the internal network.