Yes you should. A few days ago I created a TCP socket/server application, and I got the same problem. I fixed it by starting to send keep alive packets.
If you send keep alive packets, your problem will disappear.
I've heard some people say that the OS will send keep alive packets for you, I am not very familiar with this, but sending keep alive packets explicitly worked for me