You don't need to use a different server port.
If the server is implemented properly then it shouldn't matter. The server can distinguish between different socket connections.
Each connection is identified by <server port, client ip address, client port>. The client port of each connection is different.