Use another data structure to track connections by liveness, which you can measure by timestamping on each RX event. Relax the requirement that an connection idle timeout of 60 seconds must fire in exactly 60,000 milliseconds; 60, 61, 65, nobody cares.