Hard disagree. That's one person out of the billions that is your potential adversary and highly unlikely at that. It is highly improbably that person with either know they have your old IP or that they would be a risk. Even limiting login to a ASN# or large CIDR block is monumentally better than allowing the entire internet to brute force ones account.
As an example I have a few services that I do not really care about but I still limit logins to the CIDR of my ISP. That means most of my country and all of the other countries can bang away or pound sand all day and night for millions of years and they will get nowhere.