Either firewall the service, via iptables, or the ACL system that rsync supports natively:

     [foo]
          path = /srv/foo
          hosts allow 1.2.3.4, 10.20.30.0/24

In addition to that you can use passwords with rsync, or avoid exposing it directly and use ssh as a transport - secured with keys.