IMO it’s a good fit to use them together. YADM lets me do things like easily setting per-host configurations, encrypting secret files, etc.
My new host setup is to use YADM to pull my dot files, then install nix and run home-manager to install everything else. YADM even supports bootstrapping to automate that for me