There are also patterns that can really bloat compile time, especially heavy use of procedural macros to autogenerate code. Using serde to serialize big, complex data types can be a big compile time hit. Monomorphization can also be a problem, but fortunately careful use of boxing of dyn traits can help a lot with both code size and compile time (miniserde is an alternative to serde that's more optimized on these dimensions).
There's also continual and ongoing work to improve compile times. So overall I would say it's definitely a factor, but not necessarily a showstopper.