I see your concerns about performance, but 1) temporal objects are so small that there are plenty of optimization opportunities for the JS engines to eliminate or greatly reduce allocations.
2) The larger and older your project, and the more people are working on it (including authors of third-party dependencies), the murkier data ownership becomes. With that you are more likely to slip into defensive programming practices. "I have a `Date` object that is someone's birthday, but I need to pass it to multiple functions and then serialize it to storage. But I can't tell for sure what those functions are doing with that value. Should I serialize the date right away and persist that value later? Should I just create copies of the object to pass to the functions?"
But birthdays don't mutate, so now you're worrying about a thing that shouldn't be an issue! In the unlikely case that someone's birthday date is corrected, May 9th doesn't suddenly turn into July 15th. In the real world you don't drag the red circle drawn in marker on a paper calendar with your finger from one cell to another, or white out the day number and write in another one. You cross the old circle off and draw a new one in the right place.
My background is in backend services, hundreds of threads running on dozens of cores. The peace of mind that comes with being able to pass values to async functions and thread pool executors, and knowing that they won't turn into a pumpkin at midnight is a very real thing. There's a performance price to pay, sure, but it's so tiny compared to the wins in development and debugging times. And in the particular case of date/time objects, most things are around 8-16 bytes anyway so it ends up not mattering much.
Tangent: this is the fear that Rust talks about in its value proposition btw. Technically, everything is mutable in Rust. You can flip a bit in an integer that is passed to a function from inside that function if you want. But the difference is that you have control over who gets to do that, and you have visual cues in the code (the `mut` keyword), and you have assurances from the compiler that the rules are followed. One owner at a time. Many can look, but not while a value is modified. Best of both worlds. The end result is that it never bothered me that `Date` is mutable in Rust. Either I'm the owner, or I borrowed the value to look at it, or I borrowed it for mutation and have the guarantees that 1) nobody else is doing the same; and 2) nobody will see in-progress modifications until I'm done.