There’s also the fact that you lose heat pretty slowly in space, and lose heat slower the colder you are. Even in deep space away from a star, spaceship design has to take into account the fact that very hot objects (like a reactor, or extensive electronics) will heat a spaceship up beyond human tolerances unless some kind of active cooling system is put in place.
In comparison, you lose heat very quickly in a cold environment within an atmosphere, because you’re actively being cooled by the airflow and the surfaces you’re touching.
Yeah, that is a pretty interesting aspect of space. It may not be intrinsically warm, if you’re say, in the Earth’s shadow… but there’s not much stuff to take your heat away in space, because there’s not many particles in a vacuum.
(the heat concerns do hold true, as you said)