Well...
The vacuum of space is a very good insulator. It doesn't lead heat away from any space ship or planet or star. All energy (heat) transfer between objects in space is done with radiation (excluding mass flow from object to object in, say, dual star system with massive neutron star and a giant star).
Any object that has a temperature radiates some amount of heat to space, depending on the temperature of it's outer surface. There are ways to insulate the insides from the outside, so that the thermal energy stored inside a space ship's living quarters doesn't dissipate too quick even if heating fails.
However, to remain at constant temperature, a space ship needs to radiate the same amount of energy as it receives. As far as I know, normally space ships are designed so that when they are in direct sun light, the insides remain at constant temperature, because cooling in space is hella lot more difficult to arrange than heating. This obviously means that when the ship is not heated by sun, it most be heated by the same power to keep contant temperature at living quarters.
As to what the coldest possible temperature is... well, absolute zero temperature can never be achieved, because thermal energy always transfers from hotter to colder. Absolute zero temperature is the zero point of Kelvin scale, and it is 273.15 centigrades below the zero point of Celcius scale. The unit size is the same in both scales.
Okay, that's it for the short introduction in basic thermodynamics, now for the actual answer...
Using a vastly larger hull and inner space surrounding the engine with vent that de-pressurize to open space, can you not use that effect for a certain amount of cooling? (is there a mathematical formula for this?)
The only things that affects the effectiveness of cooling in space are these:
-the size of the radiators (ie. the area which radiates energy to space)
-the temperature of the radiators.
The bigger the area of radiators, the more energy is radiated to space.
The hotter the radiators, the more energy they radiate per unit of surface area.
As to being in light and shadow, the only difference is that the radiators in sun light are absorbing the amount of energy radiated from the sun - about 1.4 kW/m^2 at Earth's distance from the Sun, although it varies a bit based on the elliptic orbit of Earth, but anyway, the radiators in sun light are that much less effective than the radiators in shadow, per square meter.
If the stuff that is circulated from the heat source (ie. engine) to radiators is something with high temperature, like liquid Sodium, the radiators radiate a lot of heat to space. So much that the 1.4 kW/m^2 difference in effectiveness is almost completely irrelevant compared to the total radiation power of the radiators per square meter.
However, having a "cave" of space, or vacuum, inside the ship doesn't work at all for cooling purposes. It doesn't have any ability whatsoever to absorb the heat produced by the engine, reactor or whatever in the ship needs to be cooled. The energy radiated by radiators would only be caught by the radiators on the other side of the "cave", and the temperature of the radiators would eventually rise high enough to cause catastrophic failure. You can actually test this by putting your palms close to each other - pretty soon you should feel some amount of warmth on both of them. It's not just heat transferred from palm to palm by air (be it by conduction or convection, doesn't matter), since air is actually almost as good insulator as vacuum, what with it's lousy density and all; a great portion of that warmth is infra-red (or thermal, if you like) radiation from the other palm that you feel.
Now, the vacuum doesn't stop radiation whatsoever. You would have a situation where radiator on one "palm", or wall of the "vacuum cave" inside the ship would radiate W amount of energy to the vacuum cave, and a similar radiator on the other side. End result would be that both radiators would end up radiating W amount of energy and receiving W amount of energy from other radiators, while the heat source elsewhere in the ship would still be feeding more energy to the radiators.
Not a good equation there.
The thermal excess energy needs to be radiated away from the ship OR contained in a large heat sink during the burn and dissipated at a later time at a speed allowed by the radiators provided.
The third option is to use the fuel as a coolant before using it, which works well with liquid hydrogen chemical rockets. The chamber and the cone of the engine contain coolant tubes through which the cold liquid hydrogen is pumped before injecting it into the combustion chamber. It's a good way to cool the chemical rocket engine, because the hydrogen takes the gained thermal energy with it away from the engine and you don't need heavy and big radiators to deal with that energy.
Using liquid oxygen as a coolant is *not* a good idea, though... for reasons that should be obvious.
This option would work to some extent with NERVA-based nuclear engine as well, at least partially, since the basic idea is to pump liquid hydrogen through hot reactor core, where it would rapidly expand and eject away from the ship, resulting in opposite momentum for the ship. But it would only be usable during the burns, and for any long-term missions you'd kinda want an energy source for other reasons than propulsion as well...
So other methods of cooling are necessary.
