Orbital Energy
The energy of an object moving in a gravitational field such as the Sun's can be divided into two parts  kinetic energy or energy of motion, and potential energy or energy of position.
The kinetic energy depends upon the mass and velocity of the object according to the formula
E_{kinetic} = m v^{2} / 2
where m is the mass of the object and v is its velocity, or speed of motion (velocity also includes the direction of motion, but that is not important in this situation). This is always a positive value, and since planets move faster in smaller orbits, the smaller the orbit the greater the positive kinetic energy per unit of mass.
The gravitational potential energy is defined as a negative value equal to the kinetic energy that the object would gain by falling from an infinite distance to its current position. At very large distances from the Sun the object would have zero potential energy (since it wouldn't have picked up any speed by falling). Objects close to the Sun have large (though negative) potential energies, corresponding to the speed they would gain by falling a long way. Because the gravitational force of the Sun changes with distance, deriving the formula for potential energy requires calculus; but the result is a simple alteration of the Law of Gravity:
E_{potential} =  G m M / r
where m and M are the masses of the planet and the Sun, r is the distance from the planet to the Sun, and G is the constant of gravitation. Remember that unlike kinetic energy, which has a larger positive value in smaller orbits, potential energy has a larger negative value in smaller orbits.
The total energy of an object is the sum of its kinetic and potential energies, and since for a falling object the kinetic energy increases at exactly the same rate that the potential energy decreases, the total energy of a planetary orbit is a constant:
E_{total} = E_{kinetic} + E_{potential} = m v^{2} / 2  G m M / r = (a constant)
and the total energy per unit of planetary mass is as is shown below:
E_{total} (per unit of mass) = E_{total} / m = v^{2} / 2  G M / r
Application to Circular Orbits
Kepler's Third Law of Planetary Motion, which relates the size of an orbit to its orbital period, can be stated as (after Newton's modification of the Law)
P^{2} = 4 π^{2} a^{3} / G M
where P is the orbital period of a planet, a is the semimajor axis of the planet's orbit, or its "average" distance from the Sun, and M is the combined mass of the planet and Sun (which is treated as just the mass of the Sun in this discussion because (1) the planets have negligible masses compared to the Sun and (2) taking planetary masses into account here would require taking them into account in the calculation of the potential energy, and the extra masses cancel out, which yields the same result as simply ignoring them).
For a circular orbit Kepler's Second Law of Planetary Motion, which relates the speed of the planet to its distance from the Sun, requires the speed to be as constant as the distance from the Sun. That constant speed is simply the circumference of the orbit divided by its orbital period:
v = 2 π a / P
Squaring this yields v^{2} = 4 π^{2} a^{2} / P^{2}Squaring this yields
and using Kepler's Third Law to replace the square of the period,
v^{2} = (4 π^{2} a^{2}) / (4 π^{2} a^{3} / G M)
Division cancels most of the terms, leaving (for circular orbits)
v^{2} = G M / a or
E_{kinetic} / m = v^{2} / 2 = G M / 2 a
Since r and a are the same for a circular orbit, this is exactly half the (negative) value of the potential energy per unit of mass. As a result, adding the kinetic and potential energy gives a total energy which is half the potential energy and equal to the kinetic energy, but negative. In other words, for a circular orbit,
E_{total} = E_{potential} / 2 =  E_{kinetic} = G M / 2 a = (a constant, for a given orbital size)
If the orbit is very large, so that a is very big, the energy is a very small negative number. For smaller orbits, the total energy is a larger negative number. In smaller orbits the velocity is larger, and the kinetic energy is larger, but the potential energy is also larger (in fact twice as large), so the energy of smaller orbits is a bigger negative number than the energy of larger orbits. (This happens to have important consequences in electromagnetic and nuclear interactions, but that is beyond the scope of this discussion.)
Application to Elliptical Orbits
In elliptical orbits the velocity is not constant, so the simplifications of the preceding section do not apply and as a planet moves away from or toward the Sun the kinetic energy and potential energy continually rise and fall; but since one rises exactly as much as the other falls, their sum (the total energy of the orbit) is always the same, and in fact all orbits of a given size (or semimajor axis) have the same total energy per unit of mass, regardless of the eccentricity of the orbit. In other words, a circular orbit and an elliptical orbit, as long as they have the same semimajor axis, have exactly the same total energy. In the circular orbit the kinetic and potential energies are constant, whereas in an elliptical orbit one rises while the other falls, but the total energy remains the same as if the orbit were a circle of the same average size.
Deriving the Effect of the Sun's Loss of Mass
With the above as background we are now ready to see how the loss of mass the Sun is bound to suffer when it gets very old affects the size of planetary orbits. In doing so we will assume that the orbits are sufficiently circular that we can use the simplifications applicable to that case, and that any friction due to gases lost by the Sun passing the various planetary orbits produces negligible effects on their motion. In that case, the only thing we need worry about is that as the Sun loses mass it exerts less gravitational force, and cannot hold the planets in the orbits that their current velocities require. As a result the planets would gradually move away from the Sun, attaining higher orbits and in the process, reducing their speeds to values which the Sun's reduced gravity could maintain in such orbits.
To see how this works, let us start with the formulae for the Sun's current ("original") mass:
E_{total} = E_{potential} / 2 =  E_{kinetic} =  G M_{original} / 2 a_{original}
As the Sun loses mass a planet's orbital potential energy will become smaller (although since negative, it will actually represent a higher energy), but the kinetic energy will (at first) remain as it was, so the total energy will increase by the same amount as the potential energy. This causes the orbit to become larger, reducing the (negative) value of the potential energy still further, and at the same time (because objects move more slowly in larger orbits) reducing the kinetic energy. This continues until a stable state is reached, in which the (new) kinetic energy is numerically equal to the (new) total energy, and half the size of the (new) potential energy.
In numerical terms, the original kinetic energy
E_{kinetic (original)} = G M_{original} / 2 a_{original} plus the new potential energy
E_{potential (new)} =  G M_{new} / a_{original} yield a new total energy
E_{total (new)} = E_{kinetic (original)} + E_{potential (new)} = G M_{original} / 2 a_{original}  G M_{new} / a_{original} and in a new nearly circular orbit, this total energy equals half the new potential energy
E_{total (new)} = E_{potential (new)} / 2 =  G M_{new} / 2 a_{new} or, combining the terms,
G M_{original} / 2 a_{original}  G M_{new} / a_{original} =  G M_{new} / 2 a_{new} Multiplying each term by 2 a_{new} / G M_{original} we obtain
a_{new} / a_{original}  2 M_{new} a_{new} / M_{original} a_{original} =  M_{new} / M_{original} Rearranging terms, we obtain
(a_{new} / a_{original}) (1  2 M_{new} / M_{original}) =  M_{new} / M_{original}, or
(a_{new} / a_{original}) = ( M_{new} / M_{original}) / (1  2 M_{new} / M_{original})
or (reversing the signs on the right),
a_{new} / a_{original} = (M_{new} / M_{original}) / (2 M_{new} / M_{original}  1)
