The forces acting on a body, and its position of equilibrium, in a uniform magnetic or electric field are determined by the free energy and not by the field energy. Incorrect results are obtained if the field energy is used as a potential function since the work done in a virtual displacement on the distant sources of the field is not taken into account. The net-energy flux that results from a virtual displacement of a body is calculated here and the free energy is determined. It is shown that the flux of the Poynting vector cannot be neglected in this case even though quasistatic processes are involved. The results are illustrated by the example of a paramagnetic or a diamagnetic ellipsoid in a uniform magnetic field.