We review indications of persistent deviations from the geocentric axial dipole model of the time‐averaged geomagnetic field and present a zonal harmonic model derived from 185 deep‐sea sediment piston cores taken from low to middle latitudes (to approximately ±45°). Analysis of the paleomagnetic inclination recorded in these cores for the Brunhes (normal polarity; 0–73 Ma) and Matuyama (reverse polarity; 0.73–2.47 Ma) chrons, after plate motion correction, gives well‐constrained estimates of the dominant long‐term nondipole contributions (the axial quadrupole and axial octupole) and shows no significant deviation from axial symmetry. The amplitude of the axial quadrupole is found to vary with polarity (2.6% of the geocentric axial dipole for normal; 4.6% for reverse), while the axial octupole does not show appreciable change (−2.9% for normal; −2.1% for reverse). These estimates of the quadrupole contribution agree well with prior determinations for the Plio‐Pleistocene (0–5 Ma); however, the octupole contribution we find is opposite in sign to previous estimates. We suggest that a negative octupole is representative of the actual time‐averaged paleomagnetic field, while prior positive octupole estimates probably reflect spurious inclination shallowing. The lack of polarity asymmetry in the octupole suggests that this nondipole component may be more closely linked to the main dipole field than is the quadrupole and so supports models of the geodynamo in which dipole and quadrupole f