It is shown theoretically and experimentally that the presence of thermal conductance between a pyroelectric material and its environment affects significantly its measured dielectric loss. Depending on the details of the heat exchange with the environment, this thermal contribution to the measured dielectric loss tangent can be larger than the intrinsic loss tangent of the material, i.e., loss due to domain wall motion, defects, and impurities. In detectors fabricated from LiTaO3, temperature fluctuations due to this thermal conductance, known as temperature noise, are observed to limit the normalized detectivity (D*).