High‐density polyethylene has good spectral transmission characteristics in the range from 20–500 cm−1. This, coupled with its ease of fabrication and low cost, make it an excellent choice for far‐infrared absorption cell windows which operate at room temperature. However, because of its relatively large thermal coefficient of expansion and its brittleness (at low temperatures), the usual type of polyethylene absorption cell windows are difficult to make leaktight at low temperatures, especially if there is a moderately high pressure inside the absorption cell. In this article the problems associated with the use of such windows at low temperatures and moderately high pressures are discussed and a successful window design which overcomes these problems is described. These windows have been incorporated in a cell which has been used to measure the collision‐induced absorption of various nonpolar gases at low temperatures. In these experiments the cell was cycled hundreds of times at pressures up to 96.5 bar and at temperatures down to 126 K.