A theoretical investigation is made of the modulational and filamentation instabilities of laser radiation at the upper‐hybrid frequency in the presence of the self‐generated magnetic field of the order of a few megagauss in a laser‐produced plasma. The nonlinear Vlasov equation is solved to obtain the low‐frequency response of magnetized electrons in the plasma. The present theory is valid for short‐wavelength perturbations and arbitrary values of the wavenumber. It is observed that the growth rates of the instabilities are very sensitive to the plasma parameters near the critical density layer. The growth rate of the modulational instability is always larger than that of the filamentation instability for the same power density of the incident laser radiation. It is further noticed that the modulational instability decreases slowly with the magnetic field, while the filamentation instability remains almost insensitive to the magnetic field.