The parsimony principle is frequently used in system identification on more or less heuristical grounds. Recently, a formal proof of this principle has been given. Here the assumptions used in that proof are examined to some extent. It will be shown by means of some counterexamples that these assumptions cannot be relaxed unless additional restrictions are introduced. A result showing how the parsimony principle should be used when estimating the parameters of regression models by the least squares method is also presented.