An implementation of non-regular symbol manipulation with neural networks is presented. In particular, it is shown how a context-free language can be produced with neural networks. The rules of the language are stored as patterns in an attractor neural network. Another such network is used as a working memory, which can be enlarged without changing the production system itself. As a result, the competence of symbol manipulation with neural networks equals that of classical non-regular production systems. In actual behaviour (performance), however, there are differences between the systems, which shows the importance of implementation in the generation of rule-like behaviour.