Recent research in knowledge‐based expert systems of VLSI design tools has concentrated on placement, routing, and cell generation. This paper presents an alternative application for artificial intelligence (AI) techniques on compaction design for a VLSI mask layout‐expert compactor. In order to overcome the shortcomings of iterative search through a large problem space within a working memory, and therefore, to speed‐up the runtime of compaction, a set of rule‐based region query operations and knowledge‐based techniques for the plane sweep method are proposed in this system. Experimental results have explored the possibility of using expert system technology (EST) to automate the compaction process by “reasoning” out the layout design and applying sophisticated expert rules to its knowledge base.