The so‐called “first generation” expert systems were rule‐based and offered a successful framework for building applications systems for certain kinds of tasks. Spatial, temporal, and causal reasoning, knowledge abstractions, and structuring are among topics of research for “second generation” expert systems. It is proposed that one of the keys for such research isknowledge organization.Knowledge organization determines control structure design, explanation and evaluation capabilities for the resultant knowledge base, and has strong influence on system performance. We are exploring a framework for expert system design that focuses on knowledge organization, for a specific class of input data, namely, continuous, time‐varying data (image sequences or other signal forms). Such data are rich in temporal relationships as well as temporal changes of spatial relations, and are thus a very appropriate testbed for studies involving spatio‐temporal reasoning. In particular, the representation formalism specifies the semantics of the organization of knowledge classes along the relationships of generalization/specialization, decomposition/aggregation, temporal precedence, instantiation, and expectation‐activated similarity. Á hypothesize‐and‐test control structure is driven by the class organizational principles, and includes several interacting dimensions of search (data‐driven, model‐driven, goal‐driven temporal, and failure‐driven search). The hypothesis ranking scheme is based on temporal cooperative computation, with hypothesis “fields of influence” being defined by the hypothesis’ organizational relationships. This control structure has proven to be robust enough to handle a variety of interpretation tasks for continuous temporal data. A particular incarnation, the ALVEN system, for left ventricular performance assessment from X‐ray image seque