In a typical inductive learning scenario, instances in a data set are simply represented as ordered tuples of attribute values. In my research, I explore three methodologies to improve the accuracy and compactness of the classifiers: abstraction, aggregation, and recursion.Firstly, abstraction is aimed at the design and analysis of algorithms that generate and deal with taxonomies for the construction of compact and robust classifiers. Secondly, I apply aggregation method to constructively invent features in a multiset representation for classification tasks. Finally, I construct a set of classifiers by recursive application of weak learning algorithms. Experimental results on various benchmark data sets indicate that the proposed methodologies are useful in constructing simpler and more accurate classifiers.