The Mathematics Of Generalization PDF

Preface

This book grew out of a workshop held under the auspices of the Center for Nonlinear Studies at Los Alamos and the Santa Fe Institute in the summer of 1992. The idea for the workshop arose from a perception that there were many different fields that address supervised learning, but by and large these fields were not communicating with one another. (Examples of such fields are neural nets, conventional Bayesian statistics, conventional sampling theory statistics, computational learning theory, Al, and machine learning.) In particular, there were many different mathematical frameworks for addressing supervised learning. All had their own jargon, their own concerns, and their own results. And for the most part they were not interacting.

This was clearly a less than optimal state of affairs; we all have much to learn from one another, not only in terms of raw mathematical results, but also (perhaps more importantly) in perceptions of what the crucial issues are and how they should be addressed. Unfortunately, although it seems that this problem is abating, the rate of improvement is quite small. It seems possible that a general lack of communication amongst its practitioners will characterize supervised learning theory for some time to come.

The purpose of the workshop was try to (begin to) rectify this situation. A small group of researchers from several of the different supervised learning fields was brought together and, in effect, forced to mingle. The format of the workshop was an intensive two-day session of talks and discussion. The participants were encouraged to have their talks be as informative as possible to practitioners of the other fields. Obviously such a workshop could not make all the participants instant experts in each other’s fields. Rather the hope was that the participants would go home with a heightened sensitivity to the other fields. They could then follow up and explore those other fields “on their own time,” as it were. In this the workshop appears to have been successful, to judge by the comments made by the participants subsequent to the workshop.

This volume is an attempt to try to replicate the success of the workshop in a broader context. Its purpose is to do for the reader what the workshop did for its participants: help a practitioner in one of the fields that make up supervised learning become acquainted with the relevant work by his or her colleagues in other fields.

Obviously (and unfortunately) it is not possible to duplicate in a reader of a book the experience of “an intensive two-day session.. .(of being) forced to mingle... (with) researchers from different fields.” Given the different format, slightly different means are needed to achieve the same ends. Accordingly, it was decided that the chapters in this volume should not so much be a formal compendium of the talks presented at the workshop as an overview of the work being performed by the researchers who attended the workshop. Some of the work represented in these chapters had not even been completed at the time of the workshop. Some of the other chapters are reprints of work published shortly before or soon after the workshop. However, all of the chapters were chosen by their authors with the same goal in mind: to help those from other supervised learning fields get acquainted with the lay of those authors’ lands. Moreover, the instructions to the authors were that they should not try to provide tutorials on their individual fields. (There are many other sources for such tutorials.) Rather they should present current cutting-edge perspectives and work that provide an intuitive understanding of what their field “is all about.”

Of course, thanks are also due to the agencies that support the CNLS and the SFI, and thereby supported the workshop. In particular, I would like to thank the DOE, the NSF, Citibank, and the Mac Arthur Foundation. I would also like to thank the NIH and TXN Inc., who provided additional support during the compilation of this volume.

Authors: David Wolpert, Santa Fe