rewriting x86 binaries - The University of Texas at Dallas

The foundation of the system includes three novel approaches to static x86 disassembly ..... 3.1 Rewriting a register-indirect system call . ..... for the vast majority of COTS binaries and other untrusted mobile code to which reverse ..... the way. Figure 2.4a demonstrates the first series of valid shingles, beginning at the first.
2MB Sizes 0 Downloads 53 Views
REWRITING X86 BINARIES WITHOUT CODE PRODUCER COOPERATION by Richard Wartell

APPROVED BY SUPERVISORY COMMITTEE:

Dr. Kevin Hamlen, Chair

Dr. Gopal Gupta

Dr. Murat Kantarcioglu

Dr. Zhiqiang Lin

c 2012 Copyright Richard Wartell All rights reserved

Dedicated to my parents, for their unending support. And my brother, for his inspiration and honesty.

REWRITING X86 BINARIES WITHOUT CODE PRODUCER COOPERATION

by

RICHARD WARTELL, B.S.

DISSERTATION Presented to the Faculty of The University of Texas at Dallas in Partial Fulfillment of the Requirements for the Degree of

DOCTOR OF PHILOSOPHY IN COMPUTER SCIENCE

THE UNIVERSITY OF TEXAS AT DALLAS December 2012

ACKNOWLEDGMENTS The author is incredibly grateful for the opportunity to work under his advisor, Dr. Kevin Hamlen, who was thoroughly helpful in building the foundation of the author’s knowledge necessary to complete this dissertation and inspiring a thirst for knowledge and creativity. His patient and humble approach to teaching and direction were crucial in the author’s research and the completion of this work. This research would never have been accomplished without the help of the author’s colleague, Vishwath Mohan. His willingness to act as a sounding board for ideas and helpfulness in working on certain pieces of this work were crucial in its completion. Special thanks should also be given to Dr. Yan Zhou whose work in Machine Learning and collaboration on disassembly projects was unendingly helpful to the author. Special mention should also be given to Dr. Zhiqiang Lin for his contributions and ideas that extended the author’s research. Meera Sridhar should also be thanked for her work ethic and contribution to the work on rewriter transparency. Finally, thanks should also be given to the work done by Dr. Micah Jones and Dr. Murat Kantarcioglu whose work is closely tied to this dissertation. This research was supported in part by Air Force Office of Scientific Research awards FA955008-1-0044 and FA9550-10-1-0088, and National Science Foundation award #1054629. All opinions expressed are those of the authors and do not necessarily reflect those of the AFOSR or NSF. The author extends his heartfelt thanks Dr. Mikhail Atallah who was the original inspiration for this author’s interest in security. Additionally, the author would like to thank Matt White who was this author’s original collaborator, inspiration, and good friend. Finally, the author v

vi wishes to thank Nate Gatchell, whose friendship, support, and unending spirit and swagger were a constant inspiration. November 2012

PREFACE This dissertation was produced in accordance with guidelines which permit the inclusion as part of the dissertation the text of an original paper or papers submitted for publication. The dissertation must still conform to all other requirements explained in the “Guide for the Preparation of Master’s Theses and Doctoral Dissertations at The University of Texas at Dallas.” It must include a comprehensive abstract, a full introduction and literature review, and a final overall conclusion. Additional material (procedural and design data as well as descriptions of equipment) must be provided in sufficient detail to allow a clear and precise judgment to be made of the importance and originality of the research reported. It is acceptable for this dissertation to include as chapters authentic copies of papers already published, provided these meet type size, margin, and legibility requirements. In such cases, connecting texts which provide logical bridges between different manuscripts are mandatory. Where the student is not the sole author of a manuscript, the student is required to make an explicit statement in the introductory material to that manuscript describing the student’s contribution to the work and acknowledging the contribution of the other author(s). The signatures of the Supervising