Please do not cite or reproduce without permission
Turning Numbers Into Knowledge: Mastering The Art Of Analysis
Jonathan G. Koomey, Ph.D.
6429 Harwood Ave, Oakland, CA 94618
Get a PO Box
"...I'd always believed that a life of quality, enjoyment, and wisdom were my human birthright and would be automatically bestowed upon me as time passed. I never suspected that I would have to learn how to live – that there were specific disciplines and ways of seeing the world I had to master before I could awaken to a simple, happy, uncomplicated life."
Jonathan G. Koomey is currently a staff scientist at Lawrence Berkeley National Laboratory, one of the nation's foremost research laboratories. He holds M.S. and Ph.D. degrees from the Energy and Resources Group at the University of California at Berkeley, and an A.B. (Cum Laude) in History of Science from Harvard University. He is the author or co-author of seven books and more than one hundred articles and reports on energy efficiency and environmental policy. In 1993, his report, titled Cost-Effectiveness of Fuel Economy Improvements in 1992 Honda Civic Hatchbacks, won the Fred Burgraff Award for Excellence in Transportation Research from the National Research Council's Transportation Research Board.
Dr. Koomey has been fascinated by numbers since he was a young boy. In graduate school, he amazed and annoyed his friends with his love of back-of-the-envelope calculations. In his spare time, he enjoys Aikido, hiking, cycling, running, and playing classical contrabass.
Holdren may write this. Check with him again Mid January 2000
Blurb for cover
This is a book about numbers, but not about math. An award-winning young scientist teaches you the art of analysis, revealing tools, tricks, and heretofore unwritten rules that the best real-world problem solvers know by heart.
"Mr. Casals, you are 95 and the greatest cellist that ever lived," a young reporter commented. "Why do you still practice six hours a day?" Casals replied, "Because I think I'm making progress."
“Whatever failures I have known, whatever errors I have committed, whatever follies I have witnessed in private and public life have been the consequence of action without thought.”
Quantitative analysis is the process by which we take numbers and transform them into knowledge, using our instincts and experience to fill in when we don't have all the answers. While the technical aspects of analysis are taught at many universities, the art of analysis is rarely discussed and even more rarely written down. This book teaches that art, and will help you become a first-rate analyst in your chosen field.
After reading this book, you will be far better equipped to make independent judgments about analysis used by others. You will know which key questions to ask, so you need never again be at the mercy of those who traffic in "proof by vigorous assertion".2 You will also be more effective at conducting and presenting your own analyses, no matter what the topic.
Mastering the art of analysis takes more than proficiency with basic calculations: it requires familiarity with how people use information, and learning about things as diverse as ideology, telling good stories, and distinguishing facts from values. To give you a feeling for what to expect, I present an annotated chapter list below.
Annotated chapter list
This book contains seven major sections, separated into almost forty short chapters. Each chapter is compact and self-contained, and each summarizes key lessons I've learned over the years.
I. The information explosion: This section briefly describes how analysis can help reduce the information overload that affects us all.
II. Things to know: These chapters summarize ideas to keep in mind as you read the rest of the book. More experienced analysts should skim them and delve into the ones they find most intriguing.
1. Beginner's mind: Start fresh and approach any problem like a beginner would, and you'll surely see things that others will miss.
2. Don't be intimidated: The difference between success and failure often depends on whether you are intimidated. By consciously refusing to be intimidated, you can stack the odds in your favor.
3. Information, intention, and action: This chapter describes how humans respond to events, exploring the connections between what we measure, what we assume, and what we choose to do.
4. Peer review and scientific discovery: Progress in science can be subject to human frailty, just like any other human endeavor. The end-result, however, is something you can count on.
III. Be prepared: A key determinant of your effectiveness is the quality of your preparation. Whether you're building a house or chairing a meeting, preparation for the analysis tasks at hand can turn a potential disaster into a triumph.
5. Explore your ideology: Ideology provides a simplified model of the world that reflects our values and experiences, and prevents paralysis in the face of the many choices we make every day. Make sure you know your own and that of others.
6. Get organized: Working in a chaotic office is like running a marathon with your feet tied together. Get your workspace in shape and keep it that way.
7. Establish a filing system: Few mistakes are more maddening than knowing you have seen a relevant article, but not being able to find it. By creating a good filing system, you can prevent this annoyance from ever happening again.
8. Build a toolbox: My analytical toolbox is the set of tricks and techniques that I can pull out to solve a particular problem. This chapter describes some key tools to consider for your own.
9. Put facts at your fingertips: Every analysis requires data. Unless you've memorized the encyclopedia, you'll still want to keep some key reference sources within easy reach. This chapter describes the ones I find most useful.
10. Value your time: Remember that your time is your life, and that if someone is wasting your time, they are stealing your life. Identify your most productive times of day and protect yourself from interruptions during those periods. Unplug the phone. Go to the library. Take control of those times!
IV. Assess their analysis: When faced with the assertions of others, it's good to know the right questions to ask. These chapters summarize hard-won knowledge about deciphering other people's analyses.
11. The power of critical thinking: Careful critical thinking is at the root of all good analysis. When the steps described in this chapter become second nature to you, you will have mastered its essence.
12. Numbers aren't everything: Not everything that matters can be quantified, so make sure the unmeasurable doesn't fall through the cracks.
13. All numbers are not created equal: Numbers and calculations characterizing the physical world are almost always more certain than those describing human behavior. Many analysts wrongly imply that modeling results based on economic data are just as solid as science. They aren't, so be forewarned.
14. Question authority: This catch-phrase of the 1960s is still applicable today. Authority figures can be wrong or biased, so investigate their assertions in the same way as you'd examine those of someone with whom you're not familiar.
15. How guesses become facts: Always remember that "official" statistics are based on calculations that are often poorly documented, incorrectly cited, or otherwise hazardous to your intellectual health.
16. Don't believe everything you read: Maintain a healthy skepticism, even of well established sources. In this age of instant information transmittal, rumor and error seem to propagate even more quickly than truth.
17. Go back to the questions: Any time you rely on survey data to make an important decision, refer back to the actual questionnaire upon which the survey data are based or risk misinterpreting the data.
18. Reading tables and graphs: First check for internal consistency, then see if the results contradict other facts you know to be true. Search for cognitive dissonance: any discrepancy between the author's results and what you already know will help you investigate further.
19. Distinguish facts from values: Don't be fooled by technical people who portray their advice as totally rational, completely objective, and value-free. If they have made a choice, they have also made a value judgment.
20. The uncertainty principle and the mass media: Just as the observer of a subatomic particle can disturb that particle by the act of observation, the observer of an institution can disturb that institution by observing and reporting on it. These observers should take responsibility for the power they wield.
V. Create your analysis: Each person develops their own techniques for creating cogent analyses, and I summarize those I've learned here. The importance of organization, clear thinking, careful definitions, systematic exposition, scrupulous documentation, and consistent comparisons cannot be overestimated, and you'll learn about each of these here.
21. Reflect: Free yourself from interruptions and give yourself time to reflect. Without such time, you'll never achieve your full problem-solving potential.
22. Getting unstuck: Everyone gets stuck sometimes, but it need not hobble your efforts if you use the tricks in this chapter.
23. Inquire: When faced with a problem outside your expertise, don't surrender! It's an advantage to be unconstrained by the mental shackles most disciplines place on their practitioners. Some of the most important insights in modern thought come from people who could think "outside the box".
24. Be a detective: Detectives are real-world practitioners of the scientific method. The time-honored techniques of these seasoned problem solvers can help you in your efforts.
25. Create consistent comparisons: People often relate best to anecdotes. A consistent comparison is a well chosen set of two anecdotes that illustrates your point in a compelling way. It is a powerful technique, and one well worth learning.
26. Tell a good story: Scenario analysis is the art of structured storytelling, and it's an essential tool for any good analyst. Most people don't realize that this art is both well developed and pertinent to many everyday situations.
27. Dig into the numbers: Don't be shy about delving into the actual numbers, even if you're a highly paid executive. You'll learn things you would never see if someone else crunches the numbers.
28. Make a model: Models are "laboratories for the imagination", and in this chapter we explore the subtleties of using them to explain the world around you.
29. Reuse old envelopes: You can calculate almost anything using only common knowledge--you just need to learn how to put this knowledge to use, and this chapter (which focuses on back-of-the-envelope calculations) is just the thing to help you do it.
30. Use forecasts with care: The future is uncertain, but people keep trying to forecast it anyway. Numerous pitfalls await, and without a keen eye for the tricks of this trade, you'll be hard pressed to avoid them.
31. Hear all sides: In any intellectual dispute, it pays to hear two well-prepared debaters argue their points before drawing any conclusions. Always make such debates fodder for your deliberations and your decisions will benefit.
VI. Show your stuff: Once you've done good work, you'll want to present it effectively to readers or listeners. The chapters in this section give insights into making your results "grab" your audience, designing good tables and figures, and using those tables and figures to convey your key points. The section concludes by exploring effective use of the internet in publishing your analysis.
32. Familiarity breeds interest: The first rule of data presentation is to know your audience and present information they care about in a form they can easily grasp. Most analysts forget that other people don't care nearly as much about their results as they do.
33. Document, document, document: An astounding number of analysts routinely omit vital data and assumptions from their reports, but this pernicious practice is one to avoid. The best analysts document everything, giving credit where credit is due, leaving a trail for them to remember, and leaving a trail for others to follow. Documentation is also a key step in checking your work, because it forces you to think clearly about your analysis.
34. Let the tables and graphs do the work: When writing technical reports, create the analysis, tables, and graphs first, then write around them. If the analysis is well thought out, the tables and graphs well designed, and the audience clearly defined, the report should practically write itself.
35. Create compelling graphs and figures: Follow Edward Tufte's rules for graphical excellence and avoid the all too common pitfalls of creating charts and graphs. Your goal should be to give to the reader "the greatest number of ideas in the shortest time with the least ink in the smallest space".3
36. Create good tables: A good table is a work of art, while a bad one is worse than useless. Make your tables a resource that your readers keep as a reference for many years to come.
37. Use numbers effectively in oral presentations: Even veteran presenters show too many of the wrong numbers. Only present numbers that support a key conclusion, and focus on the conclusion, NOT the numbers themselves.
38. Use the Internet: The old ways of publishing are fast being supplanted by web-based approaches. Learn about these new tools and put them to work for you.
VII. Some parting advice-Creating the future: This chapter gives perspective on why we use analysis in the first place. Understanding the world is a prerequisite for making it better!
Who should read this book
This book grew out of my experience in training analysts I've hired in the past decade. It is written for beginning analysts in business, government, consulting, and research professions, and for students of business and public policy. It is also intended for supervisors of such analysts, as well as entrepreneurs (who may not consider themselves analysts, but who need to create analyses to justify their business plans to potential investors). Finally, it covers many topics that journalists who focus on scientific or business topics will find useful.
How to use this book
There is no need to read these short chapters in order. Go straight to those that interest you most, but skim the chapters you skip. You just might see something useful there that you did not expect.
Most chapters have "links" to other chapters, using a graphical signpost telling the reader which chapter or major section to investigate for that link (the relevant chapter number appears inside). These signposts look like the one that appears in the right margin opposite this line. In the electronic version of this document these links will be "live", so that the book is truly a hypertext document and readers can jump to related sections with ease.
All URLs discussed in the book, as well as many key data files, will be available at http://www.numbersintoknowledge.com at publication time (the web site is not yet operational, as of November 1999, but the domain is being reserved by my web hosting company). If you have questions, comments, or suggestions, you can post them at this site. I'll evaluate them for inclusion in the next edition. I'm particularly interested in examples of large and public analytical blunders by people who should know better, examples of bad or good tables and graphs, and suggestions for how the book can be improved or expanded.
Footnotes on a particular page contain information essential for understanding the material in the chapter, while Endnotes contain references, attributions, and further information for the interested reader.
The Further Reading at the end of the book does not attempt to be comprehensive. Rather, it contains selected sources for each chapter that I regard as most crucial for mastering the material. If I mention a book, I include it in the further reading section for the chapter in which I refer to it. At the beginning of the further reading section, I also compile the “Top Ten” list of my very favorite sources on this topic, which are the “must read” items that all serious analysts should have on their shelf.
"Like all other arts, the science of deduction and analysis is one which can only be acquired by long and patient study."