Excerpts from a conversation in 1996.
From a student:
Doc Offutt,
Lately, I've been reading a bunch of varying stuff lately. In doing so, I have become a bit frustrated with myself. In particular, I find that while I' m usually reading for a specific purpose, I often am doing so in an inconsistent fashion.
My question is this: How does one go about reading a technical paper? What should I be looking for? What should I be ignoring? What should concern/alarm me? How do I evaluate what I've read? How do I summarize what I've read so that it's useful to me as a research device? I think you probably get the idea of what I'm after.
My wordy answer:
As you become more in tune with research and writing, and your own research goals, your reading will naturally change.
One goal when reading a paper is to understand every detail. This goal is appropriate when reading for a class, learning a new area, or early in a graduate career (note that the first two are typical of the 3rd). At this point in my career, I seldom do that. I read to understand every detail when I review a paper and when I prep for a class.
Another goal is to try to keep up with the latest results. I look at every paper I see on testing like this. I read the abstract to get an idea of what it is about and how much I know. I look through the references to see if I "know where the paper is coming from". I scan the intro and conclusions (more or less, depending on how much the abstract told me ... I prefer abstracts that summarize the conclusions). Whether I look more depends on a number of things. I may want to know whether to believe their results. Or I may want to borrow ideas from how they present background work. Or I might want to look at what they are thinking of in the future. A lot of this is "goal directed" ... I know where my research is going, and I look for pieces of information that can help me.
Another reading goal is to mine an area looking for research ideas. This is appropriate for phd students working on the dissertation, or looking for a topic, and researchers looking for new topics (especially early in their careers). The most important thing here, of course, is the future work section. Lots of papers toss out wonderful problems! Also, intros and backgrounds give ideas. Some do it well. Some papers state these clearly (usually called "limitations"), and others don't. Sometimes I read a paper and then just close my eyes and let my brain drift. Most of the time I fall asleep :), but on a good day I have some ideas. On a real good day, I have a good idea!
By the way ... I get a lot of ideas at conferences. I listen to about half the talk, scan the paper, and then my brain drifts off onto what they should have done but did not.
Still another is to browse other research areas to keep cognizant. I will scan 3 or 4 papers month in TSE, TOSEM, Software, etc to get an idea of what's going on elsewhere. Sometimes I just read the abstract, sometimes the conclusions, and occasionally the intro. When something really catches my interest, I will read some of the internal details.
The ONLY time I read a paper from start to finish is when I'm editing it, either as a reviewer or as a prof. I usually scan the paper quickly, getting the main points, and the outline of the paper. I look at the references -- if I know most of them, I feel happy because the paper will be easy to read. If I don't, then I'm unhappy. Then I go back and read the paper again. If I'm conscientious, I take two passes -- one to spot presentation problems, writing problems, and overall concepts, and another to check the technical details carefully.
> What should I be ignoring?
Nothing now.
Laters, most of the details.
If you can, ignore on the presentation and focus on the contents
(I still cannot do that :-().
> How do I evaluate what I've read?
Does it give you anything useful?
This is from my research advisor:
>How do I summarize what I've read so that it's useful to me as a research device?
Here's a scheme that Roland Untch and Gregg Rothermel developed
For every paper that is relevant to their research area, they entered the bib entry into a bib database (latex is well set up for this kind of thing, with bibtex). In addition to the bib information, they entered in abstracts and comments. This gives a database of papers to reference and provides them with a concise reminder of the paper. If done well, they should never need to look at the paper again!
Here is an abbreviated example. The abstract is directly from the paper. The annotate is a summary of what you got from the paper. The comment is your assessment of what is good or bad about the paper. Note: I created an artificial example to protect authors from any negative opinions or misunderstandings.
@INPROCEEDINGS{SmithJones01,
author = "J. Smith and S. Jones",
title = "Using Specifications to Test Software",
booktitle = "Proceedings of the International Conference on Engineering of Complex Computing Software (ICECCS '01)",
month = "June",
year = "2001",
address = "Skovde, Sweden",
publisher = "IEEE",
pages = "1-15",
abstract = {
We take formal specifications in the language Q,
and derive finite state automata.
Then we invent a new test criterion
(that coincidentally looks just like edge coverage on a graph)
called the "automaton-shift criterion".},
annote = {Partition analysis of state-based specs (Q).
Q --> DNF test specs --> FSA
* Test individual operations
* Test system state
* Use FSA to sequence tests of operations
* Generate values to satisfy DNFs
* Verify results
Straightforwardly translates the specs (pre- and
post-conditions) into DNF predicates. The paper does not
clearly explain how the predicates are solved. The test
sequencing is quite clever, but very complicated. They have
implemented a tool for the system (in perl).
},
comment = {
The paper
seems to read more like a theorem proving paper than a
testing paper, but they are doing testing.
I suspect they
don't really understand testing.
I like the idea!
They used TRITYP as an example.
Idea ... take their tests and run them on trityp under
mutation. Then take mut-adequate test sets
and compare with
their technique. Which does more testing?"
}
Here is a comment from another paper that may be more typical:
comment = {What junk. How did they get this in TSE?},
I think this is a terrific system!!!