Why Even Read Papers?
While tutorials and docs help you write code right now, the academic papers can help you understand where programming came from and where it’s going. You also understand the paths that foundational academic research did not take. There’s a lot of things that are old that are new again, over and over and over. The idea of Stack Overflow is that someone else has had your problem before; the idea of academic papers is that someone else has thought about this problem before.
How to Read a Paper
The First Pass (5 - 10 min)
Objectives: category, context, validity of assumptions, contributions and quality of writing.
- Carefully read the title, abstract, and introduction.
- Read the section and sub-section headings, but ignore everything else
- Glance at the math to determine the underlying theoretical foundations
- Read the conclusions.
- Mentally tick off references that you’ve already read.
The Second Pass (1 hour for experienced readers)
Objectives: be able to summarize paper’s main idea, with supporting evidence, to someone else.
- Read closely, but ignore details such as proofs.
- Note down unfamiliar terms and questions for the author
- Examine figures, diagrams and illustrations.
- Mark relevant unread references for further reading
The Third Pass (2 hours for an experienced reader)
Objectives: Making the same assumptions as the authors, re-create the work.
- Identify and challenge every assumption in every statement.
- Think how you yourself would present a particular idea.
- Jot down ideas for future work.
On Coming Up With New Theories
A crackpot publicizes a theory that feels right. A theorist goes further to try and disprove a theory that feels right, before publishing it.
There is still value in an earnest well-thought out theory, even if it turns out wrong, e.g. saving others the trouble. Juniors spend far more time making sure they know everything, while seniors release more half-baked ideas - yet the senior’s half-baked ideas will probably be more widely read.
Rules for Reporting Models
Express levels of uncertainty. Report and weigh conflicting models. Describe assumptions and inputs (and sensitivity) of models. Indicate if model is from outside the mainstream.
People often forget models are simplifications. Also, labelling something gives the misconception that we understand it, e.g. democracy, privilege.
Beware of Sensational Reporting
Sometimes, the nuance gets lost when translating research into popular culture. Scientist makes modest discovery; press reports grander claims than those made in the study. Or maybe the scientist uses vague languages that aggrandizes the subject (maybe more press leads to more grant money).
Case Study: Physicists Create a Wormhole Using a Quantum Computer by . criticized the piece citing skepticism from authorities like Scott Aaronson , lack of an arXiv pre-print which would have sparked prior conversation, and past sensationalization from the physicists involved. Cal Tech’s press release came with a disclaimer that no wormhole was created .
Limits to Peer Review
Peer review usually involves vetting by 3 people of similar backgrounds, i.e. peers. Peers are susceptible to groupthink, e.g. papers in the Journal of Marxian Studies will give you a good sense of what the Marxians believe.
Outsiders are more likely to assess the evidence, e.g. are 20-percentage-point vote swings plausible during campaigns? Peers tend to assume that the nitty gritty details are correct.
Hyperlinking to other’s work makes the linkers have skin in the game. The peer reviewers are (sort of) anonymized - if the paper blows up, their reputation is not (publicly) on the line.
Beware The Man Of One Study
Even when there’s what seems like overwhelming in favor of a certain point of view, don’t trust it until you ascertain that the opposite side does not also have overwhelming evidence.
Contradictory evidence may stem from inevitable variation, some studies being higher quality than others, studies about slightly different things being lumped together, etc.
Epistemic Minor Leagues
Minor leagues in sports exist to satisfy everyone’s sports competition drive whether they’re a superstar or not. What is the epistemic minor league equivalent, where we can have important insights in a world full of people much smarter than we are?
Relatable in the case of having a publicly accessible blog-as-notebook, where each page is an approximation of a wikipedia entry. Sometimes, all I feel I’m doing is linking to other people’s ideas with some commentary/links sprinkled in. Having a blog with novel insights is hard.
Maybe the space of knowledge is vast and multi-dimensional that there are enough directions for everyone to push in.
Maybe the heap of already-discovered knowledge is so unwieldy that retrieving an already-discovered knowledge (and/or putting it in easier-to-understand words) is its own form of discovery.
Therein lies the value in classes. For example, most computer science knowledge is somewhere online, but it’s so vast that there is value in an expert (typically an educator) curating some sections for learners. For lifelong learning, the learner needs to establish their position in the knowledge space, and decide which direction to push in. If find this last step hard; I have no conviction on any specific direction.
Maybe there are spheres of knowledge, like politics, where there are no real experts.
References
- How to Read a Paper. Srinivasan Keshav. University of Waterloo. blizzard.cs.uwaterloo.ca . Feb 17, 2016.
- Crackpottery in Theory Formation. Brian Skinner. twitter.com . Sep 2, 2018.
- Separating Theory from Nonsense via Communication Norms, not Truth. Artem Kaznatcheev. egtheory.wordpress.com . Sep 8, 2018.
- All models are wrong, but some are completely wrong. Martin Goodson. rssdss.design.blog . news.ycombinator.com . Mar 31, 2020.
- Guidance Note for Lead Authors of the IPCC Fifth Assessment Report on Consistent Treatment of Uncertainties. Michael D. Mastrandrea; Christopher B. Field; Thomas F. Stocker; Ottmar Edenhofer; Kristie L. Ebi; David J. Frame; Hermann Held; Elmar Kriegler; Katharine J. Mach; Patrick R. Matschoss; Gian-Kasper Plattner; Gary W. Yohe; Francis W. Zwiers. Intergovernmental Panel on Climate Change. www.ipcc.ch . Jul 7, 2010.
- When does peer review make no damn sense? statmodeling.stat.columbia.edu . Feb 1, 2016.
- The unreasonable effectiveness of just showing up everyday. Kishore Nallan. typesense.org . news.ycombinator.com . Jul 14, 2021. Accessed Jul 14, 2021.
- Beware The Man Of One Study. Scott Alexander. slatestarcodex.com . Dec 12, 2014. Accessed Aug 25, 2021.
- Where do your eyes go? www.lesswrong.com . Sep 19, 2021. Accessed Sep 26, 2021.
- Epistemic Minor Leagues. Scott Alexander. astralcodexten.substack.com . Oct 25, 2021. Accessed Oct 26, 2021.
- Template:Academic Publishing. en.wikipedia.org . Accessed Oct 29, 2021.
- You should be reading academic computer science papers. Ryan Donovan. stackoverflow.blog . Apr 7, 2022. Accessed Apr 7, 2022.
- Physicists Create a Wormhole Using a Quantum Computer. Natalie Wolchover. www.quantamagazine.org . web.archive.org . Nov 30, 2022. Accessed Dec 3, 2022.
- The Wormhole Publicity Stunt | Not Even Wrong. Peter Woit. www.math.columbia.edu . www.math.columbia.edu . Dec 3, 2022. Accessed Dec 3, 2022.
- Physicists Create 'the Smallest, Crummiest Wormhole You Can Imagine'. Dennis Overbye. www.nytimes.com . Nov 30, 2022. Accessed Dec 3, 2022.
- Physicists observe wormhole dynamics using a quantum computer. Whitney Clavin. web.archive.org . www.caltech.edu . Accessed Nov 30, 2022.
egtheory.wordpress.com
Going to a talk is difficult for everyone because nobody understands the whole thing, but it’s especially difficult for undergraduates because they still expect to.