The cognitive basis for academic workflows

The importance of cognitive workflow in long works.


Last Fall, late into comprehensive exams, I paused to consider the effort it took to research and produce answers to questions that would only be used in a 2-hour comprehensive exam. Of course, if I was lucky, some of the writing might make it into my dissertation. And the time spent reading, thinking, and learning was potentially invaluable. But the process was painful. For each of four papers, I considered my question, amassed sources, crammed notes into Evernote, outlined, wrote, re-wrote, re-thought, re-searched, re-collected, re-assembled, re-drafted and finally spend a day inserting and formatting sources properly. That was just for a first draft.

In the end, all of my work was in disconnected Evernote notes and Word documents.

Like many other doctoral students, I came to the realization that my method was incredibly inefficient and could not scale. In fact, I realized that my largest problem was actually the ability to re-use my own thinking, knowledge, and experience.

So I read everything I could find on academic workflows.

What I found was huge discussion on the relative merits and trade-offs on organization (knowledge management) tools, brain-storming tools, note-taking, bibliography, and writing tools. Naturally, I downloaded a variety and began to explore. At first, I couldn’t see why there were so many over-lapping functionalities. For instance, various students and researchers would admit to using multiple brain-storming tools simultaneously. What the heck?! (I’ll get back to this later.)

Fairly quickly, I came to the conclusion that my best chance at settling on a workflow was to focus on studying the workflows of researchers who were very skilled. I reasoned that someone who publishes a lot probably was good at re-use of previous thinking and work. This post by Steven Berlin Johnson describing his use of DevonThink is representative.

While developing my own workflow, I ran across an article from Pirolli and Card (2005) that reminded me intensely of academic workflows! In fact, they had done a cognitive task analysis (CTA) of intelligence analysts. What they described was sense-making. In essence, sense-making is a set of processes for framing data to a model and also fitting data to a mental model (Klein, Moon, & Hoffman 2006). The first part (framing data) is posed as an “information foraging loop” while the second (fitting data to a model) as a “sense-making loop”. Another way to look at this is as an interleaving of bottom-up processes with top-down processes: search, filter, and understand versus find evidence, examine counter arguments, and re-evaluate. The diagram below is extracted from Pirolli and Card (2005). cognitive task analysis

My current workflow is represented below. In effect, it represents a simple CTA that generalizes the academic workflow that I now use. sense-making

Though I drew in arrows that indicate a sort of continuous back-and-forth process at a micro-level, there do seem to be larger foraging and sense-making loops. It seem so since I spend a day or days at a time doing focused work within specific tools. In fact, I spend a lot of time in Scrivener. Writing seems to require much more effort than note-taking in other parts of my workflow. Probably, before adopting this workflow, I spent most of my time in the foraging loop - and then losing or forgetting much of the learning I had attained. (Let’s face it, hunting is more fun.)

The major tools I use appear under the loops in the diagram above:

Not shown is a production tool. I typically export from Scrivener to LaTeX (the subject of a future post).

Another way to look at this process is in terms of note-taking. At each stage in the process, my notes become more compact and inter-connected. When I read an article, I may outline summaries and extract concepts and quotes. But when I’m sense-making, I’m primarily working in Tinderbox or OneNote and what’s important is thinking and bridging – the relationship between notes. They are typically short, but need to link back to summaries, extracts, and sources. When I write prose, I work in Scrivener. The link back to intermediate sources (Tinderbox & OneNote) and original sources (Pages & DT) is important to maintain. Sometimes there is the need to go back and work on material in more depth. But, in the end, all of these intermediary products are re-usable.

Now, back to the question of overlapping functionality. Why is it that, at least on the Mac, there is so much overlap in functionality between “academic workflow” tools? Why would I want to take notes in multiple tools? It turns out that note-taking is a multi-faceted beast. Depending where I am in the sense-making process, I’m taking different sorts of notes. For example, below are different sorts of notes serving different sorts of purposes - which may be used in different parts of the sense-making loop. I adapted the scheme below based on an analysis of historians using DT.

tabular system of note-taking

Labels, tags, notes (summaries, maps, highlights, etc.) are all really just notes. We just use them in different ways. And some are obviously more compact than others. As Mark Bernstein of eastgate.com remarks:

(These remarks can be found in Bernstein’s talk on the youtube video link at the bottom of this post.)

There are also other small workflows which impact productivity in a cumulative sense. One of the most valuable has been to create a small workflow which is triggered whenever I save a PDF to the desktop. When this happens, a hazel script immediately imports the document into Papers2. This causes me to source the document right away so that when I need to reference a source downstream, its already entered into Papers2.

In the end, I hope to capture my own knowledge and experience over time so that I can build on previous work without having to re-think and re-organize every time I start a new paper. It’s an experiment in progress. FYI - I still use Evernote. But more for random odds-and-ends such as notes on programming and how to keep my blog running smoothly.

Here are some of the references online that I found particularly inspiring for my own workflow. Interestingly, I found lots of useful information about DT from historians, while historical fiction writers seem more excited by tools like Tinderbox. Scrivener, however, definitely crosses boundaries. Apparently, all of us are deeply concerned with writing and production. Enjoy!

Klein, G., Moon, B., & Hoffman, R. (2006). Making Sense of Sensemaking 2: A Macrocognitive Model. Intelligent Systems, IEEE, 21(5).

Pirolli, P., & Card, S. (2005). The sensemaking process and leverage points for analyst technology as identified through cognitive task analysis. Presented at the Proceedings of International Conference on Intelligence Analysis.