▲ 147 points
back
50 comments
Gwern wrote a good overview of spaced repetition and how to apply it [1]. If you're interested in going deeper, it serves as a good starting point as he provides all kinds of references.
Spaced repetition(with mnemosyne) worked very very well for me when I was preparing for GRE's verbal section. However, I have no idea how I could use it for technical knowledge.
I use spaced repetition every day for technical knowledge. It's very easy:
Every small piece of information that you want to memorize, create a new card.
Let me give some examples from my current technical deck:
Q: This is an open source data collection system for monitoring large distributed systems. It's a distributed data collection and analysis system in Hadoop. It runs collectors that store data in HDFS and it uses MapReduce to produce reports.
A: Chukwa
Q: This is an Apache templating language and processing engine that can be used to generate Java code, HTML, JSP's, Hibernate modules and most anything else that can be generalized.
A: Velocity
Q: In Ruby, what command lets you determine whether you're running as a main program?
A: if __FILE__ == $0
Q: This C++ function rearranges the elements in the range [first, last) into the lexicographically next smaller permutation of elements.
A: prev_permutation
My biggest roadblock with using Anki is figuring out what exactly is worth typing up into a card, when the material isn't just unrelated discrete items (i.e. vocabulary) that don't stick easily.
For example, in lectures I'm usually able to understand most of what is taught on that first exposure, but of course I don't have everything committed to long term memory right away. So, sometimes I'm tempted to write a lot of things down or type them up in Anki (which can get quite tedious).
However, in math for instance, you pretty much have to know the exact definitions of theorems to apply them in proofs. Thus you refer back to the textbook or lecture notes/video (if available) when applying a theorem in a problem, and soon you have that definition down cold in memory. Similarly, in courses or subjects where you need to do a lot of problem solving or application, you can look back at source material for concepts or definitions as needed (this itself is probably a form of spaced repetition). In cases where I wrote down detailed notes or entered them into Anki, I find that I probably recorded a lot more than I should have.
So how do you figure out what is worth (most efficient) entering into cards in that case?
Virtually everything is worth entering into cards. Anki's interface is very tedious, but it's nearly always worth it.
Looking back at source material is not a form of spaced repetition. By learning that you don't need to remember it, it's actually harmful for memorization. Trying to remember things without looking them up (really making an effort) is better and can be almost as good if you fit between the spaced repetition intervals, but you might as well systemitize it.
This is a very nice idea. Thank you! I am currently using spaced repetition to improve my German vocabulary. Using this to remember APIs and technical documentation is a great idea!
While there are a number of comments questioning the utility of doing so and comparing it to having notes in a file, the fact that this is stored in my memory reduces latency when thinking or writing code. I no longer have to look things up which really improves my flow. I see the difference whenever I work with an API I already know and one I'm relatively new to.
I find that when I have more information in my local storage (memory), I can make more creative associations among the data and come up with better solutions to problems.
I do something similar to this as I'm reading a tech book, especially on something new, I build a Q&A of 1) high-level vocabulary 2) some of the finer points that are worth remembering, like maybe language syntax. I ignore the stuff that I'd probably just look up in the future anyway.
Why would you do that instead of just keeping everything in a notes.txt?
Because there's still no good exabrain. Notes.txt quickly becomes an unsearchable, undiscoverable mess of crap; wikis are hard to maintain and too heavyweight; desktop wikis never really work right and usually have shit mobile experiences.
In the meantime, to remember a fact for the rest of your life, you need to add it to the SR deck and spend 30 seconds in the rest of your life reviewing it.
Spaced repetition is a clear win; any attempt to resist it is madness. Imagine being able to just keep everything in RAM and never, ever touch disk.
Human competitive advantage (creative latency at human timescale) vs. computer/robot algorithm operating at HFT timescale.
As for the benefits of human-robot brain-algorithm cooperation (e.g. human+Google), that depends on the level of coherence between the goals of the human and the goals of the creators/financiers of the algorithm.
Agree with parent. chipuni's strategy seems like trying to pigeonhole a reference when statistically, you're most likely always going to be on a computer and have access to a reference that's searchable.
The advantage for many is the actual physical association.
I liken it to learning how to get around a building by actually walking around the building, as opposed to just seeing a map. Ultimately, when there are no other distractions going on, one is probably as good as the other.
Not that I don't think you can't get this in a notes.txt. Just, I understand why some would want a physical card deck where they flip a question into an answer. Involving your body seems logical as a benefit for some.
Mostly, I put into Anki exact words that I need (or needed) to have at the tip of my tongue.
So I've been using Mnemosyne for a few months for physics. (I started it as a quals study-aid, after reading gwern's article.) I've found that two types of card work well:
1. Frequently (in physics) you can find a core skeleton of fundamental definitions & results. If you know these, you can solve a whole bunch of problems very easily; also, in reviewing those results, you inevitably remind yourself of why they're true (I'm starting to make a re-derivation a criterion for listing a "fact" as memorized), and of the broader logical structure of the discipline.
2. Certain problems exercise a wide variety of important techniques very quickly; I use Mnemosyne to prompt myself to re-work certain problems every so often. I try to be aggressive enough about rating "how well I know this" that I'm actually re-working the problems, not just memorizing. This is sort of like the "code kata" I've seen people suggest every now and then.
Now, this is for physics, which is probably not exactly what you mean by "technical knowledge" :-). I expect this approach would generalize very nicely to mathematics and theoretical CS (and in fact I'm using it for bits and pieces in those fields); code & software engineering would be harder, but probably doable.
Edit: removed pointer to (link to) Gwern's article: it's in the gp. Also assorted stylistic matters.
Another advantage of full stack development. Code a little, chase a bug in the webserver, google site:serverfault.com in frustration, knock out a couple of tickets from users, finally drive to the dc to change a failing drive. Related, yet different, and get you to change the scenery.
Specialization is for insects.
Here's a paper with an evaluation of those techniques[1] and here's a pretty, high-level summary[2].
[1] http://psi.sagepub.com/content/14/1/4.full
[2] http://bigthink.com/neurobonkers/assessing-the-evidence-for-...
Amazing stuff, it confirms what I've recently been able to discover for myself. Narrow focus on one specific problem is a fools game I am learning, rather keep switching dishes, take a little bite each time. This incremental practice got rid of burn out, improved quality and was able to get more work done for less time and stress.
A more up-to-date and complete treatment of these exact topics can be found in the recently-published book, _Make it Stick: The Science of Successful Learning_ (http://www.amazon.com/dp/0674729013/). For those that have already immersed themselves in spaced repetition techniques via Anki, SuperMemo, Mnemosyne, etc., some of the material in the book will be review. But there is a wealth of useful information for both students and lifelong learners. In addition to that, the intertwined, real-life case studies and anecdotes drive home the points. And lest my comments come off like a back-cover endorsement, I will say that I was a bit disappointed that the authors seemed to have little awareness of SRS' momentum on the Internet with the aforementioned programs. For example, even though the book spends quite a bit of time discussing spaced repetition and flashcards, it never once mentions Anki, SuperMemo, or other popular SRS software. Aside from that, it's still an excellent book.
This is exactly how the curriculum is designed in a Waldorf school; each main subject is taught for a few weeks intensively, then left aside for a month or so before next re-visit.
Only things you can forget is truly learned!
There was a famous writer once who said that one of his greatest assets was his extreme forgetfulness. Appropriately, I am blanking on his name at the moment, but I'll pop back in here if I recall it. It was a long time ago.
However, his point was something like focusing on rote memorization of detail makes for trite, superficial, and boring fiction-writing, and that you're better off experiencing something as intensely as possible and then forgetting it almost immediately. The idea being that your brain will recall the crucial bits more relevantly and more vividly once they have been yanked back from the fog of utter forgetfulness. I'm not sure how closely related this is to the ideas presented in the article, but for me personally, it became significantly easier to write fiction with a real sense of "immediacy" once embracing this principle.
However, fiction can require a strange sort of learning: it can involve sensory knowledge of scattered quotidian detail. Back when I wrote a lot, things like the discrete texture and smell of different wooden tables were very interesting to me. Or exactly how it feels to be in a room lit by a single large fluorescent light vs other kinds of light. I'm not sure that kind of learning transfers to any other domain, really. But forgetfulness was very useful for it, at any rate. I'm done now.
> you’d want to interleave serves, backhands, volleys, smashes, and footwork — not serves, synchronized swimming, European capitals, and programming in Java.
So if I want to learn European capitals I interleave it with what? The tennis/serve example is obvious, the rest is not.
>So if I want to learn European capitals I interleave it with what? The tennis/serve example is obvious, the rest is not.
In principle it should be another piece of knowledge which makes "European capitals" useful, in the same way that having a good backhand gives you more opportunities to serve. Examples would be "European systems of government", "European political movements", "European international politics and diplomacy", "European countries", etc. It's a little harder to think of, perhaps, because "European capitals" per se seems to many people to be an almost useless piece of knowledge.
I agree. A simple strategy for the "European capitals" would be to learn the size of them in conjunction. I guess the key is to search for something useful that you can relate to easier? For Example: London, 8.3 Mil; Berlin 4.3 Mil and so on...
European capitals are the centers of political decision for a country, but one could find the capitals for cheese, mathematics, water colors, etc.
You can't think of anything?
* European history
* European nation-states and borders
* European language
* European physical geography (major rivers, mountain ranges)
* Famous Europeans and their hometowns
* European architecture
I can think of a million things. The problem is where to draw the line. For tennis it's obvious since you need both the serve and the backhand. Compare that to European capitals and European architecture. When is it related in a sense that it's effective to learn interleaved?
Respectfully, the problem (I believe) is seeing it as a line.
Interleaving, isn't a "right" or "wrong", "black" or "white"... Think of it as, this topic is "less" or "more" effective.
What do I mean? Well, as someone else mentioned, studying "European Nation States" might be a good option for you. Certainly a better option than, say, Swan Diving. You should accept that "European Nation States" is the best you can do right now.
One of the hallmark traits of intelligence is the ability to "relate" unrelated things. In this way, you could - theoretically - interleave ANYTHING, you just need to think abstractly enough about it.
Good luck
There isn't a line. We're talking about the human brain, these things are fuzzy. Some things will work better than others, and it'll surely vary between people, and you'll have no idea which are which until you try it. But it seems like the theme is association: Find the common information between the things you're studying.
Context here is important. Your hypothetical leaves out why. If you just need to learn a skill for no reason other than to catalogue it as a line-item in your accomplishment list it will be hard. If what you are learning is relevant there will be a bunch of similar data streams or skills that you can practice in parrallel.
Interleaving is building a context relevant association graph. It can be anything you want. It can be as crazy as James Burke[0]
> When is it related in a sense that it's effective to learn interleaved?
A concept, person or place that is emotionally important to you.
It probably depends on why you're learning this. If I understand the concept correctly, the idea is to interleaving complementary skills. i.e. you stay within the same broad skill/activity domain.
So if learning this to prepare for a quiz show, I'd probably interleave it with specific trivia-esque facts about each capital city's culture, history, etc. That way you both learn the names of the capitals, and you attach varied bits of context to each name.
If I needed to know them because I'm moving to somewhere in Europe and I want to have a "lay of the land," I might interleave other useful tips and tricks I would need to know. Any quirks in how they drive over there, how to order food in whatever languages I need to know, the different new social customs I need to be aware of, how to translate in my head between English and metric measurements perhaps, etc ...
For me the problem isn't packing things into my brain in a way that keeps them from leaking out ... the problem for me is figuring out what are the right things I want packed into my brain
Crude, but what I use: text file with each note a paragraph.
Each paragraph get's an integer weight. A filter repeats
a given note (paragraph) as the integer indicates. Every
twenty minutes 3 random notes are echoed to one of
my GNU screen terminals. I go to that 'screen' window when
I'm bored. This is somewhat like UNIX 'fortune'. The weights
are arbitary, quite a few '1's, and some urgent ones as high
as 1500 - some for learning, some for nagging.
Very neat! This is actually an implementation of "spaced repetition", a learning technique that's been employed for many decades. The Pimsleur foreign language courses, famous for their efficacy, use it.
For folks less interested in injecting this into their terminal, check out Anki. It implements spaced repetition as flash cards, and automatically computes how often to show a card based on how often you say you already know the material.
for spaced repetition learning, I can recommend ankisrs.net
Anki is great, but I'm currently having a bit of trouble in that after reaching a few thousand cards that are all fairly mature, I can't seem to get below 20 cards a day review. When it's that many, I'm experiencing a bit of disincentive in creating new decks to drill myself on new knowledge. I thought they'd continually get slower but the rate at which card appearances slow down seems to slow down over time as well.
I had a similar issue using Mnemosyne. Then I read "Moonwalking with Einstein" and I'm pretty convinced that spaced repetition is a solution to an adjacent problem. More important is the vividness of the memory, and the use of spatial memory (memory palace technique), along with some other techniques, for engaging many parts of the brain. These make the memory stick much more clearly and for a longer time.
It's a great book but I understand there are others that are more focused specifically on techniques, with much less story.
I really like the broad coverage that articles like this give to the psychology of learning, but I wonder how important some of these principles are in a practical sense. Sure, the effect of studying and retrieving in the same context is a classic effect, but how strong is it?
It's like reading an article on how to make money that doesn't even give you a ballpark estimate of how much money each strategy could make you.
> doesn't even give you a ballpark estimate of how much money each strategy could make you
Public strategies have short lives in efficient zero-sum markets. Here is a site with resources and active discussion on mnemonics, memory palaces and other practical memory improvement techniques: http://artofmemory.com/
The article makes interesting claims. Some of them ring true. But there is very little backing for the claims in the article. Therefore I'm surprised to see it still on page 1.
With the title being “… and keep them from leaking out”, I thought this would be about mental compartmentalization and doublethink.
Think of it as knowledge infiltration instead.
> "Because humans have unlimited storage capacity..."
this is patently untrue, unless human brains defy the known laws of physics. this makes it hard to believe everything that he mentions.
You don't have to take it so literally. Remember we're humans, he probably meant unlimited in the sense of human experience.
If a human's brain never gets full in a lifetime, the it's seemingly "unlimited" for our purposes. Just like how having 500 PB on a camera SD card would be unlimited for any photographer, yes, it has a limit, but you would never reach it without doing some crazy shit.
It's functionally infinite, because it's bounded by encoding time, not storage capacity. If you spent your entire life learning you would not learn enough to exhaust your long-term memory.
This is a very basic fact about human memory that you can find in any psych 101 textbook. You should consider picking one up and reading it.
The underlying assumption being that you won't live much longer than anyone else. Hopefully this will change, and soon. Do those psych textbooks make any guesses to how long an average human would have to live before they exhausted their brain's storage capacity?
I am reminded of the immortal character in Douglas Adam's Dirk Gently's, who has been alive too long and forgotten all of his early life (including who he is). I believe he was supposed to be a Time Lord.
This is close to the truth – that character was not supposed to be a Time Lord in the book, but the storyline of the book is loosely based on two separate scripts written by the same author for two separate Doctor Who serials. (For unrelated reasons, the one script containing that character was only partially filmed, and the serial was therefore not broadcast at the time.)
Similar to the method in this article, I had already applied it to development.
I used to work on each tickets one by one, move on when it's fully done but this got very tiring, sometimes I would have to come back to a bug because it was related to another one.
I started to work on each ticket for 15~30min max, moving on regardless of whether I had got anything done. When I started to feel the burn from a ticket I would skip it and move on to the next ticket.
What I found was that I ended up getting far more done in less time. What happened was kind of like skipping a question on a test and coming back to it later. Your brain keeps working on it, or maybe you had to ask stackoverflow and it took time for someone to answer it. Anyways, time automatically began working for me, instead of me trying to bang my head against the keyboard until one thing was completely finished.
Once you shake off the feeling of unproductivity and begin seeing just how easy things are when you work on multiple tickets in short time intervals, its far worth the effort.
I've yet to name this technique, I shall call it 'notastartups maximum ticket resolve method'
We changed the title to a sentence from the article.
Edit: changed it again, because this piece is more interesting than "Interview with Robert Bjork, director of the UCLA Learning and Forgetting Lab" made it sound.