▲ 250 points
back
82 comments
My dad (an engineer not a mathematician) would use Newton-Raphson[1] to solve basically any problem that wasn’t very obviously linear. When I was a kid, some of my first programming memories were my dad getting me and my brother to implement Newton-Raphson in basic on an HP85a, getting me to implement Newton-Raphson in rpn on an HP calculator, debugging my dad’s (genuinely revolting) basic program[2] which wouldn’t run but (who would have guessed?) used Newton-Raphson to compute something or other.
He basically learned the one numerical root-finder and how to evaluate basic second derivatives and he was set for life on all the problems a career in chemical and process engineering could throw at someone.
[1] https://sheffield.ac.uk/media/31988/download?attachment
[2] He learned to program in FORTRAN and lived by the maxim that a determined FORTRAN programmer can write FORTRAN in any language.
The most brilliant software developer (an EE PhD) that I have ever worked with has been using the singular value decomposition (SVD) to solve an enormous number of linear algebra and numerical computing problems in engineering software. The SVD seems to be useful for many engineering computations if you know how to apply it.
It’s the optimal low-rank approximation of a matrix in some norm-2 sense, it has those nice beautiful orthogonal factors on both, and it is often quite easy to compute. Why wouldn’t you use it, right? If you are going to have one trick, truncated SVD is a good pick.
Does he like Halko, Martinsson, and Tropp’s randomized SVD? It is pretty slick.
What are some interesting things he has used the SVD to solve?
My dad was also an Engineer. He was also a Fortran fan.
Once he asked me to explain OOP. After I explained the basics he said it was useless and never looked back.
Where OOP shines is implementing user interfaces. Most engineers doing math can ignore objects. But when a programmer has to implement a partitionable window with scroll bars and menus, constructing complicated objects built of simpler objects is a mess without OOP.
Get your dad a copy of the classic Scientific and Engineering C++: An Introduction With Advanced Techniques and Examples by John Barton and Lee Nackman. It was written to introduce C++ to Fortran programmers using examples from scientific/mathematical domains. The fact that it is old (from 1994) makes it better suited for folks from Fortran (or other languages) since there is none of the complexity of "Modern C++" to confuse them. Check reviews on Amazon etc.
I guess as a typical engineer (not CS person type of engineer or software engineer) it is easy to think that. One might be working with machines or buildings and so on, all which require _calculation_ of processes. Those are typical cases for "just write a correct function", possibly one that takes many things into account. For such scenario OOP is truly useless and only over-complicates the matter. However, when we get to simulations, where maybe there is no known formula, or the precise calculation would be too expensive, then OOP can make sense. Doesn't have to, but could.
Seems like a common theme of every veteran to dress the way used to in their prime for the rest of their lives, listen to the same music, watch the same movies, etc. and to use the same belief systems as well. On the one hand, if it worked for them, why not? There's no incentive to change. Heck, it is very much the definition of conservatism. Old men who don't change is so common that it borders on proverbial.
Very rarely, however, do you see a brilliant mind like Richard Feynman, a man who was so open to new ideas and out of the box thinking. Even in old age. Seeing someone, in good faith, question what they believe in light of new knowledge is very rare. Now that is a special thing.
Chad Dad
Fortran has OO features these days. It is nice.
I am kind of dumb and old-school so I wrote a bunch of code using macros to handle multiple precisions. If I could go back in time, I’d definitely just use object oriented code. In either case, though, “we can try a mixed precision implementation, I automatically generated single precision” is an incredibly liberating thing to be able to say as a computational scientist!
Relatedly, I've found Newton-Raphson is a great example of an algorithm where Knuth's "I have only proven it correct, not tried it" rears its head very prominently. The obvious implementations can work flawlessly on toy examples and then fail miserably on real-world examples.
There's a reason why numeric analysis is still actively studied by research mathematicians. If we could just throw something as simple as newton's method at any nonlinear problem, we'd only need people to learn this once in school and everyone could solve everything.
I fear not the man who practiced 1000 kicks but I fear the man who practiced one kick 1000 times
This applies to musicians. E.g. early blues guitarists only had a few licks but they played them so many times they honed them into perfection.
I would also toss differential evolution in the ring as a widely applicable and easy to implement technique.
Nobody has enough memory or patience for third order derivatives so Newton's method (aka Newton Raphson) it is.
The modern world of autodiff actually makes real second and third order derivatives fairly cheap to compute.
The stuff that is actually used most commonly, only uses first order derivatives though (gradient descent, Levenberg-Marquardt, Kalman filters...)
I think this is true for engineers as well! I enjoy getting to know the "theme" of my favorite coworkers over the years. There was:
* The fellow who always looked for the simplest hack possible. Give him the most annoying problem, he'd pause, go Wait a minute! and redefine it to have a very easy solution. He typed very slowly, but it didn't really matter.
* The one who truly loved code itself. He would climb mountains to find the most elegant, idiomatic way to express any program. Used the very best practices for testing, libraries, all that. He typed very fast.
* The former physicist who spent all his time reading obscure mailing lists on his favorite topics. His level of understanding of problems in his domains of interest was incredible.
I could go on and on! It's such a fun taxonomy to collect. All of these friends were marvelous at solving their particular flavor of problem.
As for myself, I like to think that my "trick" is to spend a long time poking at the structure of a problem. Eventually the solution I was looking for doesn't matter anymore, but the tools I developed along the way are pretty useful to everyone!
Here are a few more.
* The (brilliant) infrastructure engineer who described his modus operandi as 'I read stuff on Reddit and then try it out.' This engineer is now worth, as a conservative estimate, in the neighborhood of $50 million. So maybe more of us should be doing that.
* Another infrastructure engineer, also very effective, who made a habit of booking an external training session (sometimes a series, weekly) for how to set up and integrate every piece of technology we used.
* An engineer (this one is quite famous, and we have only interacted professionally a few times) who simply wrote the best comments in the world. Every method was adorned with a miniature essay exploring the problem to be solved, the tradeoffs, the performance, the bits left undone. I think about those comments quite often.
As an addendum, though, I will say that the best engineers overall all shared a trait - they kept trying things until they got something working. That alone will take you pretty far.
For me, it's tracing code/pipelines to figure out how a result was produced, typically in the context of that result being wrong somehow. Go To Definition is the most useful function in any editor.
I'm always surprised by how frequently colleagues don't think to do this and are left helpless.
My impression from math lectures as a CS student was, that you have to know the tricks, otherwise you just sit there dumbfounded, not knowing how you can make progress. Obviously, there will also be some kind of mathematical understanding of things at play, but a lot of the math stuff is pattern recognition and applying tricks. If you don't know them, it becomes very frustrating. I've not ever seen a way of teaching math though, that focuses on teaching learners these tricks or techniques. They always fell out of things on the side or the teaching person assumed you know it, or was surprised, that the learning people were not able to see it, or able to apply some trick. Or they just didn't care, as long as a small portion of the learners somehow were able to solve it, because then it obviously cannot possibly have been their way of teaching, but other students must have been lazy...
Feynman wrote in his autobiography that much of his success came from having different mathematical tricks than most of his peers. So when they were stuck, he could sometimes make progress.
Interestingly, one of his famous tricks of computing complex integrals by parameterizing[1] and then differentiating under the integral is known as “Feynman’s trick” in his honour spite of it having been invented by Euler over 250 years before.
A popular naming convention in Mathematics is to name things after the second person to discover them. The first person is Leonhard Euler.
Feynman used to read his own books. When asked he said, "it's all in here". He used to revise and refresh his own understanding.
I get that. Sometimes it feels like we are like Dynamic RAM.
He had many tricks in limited area. All his his tricks were limited to classical calculus and nineteenth-century mathematics. He didn't do anything fancy.
His style was always the same, he just mastered it really well.
That is the actual trick. To master one trick so well that you can spot anywhere its hiding. Unfortunately, this takes a long time, which is why most mathematicians can only do it for a couple of tricks in their lifetime
I remember annoying one of our professors at the university. Whenever he was discussing a problem and found me falling asleep, he called my name. I woke up, said Chinese remainder theorem and with like 90% success rate it solved the problem handily. Yes, it was an Algebra class. We were still surprised how well it worked.
One of the answers links https://www.tricki.org/, which describes itself as a 'Wiki-style site with a large store of useful mathematical problem-solving techniques.' no longer maintained, but looks neat.
Thank You for pointing that out; that is indeed nice!
My lecturer was once unable to solve a problem at the blackboard.
After trying for some time, he said he needed a short break to go back to his office to look at his notes. He brought the notes, and there he found a hint he had written for himself: "Use a trick"
There is a joke in applied mathematics that we’re like Taco Bell. We all use the same six ingredients, mixing them in different ways.
For myself, I’ve found several techniques I use over and over again. Some of this is a “when you’re a hammer, everything looks like a nail.” But fundamentally there are only a handful of ideas. One professor of mine once said the only groundbreaking result in the past few decades was compressive sensing.
For programmers I would say graphs or at least thinking in graphs is a common one.
Some would say SAT is also a handy trick but I’ve personally never used it.
SAT, SMT, ILP, MILP.
- The hard part of a compiler is the parser. Find an existing parser, then emit using that language's web templates.
- Your DB query is better in an inverted index.
- Consider your data locality carefully.
From a comment:
> These methods can be combined. First generalize the problem, making it more complicated. Then simplify along a different axis. – Stig Hemmer
Very relevant to software design too.
I feel like our minds work a bit like this.
For most of my computer science PhD the “trick” was just to get the inductive definition to work, and then how to tweak it for the next paper. Or, get enough structuret we can do an “abstract nonsense” proof[0].
[0]:https://ncatlab.org/nlab/show/category+theory#AbstractNonsen...
I do various things where for example I need to convert a linear movement of a control knob to a non-linear value.
Quite often I'll look at it and go "aha, it's exponential subtracted from a constant", and then go "aha, that doesn't quite work, it's a polynomial", and then fiddle with that for twice as long.
Eventually, "meh, sod it, I'll just sketch out a graph of what I want on paper, turn it into a lookup table, and LERP for the correct value".
It's amazing how quickly you get a sense of "I just need to bend that in a little so that if these are both up full it doesn't go absolutely mental", and plan your tables accordingly. Also ROM is cheap these days.
One surprise of reading this thread was learning that Terry Tao is an active user of Math Overflow!
A mathematician tricks.always.
There is no point to untrick the trickster. Misuse of your time
In my world every triangle is a right triangle and I my hammer is the Pythagorean Theorem.
I like the trick of adding zero to an equation.