▲ 1667 points
back
506 comments
We had a similar problem at work in the late 90s. A member of staff reported that their mouse would stop working between certain hours of the day. It had apparently been okay in the morning, stopped working over lunchtime then started again later.
On some days it would work perfectly all day long, but on others it would stop working between those hours.
The biggest clue was it would always work perfectly on overcast days, but on sunny days this strange behaviour would manifest again.
Turns out the problem was related to the mouse being a cheap mouse. The case had very thin plastic.
The mouse was a ball mouse, and it worked by shining an LED into a sensor on each of the X and Y axes. On sunny days the sun would completely overpower the sensor due to the plastic case being very thin and on overcast days it would not. On sunny days the mouse would only work when the sun had moved around the sky to cast a shadow over where the mouse was being used.
Perfectly logical but baffling at first.
Reminds me of a problem that I had (many years ago) with my iPhone 4 - if I tried to boot it in a dark place, it would get stuck on the Apple logo in an infinite boot loop.
Turns out some versions of the Pangu jailbreak for iOS 7.1.x would crash during boot if the reading from the ambient light sensor was below some threshold. To this day I don't know the exact explanation of this bug, but it seems that Pangu included some unnecessary code that messed with the light sensor [1].
If you don't believe me, there is a huge reddit thread[2] with a lot of people confirming this.
[1] https://www.reddit.com/r/jailbreak/comments/294wob/jailbreak... [2] https://www.reddit.com/294wob/
I had a similar problem, but in the opposite direction. My cable internet speeds at home were fairly good (for the US, anyway), but sometimes would absolutely bottom out. Not dead, just glacially slow. After troubleshooting everything under the sun, I came to realize that the problems would happen not when it was raining per se, but when it was heavily foggy or misting. Normal to heavy rain was fine.
Called the cable company, tech came out. Everything inside was fine, but the cable from the main line to the house had a tiny cut in one spot, not enough to really affect the connection, but enough for ambient moisture to work its way in and foul the connection.
We have a countertop ice maker that gets jammed up and overloaded with ice on sunny days for a similar reason.
There's an infrared beam and sensor. When the ice tray is full, it is supposed to block the beam, and then the machine stops making ice.
On a sunny day, there's enough bright light in our kitchen to fool the sensor so it keeps making ice.
We have a random magazine that we put on top of it to make it work correctly.
Your mouse story makes me think of the day the CI system at work turned out not to be robust to vibrations.
One day we started having flaky tests, seemingly out of nowhere. We quickly identified that the issue affected tests involving graphical X client applications, but then we struggled to make further progress. The issue was just impossible to reproduce in other conditions... Well, as it happens, the CI jobs were running on some desktop machines we had installed somewhere within our premises. It turned out that some gentleman had plugged a mouse into one of the machines, and left it lying around on the shelf. Since then, when one of the machines was under a heavy load, the fans would spin faster, causing more vibrations, in turn causing the mouse to move, ever so slightly. And for ungodly reasons, this had side effects on tests.
Fun fact: the machines were not on my site, I managed to diagnose this over SSH. I was quite proud :-)
When it’s sunny, my wife’s car can’t open the garage door, and my car requires getting extremely close. Once the sun goes down, we can both open the door from the street.
It turns out our solar panels (or the optimizers, or the inverter) emit radio frequencies that interfere with our garage door opener. When the sun is out and they are producing energy, the interference is stronger than the homelink garage door opener.
A few years ago the garage door openers started working fine. It took a few days to realize it was because the inverter had failed.
I’m fairly certain there are some FCC regulations that would require our installer to fix it, but that relationship soured during installation and I’d rather deal with an unusable garage remote than dealing with them for warranty work.
Argh. My own mouse not working one:
- Use to fix PCs professionally in the early 90s.
- Guy comes in with PC. Right-mouse button stopped working.
- Replace mouse. Still not working.
- Play with Windows 3.1 drivers. Nothing helps.
- Pull HDD from another PC, install, boot. Mouse button still broken. WTF.
- Pull whole mobo, put another spare mobo in, with replacement HDD and replacement mouse. Still don't work.
- Replace PSU. Right-button works.
- Give up on computers, live in wilderness, eat squirrels.
I have a similar stories.
The first was a VDSL connection I had at home. It worked great (fast, for the time) except when it didn't. It always failed in the evening. Techs would come out, bless it as being good, and leave -- because of course it worked while they were there. Unless they showed up and it was broken and then they'd declare that it was an outside problem, and that they'd have to get someone else to fix it (because the residential techs can't do overhead work).
I made lots of (very polite) phone calls, which results in more refunds and more service calls. More than once, my driveway and the street in front of my house looked like an AT&T convention.
This went on for months.
I had direct numbers and emails for tier 3 support and the local manager who oversaw this plant. We were all getting to know eachother too well, and there were boots on the ground addressing this problem as many as three times in week.
I eventually noticed that as the days got shorter so did the evening outages...and that if it was a cloudy day, then that day was often outage-free.
I had an epiphany: The problem might correlate with the angle of the sun, and the duration of exposure!
I checked my logs and the past weather, and sure enough: It lined up.
So I reported my findings, even though they seemed like nonsense as the words came out of my mouth, and they sent out some crazy-haired guy with bluejeans and an untucked shirt who was clearly not used to wearing a uniform, and who was also obviously not normally customer-facing.
"I know exactly why they can't find the problem," he said after I reiterated what I'd learned. "Your neighborhood still has old lead-sheathed overhead lines, and nobody knows how to work on that anymore."
"But I'm certified on that. I'm going to go back to the shop, pick up a bucket truck and get your line fixed. It will take me most of a day to do this, but I will be back when I'm done."
And it was getting pretty late, but he did come back to let me know that he found some things and fixed them. And I don't know what those things were, but it was fine after that -- and it stayed fine.
Thermal expansion letting cosmic rays leak into copper pairs wrapped in paper, tar, and lead? Who knows. I certainly don't know.
I've never encountered that stuff professionally (and it isn't your grandfather's 25-pair cable) and as this dude said, "nobody knows how to work on that anymore."
We had a similiar problem with a label printer.
On some days, exclusively in the morning hours, the printer would fail to detect the start of a new label, printing over several labels.
After connecting remotely and checking the usual (queue, network connection, drivers etc), I asked my colleague to call me, as soon as it happened again.
When I went there, I saw that a ray of sunlight hit the printer. The windows had shutters, but there was a gap.
Label printers detect the gap between labels using a laser. And for some reason, the printer's case had a clear window at the top.
I printed an empty label and stuck it on the little window.
I'm amazed an IT department would troubleshoot deeply enough to figure out it was the thin plastic letting in interfering light on sunny days.
I would have guessed they'd shrug at the first sign of trouble, swap it out with a known-working mouse and mark the ticket resolved... unless all the replacement mice were thin plastic too, I suppose.
This is a secondhand anecdote, but it’s pretty funny. Back in the days of server rooms, a friend’s server for his company would reboot every day around 5pm. They checked everything they possibly could with the OS, they would be logged in and running checks on it and it would spontaneously go offline for about 5 minutes and reboot every day. Finally they decided to go stand in the presence of the server around the time it goes down every day. They watched a cleaner come into the room, unplug the server rack, plug in their vacuum and vacuum around the servers, and then plug the server rack back in.
This reminds me of a GM minivan that my youngest brother-in-law drove, back in the '80's. He'd gotten it from his father, who was a career GM automotive engineer - and complained that, occasionally & randomly, it would not start. It seemed like the minivan's whole electrical system was dead...
Brother-in-law was known to be "not so good" with cars - so his automotive engineer father didn't take the complaints seriously.
Complaints and emotions escalated, until brother-in-law convinced his dad to swap vehicles for a month, so (he hoped) his dad could experience the problem for himself.
After the problem manifested in the parking lot of the GM Technical Center, and the whole crowd of GM engineers surrounding the vehicle couldn't figure out why the heck the electrical system seemed to be dead, my brother-in-law felt pretty vindicated.
This reminds me of a strange recurring experience I used to have with electronics in my house.
In that house,on certain days of the week at midnight, something that sounded like recordings of political speeches from WW2 would be audibly (but faintly so) from all of stationary electronics. It was so faint, that it was usually hard to pinpoint any one source, and the words used weren't understandable, so it usually sounded like it was coming from everywhere all at once, and like it was an old recording.
This would happen even if they were unplugged. Even if my power went out.
I though I was either going crazy, or my house was literally haunted by hitlers ghost.
Well one day I was in my car and recognized something on the radio that reminded me of this spooky problem I had.
It was the signon of a Catholic am radio station that opened up with Gregorian chanting, and a sermon. This signon happened at the same time on the same days of every week.
Turns out, the wiring in that house was somehow functioning as an am radio receiver, and some common components would vibrate out the audio encoded in the radio signal.
My son's laptop screen kept shutting off while he was playing American Truck Simulator. His truck would drive off the road while the screen was black.
Every time I played on his laptop, this did not happen. He swore he was cursed.
This went on for many days, with many instances where it would happen for him but not for me. Then one day I just sat and observed him while he played, looking for any difference. That's when I noticed his watch band is metal with a magnetic clasp. The position of his wrist on the laptop was tripping the hall sensor, making the laptop think the lid was closed.
Him and I (and his mother) were glad to find out he is not cursed. :-D
These are the kinds of stories that made Car Talk so much fun. It was so much more than just hearing about mechanics repairing cars. It was the fact that the situations were so odd and unusual that the stories were interesting. It was also fun hearing how these mechanics had been around so long and seen so many of these unusual situations that they became normal to them. It didn't hurt that they were good story tellers
> Moral of the story: even insane-looking problems are sometimes real.
To me the moral of the story (and my experience) is: user's problems are usually real, but don't trust their ability to diagnose the actual cause.
My wife's car right now is low on windshield wiper fluid. It warns us at a particular spot while we are driving. Not a particular distance from home, a specific geolocation. First thoughts were that it was the amount of time as she went to something and came back. But it does the on screen and audible warning at that location every time, no matter what was driven before that, whether it was 5 minutes, 5 miles, an hour, or 60 miles. The only additional clue is that exact same spot is a dead zone for most phones/carriers.
We got the wiper fluid filled, so the mystery is in remission, but I'm wondering if all warnings will pop up right then. I'm guessing it has something to do with the telemetry of the car being nudged in that spot, waking up and saying something.
One night while in the computer science lab doing a Java assignment, my professor for that class happened to walk by the lab and quipped to me, "oh, good luck on _that_ machine."
He explained: once upon a time, the machine refused to run _any_ Java programs, and would spectacularly crash and burn instead. C++ fine, python fine, anything Java was a hard nope. He didn't believe this at first until his program also started crashing the machine.
It took a tech, him, and another professor about two weeks to work out that the JVM happened to allocate the same RAM address to the integer 12 on that particular machine every time the JVM started. The actual chip of RAM that contained that hardware address was faulty, so whenever the machine tried to allocate to that address, it would crash.
Swapping out the bad RAM stick immediately solved the problem.
This story is a playful example of confounding:
> In causal inference, a confounder (also confounding variable, confounding factor, extraneous determinant or lurking variable) is a variable that influences both the dependent variable and independent variable, causing a spurious association. Confounding is a causal concept, and as such, cannot be described in terms of correlations or associations. The existence of confounders is an important quantitative explanation why correlation does not imply causation. Some notations are explicitly designed to identify the existence, possible existence, or non-existence of confounders in causal relationships between elements of a system. / Confounds are threats to internal validity.
https://en.wikipedia.org/wiki/Confounding
Here is a sketch of a statistical model that shows a confounder (a variable affecting both the dependent and independent variables)
S = f(H, I, T)
`S`: car starting or not (dependent variable)
`H`: how hot is the car engine (independent variable)
`I`: ice cream type chosen (independent variable)
`T`: time taken to buy the ice cream (a confounder)
When I moved into my last house, I had a fun electrical problem. We moved in October and I would end the day by taking the trash out. Now, my garbage can was outside the door from my garage to the backyard. One night the light outside that door didn't go on. It was late, so I figured I'd look at it the next day. Worked fine all day, then at night it didn't work again. It was a new house, so I called the builder.
The electrician came out to check it and gave me the most incredulous look when I told him it didn't work at night. But he went to take a look. Came back later and said that the wire was barely touching the light fixture. So, at night, when it go colder it would slightly pull back and no longer be touching. During the day it would warm up, expand and would work just fine.
I was trying to fix the flaky Bluetooth connectivity on my Subaru. There was one post on the forums that said "Just slam the glove compartment closed really hard."
Knowing that this was bullshit, I tried everything else to no avail. I finally caved and slammed the glove compartment. To my surprise, Bluetooth has been solid ever since...
Simpler but funny tech support anecdote: at some point, someone called me over and said their keyboard would print the letter 'e' repeatedly. I tried another keyboard of course, and it still did it! Rebooted the computer, still 'e's all over.
Finally, I looked at her mouse: it was half of a wireless mouse-keyboard combo. Turns out, the keyboard hadn't been turned off before it went in the pile, and other keyboards on top of it were pressing its keys, or rather just the one key.
Like all prefunded startups we rented a crappy apt to work out of. Outside of the neighboring fire station which woke us at all hours, everything worked fine.
A month into summer, the internet started to die at around 1pm… then magically restore itself in the early evening before the tech would arrive. This went on for a solid week or more.
Eventually the provider agreed to send a tech immediately after we called. On first inspection, everything looked good. Thankfully he was diligent and found that an old pin based IC was likely expanding in the heat and every so slightly unseating itself in its socket. Properly seating it and adding some hot glue solved the issue.
Never underestimate heat dissipation in product design.
This reminds me of the "my monitor blinks every time I sit down in my office chair" turning out to be EMI spikes from the gas lift affecting the signal traveling on monitor cables.
A DisplayLink KB article even mentions it (and the associated white paper about the issue), stating:
Surprisingly, we have also seen this issue connected to gas lift office chairs. When people stand or sit on gas lift chairs, they can generate an EMI spike which is picked up on the video cables, causing a loss of sync. If you have users complaining about displays randomly flickering it could actually be connected to people sitting on gas lift chairs. Again swapping video cables, especially for ones with magnetic ferrite ring on the cable, can eliminate this problem. There is even a white paper about this issue.
Reminds me of a short book I like that talks about these sort of tings titled "An Engineer's Guide to Solving Problems." It starts with a similar situation - "The Dog Barks When the Phone Rings." Eventually, you get to a section called “If I Could See it, I Could Fix it!” which discusses the importance of understanding the problem before attempting fixes.
I find it incredible that not only a family would eat ice cream every, but that they would travel to get it everyday instead of just having it on hand in the house.
This reminds me of my village story. Two brothers married and lived next to each other. The wives hated each other and competed on everything. If brother 1 got something brother 2 had to buy the same thing but bigger.
Over time they ended up building homes in this manner too and one of the brothers purchases the largest available plastic watertant on roof. Offended the other brother decided that he will build even bigger water tank. So they built a massive concrete water taken on the roof.
However the problem with this tank was that it would simply not retain the water which was pumped into it via a small motor pump which pulled the water from nearby well.
This resulted into the brother accusing the brother of "black magic" and engaging in daily fist fights and abuse.
Eventually someone figured out that the person who connected the intake pipe connected it at the bottom of the tank. So wehn the tank was full the water would simply go back to the well.
I was debugging a Heisenbug once, developing embedded FW for a mobile phone.
After some time, I noticed that the phone seemingly only crashed in one area of the open office floorplan where I was working.
I started walking around the office testing this theory, not really believing it. But after a while, I had hard evidence that the bug would only manifest once I entered that part of the office.
When I came to terms that I wasn’t hallucinating, I realised what the problem was. There was poor reception in that part of the office, causing the phone’s modem to switch from 4G wideband to narrowband (glossing over details here), which triggered the bug.
Easy to see with hindsight, but I was very confused there and then
We had a problem like this too at a satellite company. A customer claimed their service was cutting out every day between 3-4pm. There were no service outages, and it only seemed to affect them, but we did see their modem drop consistently each day. Turned out it was a very low elevation location to where the dish was virtually horizontal. They received UPS packages almost daily around that time, and the truck was blocking the signal long enough to lose service momentarily.
Oh man, I'm currently fighting a problem with a mid-70s coupe that's driving me equally batty.
Randomly during longer trips, the car will just die for no discernible reason. It's the Car of Theseus at this point with how much I've replaced, but the issue persists, and the nature of these intermittent problems makes debugging a nightmare. More puzzling still that the car starts up fine after a short nap.
I can't vouch for the veracity of this story, but an older engineer who worked on a particle accelerator once told me this story from the 1980s:
The particle accelerator would start overheating every day right after lunch time. They eventually figured out that enough people were using the bathroom after lunch that it was affecting the water pressure in the cooling system!
I'm not very good at retelling stories.
Is this the new 500 mile story (or the second-place replacement?)
I don't know whether this is true or not, but either way: incredible. Love this, and love the company for actually sending someone on company time (!??!?!?!?) to check it out. <3 :'))))
> Vanilla, being the most popular flavor, was in a separate case at the front of the store for quick pickup.
This feels very contrived; who organises a supermarket like that? "Where's the ice-cream?" "Which flavour? We keep them separately, to make an anecdote work."
(Like, the moral of the story still works, but the specifics feel very dubious...)
Fabricated anecdotes like this miss the point: debug is a multi-layer approach, not a think-it-through logic puzzle. In reality, the driver would have encountered vapor lock in multiple situations, not just shopping, and as a result I think this example does more damage than good by promoting a furrowed-brow approach.
So this guy never ever restarted his car after a short interval except for when he bought vanilla ice cream? Additionally, he never varied the time intervals in the shop when he was buying aforementioned ice cream?
Is it is an understatement to suggest this is a highly unlikely circumstance?
A few years back, I had the opportunity to work on a project in rural India. Our team was responsible for creating a new patient booking system for a community hospital. We designed a web-based application that could be accessed through the hospital's intranet.
Everything seemed to be going well, but then we hit a snag. Our server began restarting unexpectedly at random intervals throughout the day. Initially, we were baffled. We assumed that the application was crashing because it could not handle the load, and we spent hours digging through code and logs.
But the real culprit turned out to be something entirely different. The server was located next to a storage cabinet, and its power supply line was inconveniently blocking the cabinet's door. Whenever the nurses needed to access the cabinet, they would simply unplug the server, get what they needed, and then plug it back in, oblivious to the chaos it caused in the system.
It taught us the importance of considering all variables, even the human ones, when troubleshooting technical issues. And so, we moved the server, cleared the path to the storage cabinet, and things were back to normal :)
Someone in my extended family had a Toyota Highlander which would randomly decide "you don't need window defog today", leading to a near inability to drive the car if fogging was an issue. The dealer didn't believe him until one day it actually happened while they were watching. They had no idea what to do or how to fix it. Solution: Trade the thing in against a Honda Passport. He's had that for quite a while now and I haven't heard any complaints.
On the other hand, my previous Honda Civic with its dual voltage electric system (computer thinks the battery needs charging? Generate 14.4V. Computer thinks it doesn't? Generate 12.6V) caused us considerable grief until we just started driving around with the headlight switch on all the time (this forced it into the higher charging voltage). This "feature" is not well known/understood even by mechanics and has probably caused untold numbers of alternator replacements.
Current Civic is so automated that even the headlight on and high/low beam is under the computer's control. Hopefully no weird chronic computer bugs in this one.
I had a single user whose machine would have intermittent disconnections while compiling his codebase. Not every time, but some of the time and every day, at that. This was back in the day when compiling the linux kernel was a 1hr+ task.
The key piece of evidence was when we moved a printer into his office, and it had the same disconnects, at the same times. Every day around midday. Not one disconnection, mind you - starting at about 11 the disconnects would get more and more frequent. From noon to one it was every few minutes. In the afternoon and evening it was just fine. It just so happened that the engineer would work in the morning, and his first compiles if the day were around that time. So, compiles were a red herring.
In the end the problem was the kitchen one floor down from his office, where a microwave was heating up peoples' lunches and causing enough EMI to disrupt (in those days rather brittle) wifi. We bought them a new microwave, and the disconnects stopped.
This reminds me of the case of the 500 mile email:
I have one of these to share. My first job out of college the office had an issue where at ~3pm every day, the network would go down for a minute or two, then be slow for a while after coming back up. This as a digital marketing company, so lots of file/image/video uploads happening all the time. We ultimately determined it was the router hitting max load, crashing then rebooting and having to manage all the pending uploads. From everyone realizing it was getting close to the end of the day, and making sure their video uploads were started. The company had grown a lot in the last year, but was still using a home grade router/wap combo. I replaced it with a UniFi AP and an UniFi edgerouter. And the problem went away!
My DSL internet would get incredibly slow at certain hours of the day when web traffic would be expected to be heavy (evening, as people are getting home, and around 10, as people are watching Netflix). AT&T refused to believe that the cause was congestion. No no, they said, it must be that there's a sprinkler that's going on at the same time each day, or a microwave that was causing a problem with your internet. Eventually they sent someone out and discovered that it was, in fact, a congested circuit that I was on. I couldn't believe they were so steadfast in claiming that it was some crazy-sounding external cause.
This was a puzzler on Car Talk at least once. There are more like it: https://www.cartalk.com/radio/puzzler
One past thread. Others?
Car allergic to vanilla ice cream (2000) - https://news.ycombinator.com/item?id=13347852 - Jan 2017 (133 comments)
Since 2000 we they have publicly disclosed Janet Jackson crashing harddrives:
https://www.billboard.com/music/pop/janet-jackson-rhythm-nat...
and moving a controller crashes the PS1.
https://www.gamedeveloper.com/programming/my-hardest-bug-eve...
Makes me thinks there are tons of these issues out there.
After 500 miles email story y'day now this.
Ah, reminds of my first car. I got a new Corolla and went on my merry way. But on the third day I found the steering stuck and extremely hard to maneuver. Scared the heck out of me, so I went back to the dealership.
They inspected the car and informed me the next day that there was nothing wrong with it and asked me to pick it up. I drive out of the parking lot, and again the steering gets stuck, and I just circle around and give them the car back, showing the live problem.
They inspect it another day, and aren't able to find anything wrong, and ask me to pick it back up.
Sure enough, I haven't even gotten to my home yet and the problem reappears, so I drive it back to the dealership.
This time they open up the car, the steering wheel and changed somethings in it and asked me to pick the car back up. Took me 4 mins of driving to land back in the same problem after which I made them refund my money.
I decided to shelve a few extra bucks and purchased a Camry from the same dealership. The second day while I was parking my new Camry, I found the same issue of the steering wheel getting stuck all over again, with this new car. So this was surely a me-problem, and not the car problem. After some analysis I relaized that I was turning off the car at red lights, without shifting to park. So when I turned it back on, the steering would lock itself. But if I shifted to park first then it would work perfectly!
I intentionally caused a problem like this for a co-worker in 1996.
He was a flamboyant character and had a habit of talking loudly on the phone for an hour or so a day. I modified his mouse driver to give me remote control on demand and installed it when he was away from his computer.
For the next few weeks, I would wait for his daily call and gradually move his mouse as he was trying to use his computer. He was visibly frustrated and eventually caught me when he went on a rampage during his conversation and he caught me laughing.
When I was a student of mechanical engineering many years ago I was very lucky to do an internship at a major car supplier in Japan. We did the engine application for a car on the Chinese market and being unexperienced as I was, I was more of a burden than help to the engineer who I was assigned to.
However, when driving around with our test car, from time to time the engine stalled, or stalled an restarted. This seemed to appear randomly, and since the engineer was busy with doing the engine application, he asked me if I could try to take care of the issue.
I noticed that the problem occured only during rain, then I noticed that it only occured when the windscreen wiper was set to interval. With some help from engineers from the main office, I was able to find the relais controlling the interval mode for the windscreen wiper as the root cause. I was sending voltage spikes into the cable tree, causing a reset in the central control unit. A decision was made to change the supplier for the relais (this was not supplied by our company), and the problem disappeared and never occured again. Everybody was happy and I felt really proud that I was actually able to contribute to the success of the project.
engineer, being a logical man, refused to believe that this man's car was allergic to vanilla ice cream. He arranged, therefore, to continue his visits for as long as it took to solve the problem.
the moral of this story, the engineer being a logical man refused to try reproducing the problem and devoted infinite time to watching the user shop every night. that way he gets paid to hang and eat ice cream