Relating The Unrelated

[Charles Lent]

So this is #2 in my "Relating the Unrelated" stories.

I hope you enjoy them. Sorry Glenn for taking your woodworking post on a hard left with my first post on this subject. I truly didn't mean to. It was just the thought as I read your post, and responded.

So this is a new post, and #2 of "Relating The Unrelated". I'm not certain how many that I can remember, but I will try to continue here in this post, if you are all interested.


The large building that I worked in for about 20 years before my retirement was connected to many other manufacturing office, and warehouse buildings, and the facility covered 3.2 million square Feet on about 1,300 acres. It even had a cafeteria building, also attached to the manufacturing and warehouse facility. Every once in a while the lunch meal in the cafeteria would offer a healthy amount of very large shrimp in the bowl along with the salad, but these days were very few and far between and not on any schedule that I could figure out. It seemed to be quite random, but it was a new challenge for me, and I loved relating the unrelated challenges like this. It took me awhile to figure out when there was a good chance that shrimp would be in the salads, but only the first 8 or 10 bowls at most would be put out containing the shrimp. So it was necessary to be there and ready to buy and eat lunch as soon as they opened the entrance doors to the serving line those days. Being an engineer, I wasn't held to any specific time that I had to take my lunch, and I love shrimp cocktail style prepared shrimp, especially the large ones. So it took me a while, but I found a way to predict when the shrimp might be in the salads and from then on I knew when to go to lunch early, as soon as they opened the doors to the serving line, and I rarely missed getting shrimp in my salads from then on.

After finding me eating my shrimp several times, when a friend and co worker arrived to eat and saw my salad with the shrimp on the top several times, he asked me how I knew when there would be shrimp in the salads. I replied that "if I tell tell you, then I will have to kill you", because you will tell others and I won't get any more. He persisted for many weeks, and again and again when he would find me eating shrimp on my salads. It was becoming an obsession with him to find out how I knew the day before that shrimp would be in the first salads the next day. His begging and pleading made me finally give in, but I warned him that if I told him and he told anyone else than neither of us would be first in line when the serving line doors opened to get the shrimp. I didn't tell him, until he again found me eating shrimp. Then he wouldn't leave me alone and kept pestering me to tell him.

So I gave in finally and said to him in a whisper, "OK, but you have to promise to tell no one else. So I leaned over and whispered, "Watch The Trees". He yelled out "WHAT"? and everyone started paying attention to us, so I told him that I would explain later. When I finally did tell him, he, at first, didn't believe me. So the next time that I noticed the trees being moved, I told him "The Trees Are Moving". He looked at me, a bit confused at first, and then got all excited and said "I'll be joining you for lunch tomorrow". We were two of the very first in line the next day, and sure enough, there were five bowls of salad with shrimp and the rest were just plain salads

So my Big Secret was -

Whenever there was an awards dinner, or other very special occasion in the evenings, they would have a moving company contractor push the big chrome pots on wheels, containing the ornamental trees that were usually in the main hallways, down to the cafeteria to make a more Restaurant Style atmosphere in one end of the cafeteria for the special dinner. So, every time that I noticed that the trees were being moved to the cafeteria, I knew that there would almost certainly be left-over shrimp in the salads the next day. He was true to his promise, and we were the only two making the mad dash to the cafeteria the day after we saw the trees move and from then on, if one if us saw this, he would tell the other. This continued until I retired about 7 years later.

Sometimes it really pays to watch the unrelated around you, as well as the problem that you are trying to solve, and especially on the problems that nobody else could solve. I quite frequently would just watch the machines, operators, rest rooms, and any other things that might be happening in the area, to get clues about how to solve the problems that nobody else could solve. This was one of the more enjoyable solutions to the problem.

Charley

 

[Vaughn McMillan]

On the subject of Relating the Unrelated...

Years ago when I was doing a lot of computer symposiums, many of them had a special "War Stories" session where people would go up and tell stories about various computer problems they had solved. One of the presenters told the story of a time he was hired as a consultant by Purina to help debug a problem they were having at one of their labs where they tested various dog foods. It was a big warehouse building with dogs in cages stacked 4 or 5 high (don't get me started about that aspect). They had a computer controlled system that fed a measured amount of food to each of the dogs at specific times of the day. The problem was that in one area the dogs were getting overfed, and the lab techs couldn't figure out what was causing it. At random times during the night, the system would just dump more food in the bowls for just this group of dogs. The consultant guy studied the computer code that was controlling the auto-feeding system and found no problems. Then he checked out all the electro-mechanical parts and again found no issues. Eventually he decided to stay one night just to watch and listen to see if he could figure out what was going on.

The dog cages were stacked next to one of the wall boxes that held electronic controllers, and that night he found the cause. Turns out the dog in the end cage of the top row of cages in this section had discovered that if he raised his leg and peed on the box, food would get dispensed for him and all his buddies in the same cell block. The Purina folks installed a plastic shield over the box and moved the mastermind of the entire operation to a different cage. Problem solved.

 

[Charles Lent]

On the subject of Relating the Unrelated...

Years ago when I was doing a lot of computer symposiums, many of them had a special "War Stories" session where people would go up and tell stories about various computer problems they had solved. One of the presenters told the story of a time he was hired as a consultant by Purina to help debug a problem they were having at one of their labs where they tested various dog foods. It was a big warehouse building with dogs in cages stacked 4 or 5 high (don't get me started about that aspect). They had a computer controlled system that fed a measured amount of food to each of the dogs at specific times of the day. The problem was that in one area the dogs were getting overfed, and the lab techs couldn't figure out what was causing it. At random times during the night, the system would just dump more food in the bowls for just this group of dogs. The consultant guy studied the computer code that was controlling the auto-feeding system and found no problems. Then he checked out all the electro-mechanical parts and again found no issues. Eventually he decided to stay one night just to watch and listen to see if he could figure out what was going on.

The dog cages were stacked next to one of the wall boxes that held electronic controllers, and that night he found the cause. Turns out the dog in the end cage of the top row of cages in this section had discovered that if he raised his leg and peed on the box, food would get dispensed for him and all his buddies in the same cell block. The Purina folks installed a plastic shield over the box and moved the mastermind of the entire operation to a different cage. Problem solved.

That's exactly why I always watch what is going on around the area where the problem that I am trying to solve is located, and it sometimes takes watching for at least 24 hours. Frequently, the cause of the problem is completely un-related to the problem that I was trying to solve. I was actually getting quite good at this during my working career.

Does anyone else want to add their similar stories? Sometimes it can be a great help when people are trying to solve problems that no one else can fix. It doesn't need to be electrical problems. Any kind of similar problem on any subject is welcome here. It's worth a good laugh, even if this unrelated problem can or can't be fixed.

Charley

 

[glenn bradley]

So this is #2 in my "Relating the Unrelated" stories.

I hope you enjoy them. Sorry Glenn for taking your woodworking post on a hard left with my first post on this subject. I truly didn't mean to. It was just the thought as I read your post, and responded.

No problem for me Charley. Like I have often said; always enjoy your posts and often learn from them. Any of us who have done something for decades will have tales to tell. We have at least two generations who will never experience that . . . uh . . . er . . . experience. Many of our jobs lasted longer than companies do today. My computing career started with "heavy iron" shops and 1200 bits per second was considered fast, Ethernet was still in the lab and 802.3 was still a "working group". Fast forward 40 years and the dot.coms had exploded and crashed, survivors had become legitimate businesses and millionaires had been made and broken. Too many stories to tell unless you have a high tolerance for boredom .

 

[Ryan Mooney]

I'll try to keep the technical gobbllty gook to a minimum here..

One thing I've found is that if you pay attention to timing a lot of things that are "obvious" to you can seem kind of magical to other people. This also often requires some deeper knowledge of HOW things work.

This goes back to my screed that there's no such thing as common sense, only deep knowledge masquerading as intuition.

There's a feature on network switches called "spanning tree", this allows the switch to make sure that if you plug switch A into switch B and then back into switch A it doesn't cause an endless loop of packets ("a broadcast storm"). This can quickly bring down a network. The problem with this is that with this enabled in the default configuration it takes the switch a minute or two to decide that a port is "safe" to transmit packets on after something is connected (basically it "listens" on the port and if it doesn't see any of it's own packets it's decides its good.. more or less).

This works great for switch<->switch communication but it can be problematic for switch->computer connections. Some setups have computers setup to boot (load their operating system) off of the network and that delay can make that fail because they time out before the switch decides that it's ok to transmit packets to that port. The solution to this is to enable a feature called "spanning tree portfast" which turns the port on immediately and then watch and see if there are loops and terminate the link later if there are (I think this is now the default on most equipment, historically it wasn't because it's more computationally expensive).

I had done some years of networking and had learned all of this the hard way... but was recently employed at a new job as a programmer. one day the tech support guy was running around swearing about how he couldn't get any computers on one half of the office to boot from the network, but they'd work fine if you plugged in an already installed computer (the time it took for the OS to load from disk was slow enough that the port would activate). I glanced up and said "you need to have netops enable spanning-tree portfast on that switch". He looked at me and was like "how can you know that, you can't see the switches configuration" and I was like "trust me on this one... and I don't care what they say about having already enabled it cause they haven't". Sure enough about a hour later he came up to me and said "well they enabled spanning tree portfast and it just works now" with a bit of wonder on his face.

Aside - if your DNS setup (domain name services - maps names like www.whatever.com <-> 192.168.0.1 IP addresses) is broken and especially if the side that maps from the IP address -> your local hostname is missing you'll occasionally get random 30-60s timeouts when loading some websites (or using other services especially things like ssh). This is MUCH less common than it used to be (I don't know of any modern setups that do this) but some older web servers would block on looking up the source hostname from the connecting IP address for logging and if it was missing you'd have to wait for the timeout which was between 30 and 60s.

 

[Charles Lent]

Which brings me to #3.

When I came to the Charlotte facility I became the electrical engineer that was responsible to get a large computer controlled manufacturing line with 18 work stations, each containing printed circuit card component insertion machines, a load station, an unload station, and a carousel storage system much like those in dry cleaner stores to distribute the work from machine to machine and to this carousel temporary storage, plus a load station and an unload station. There were four 7' high robots, two on each track, that picked up magazines (think slotted boxes) and move these magazines from the load station to the work station, and then to other stations to receive all of the electrical components needed to complete the printed circuit cards. More than one component insertion machine was required for each model printed circuit card being manufactured. The magazines of cards with all of their needed components were then unloaded and sent to a soldering station, inspection, and then to test. All of these stations were programmed to work on their own, with several computers keeping track of the status of each card, where it was. When the magazine was full, one of these big robots would come to the work station with the full magazine, reach out 5' from the track, pick up the magazine full of cards, and transport this magazine to the temporary storage or to the next work station for additional components to be inserted. Some of these cards required five and 6 moves station to station before the card was fully assembled. Many different card part numbers could be handled at the same time and each work station would receive new instructions from the main computers automatically as the magazine full of cards moved from one work station to the next.

Getting all of this assembled together required machines and robots from many companies around the World and each had their own programming requirements, connection requirements, and way of communicating with the main computers. It was a bit of a nightmare to get this line together and working well.

The 7' robots, two robots on each of two tracks, needed special communication requirements and a means of preventing a crash between them. Most of this was the responsibility of the software in one of the main computers, but there were sensors on each robot to detect end of track as well as each other ( hard lesson to learn until you have done this before - (never fully trust the program or the hardware to avoid collisions).

We started with two robots, one on each track, and then when the second two arrived, they were added and the software in the main computers were given the job of not only handling the magazines, but to keep aware of each other to avoid a crash. There was a kind-of binary sensing bar set located on one side of the track and each set was positioned for where each workstation needed the robot to stop for transferring the magazines into and out of that work station.

With the engineers from the robot company (a Swiss Company) working with me, all of this was set up and running quite quickly, with the rest of the line being built around them. Then we started testing them for stopping position accuracy, reliability, etc. and we had several programs of different simulations to run them through. This was, for the most part, working very well, but every once in a while the robot would receive a command to move to another position, but it would just sit at whatever station that it was at and not move. Only a full reset could get it working again.

The programmers kept checking their code, and I was running tests on the robots to find out why they would suddenly not move. With about 70 positions on each track, these programs took quite a while to run, and we kept having these stops, with no solution found, except to reset the offending robot.

Finally, one day I took the time to just sit and watch the robot moves for a couple of hours. Then it came to me why they were stopping. Whenever the robots were told to move a small distance, like less than a foot between stops, the robots would fail to move. Then I asked the programmer some questions about how he connected the communication leads to the computer and how his program worked step by step. We got into how each 16 bit word of data was received by the computer and how each word was transmitted back to the robot. His program was looking at a whole word at a time, but then the same communication lines for talking to the robot were changed to receive, to look for when the robot left one position and then had reached this new close-by station.

The problem was that the robot could reach this next close-by station before the computer could switch to receive and see that the robot had reached this new station. The robot was answering that it had completed the move before his program was set up to receive it, so it missed the reply completely and just sat there waiting for it's next move instruction. It's about like someone answering you before you can get all the words out of your mouth. Since you were talking when they answered you, you very likely missed at least part of their answer. So he has already answered you, but you are waiting for his answer because you were talking when he answered, so you didn't hear his answer.

When I finally managed to get him (A very bright, but new from Vietnam, programmer) to understand that he needed to be looking to see the robot leave the one station that it was on and then continue listening for it to arrive at the next station. The robot was answering the computer too soon for his program to detect this, the way that he had designed it. When he finally realized what I was saying, he re-wrote this part of his program and moved the receive signals from the one word to another. This allowed him to watch for the reply before he even sent the command. Result = no more robot stopping problems.

This wasn't so much of "watching for unrelated changes around me", but watching closely to see when the computer was missing a step, and realize why, when it had commanded a very short distance to move and not received the reply, because the robot answered too quickly for the program to detect it. The test programs were long, so these short steps weren't happening for about an hour between when a short station to station move was made again. This problem seemed to have been happening randomly because of the length of the test program, because very few moves were short moves like this.

We had been looking for this problem for about 4 months before I finally sat for a couple of hours and just watched it that day. I had been so busy getting everything else together and working, that I just wasn't spending the time needed to find this one. This manufacturing line was up and running for manufacturing about 8 months after we started assembling it. In new "One of a Kind" projects this big it was very quick, when compared to other projects that I have been on. But again, no significant recognition was received when it was finally up and running.

Charley

 

[Ryan Mooney]

One job I was at we had some big tape libraries. These were serviced by robots that would run down the library, grab the necessary tape and deliver it to or from the tape reader. The robots moved at.. impressive.. speeds. our favorite thing to so with new folks was to position them at the end of the 30' tape library and then send a command to the robot to move to the end of library where they were standing. Watching them jump back involuntarily when the robot came at them at 30+mph was always good for a chuckle.

Not quite on the same lines, but I guess similar, is don't ever underestimate what a farm boy can do with a piece of rope and a stick . I was working on a job where we were developing some wireless radios. These had fairly large and heavy 2' on the small side and more usually 4' dishes with a bunch of electronics in a heavy box behind them. The radios had to be aligned within about 1/10 of a degree which is extremely fine and narrow. The geniuses at HQ had decided that stability mattered the most and that we should pre-determine all of the angles and use extremely precise pre-cut aluminum block standoffs to align the dishes. This worked about as well as you'd expect which is to say not at all. Another fellow and I designed a dynamic alignment system where we used valve springs as pre-tensioners and all thread with nuts to do the final alignment and lock down. This worked pretty well, but we had to remove the non-functioning spacer block system in order to put in place the new brackets... being that our test setup was somewhere between 20' and 80' up on the side of a building or tower depending on where it was.. this was a minor problem considering the dishes weighed some 100's of lbs and were large and unwieldy. So I climbed up there and took some wraps with the climbing rope in a few strategic points and then used a stick to add some twist to add lift to the system.. Meanwhile the PHD on the ground was opining loudly about how all this would never work.. I then removed the original brackets.. the dish gently lifted about 2 degrees up.. so I took a half twist out of the rope tension at which point it settled back to it's desired elevation.. I then proceeded to install the new brackets quite smoothly and then removed the jury rigged rope hoist.. It's amazing what a few simple mechanical aids can do to manage the situation when you know how to use them..

 

[Charles Lent]

@Ryan Mooney,
So you high climbed too? In my early 20's I was climbing radio towers and water towers, and sometimes buildings, mostly to change the bulbs in the aircraft warning lights, but sometimes to work on the antennas too.

My highest climb was a 380+' FM Radio Tower, but it sat at the peak of an 1,800' mountain with steep slopes on it's opposite sides. The flat point where the tower was located, was large enough for the tower base and security fence, plus a 10 X 10' building that housed the transmitter, and just enough parking space was there for one vehicle. If you got all the way to that parking space, it was then necessary to back down this narrow gravel drive about 800' to a point large enough for you and your vehicle to turn around, with a steep fall off on both sides of this driveway. I was up there, probably about 5 or 6 times in about 2 years, again for the same reasons.

I had to climb their AM tower too, but it was next to their radio station in a field at the top of another nearby mountain. The challenge for this one was that the whole tower sat on a large glass insulator, since the tower itself was the antenna. Getting onto it or off of it without getting RF electrical burned was the challenge. So it was necessary to climb a high step ladder or roof of a truck next to the tower, and then jump to the tower above that glass insulator, so as to never touch the ground or the tower at the same time. While on the tower I was as safe as a bird standing on an electric wire carrying 1.300 volts or more. As long as the bird, or me, never come into contact with the ground or a grounded piece of metal at the same time we were in contact with the wire or the antenna tower, we were both safe.

When coming back down, I jumped from the tower to the grass, since jumping to the ladder or truck was so risky. I would throw my tool bag in one direction, and then jump in the opposite direction, to land free of the tower, that glass insulator, and concrete base under it, a jump of about 10', to land on the soft lawn, and it wasn't usually soft enough, but I never got injured more than a bruise or two. Now at 83, I even have problems when climbing two steps on a platform step ladder that has a railing around 3 sides of the platform.

The 146' water tower that I climbed had a steel ladder that went up the outside of one of the legs, but then tilted out to almost vertical to intersect with the walkway and railing that circled the tank about 1/2 way up it's side. There was another ladder that curved up the tank side from this walkway, to reach the aircraft warning lights. A metal trap door was next to the top of this ladder, with a metal vertical ladder inside the tank, likely to the bottom of the tank.

Climbing this water tower was a bit safer than the radio towers, because there were several ladders to reach the top. The reason for the climb was always to change the aircraft warning light bulbs, except for one. I was asked to climb up to that trap door and then go down to the water level and get two water samples for them. All went well, until I got almost to the water. Then the ladder rungs were slimy and very slippery. I got the water samples, but got wet from my waist down, and it was November. My flashlight is likely still in the bottom of that water tower. So, I managed to climb back out of that tank because it was a nice Sunny day and Sunlight was coming through that open trap door in the top of the tank. I made one more climb on that water tower, to change the aircraft warning lights after that, and before I quit high climbing. Married life and high climbing don't go well together, but I could make over a month's pay in just an hour or two when doing it, so having the extra cash made doing it acceptable for me. Most others wouldn't do it for any amount of money.

Charley

 

[Charles Lent]

On the subject of Relating the Unrelated...

Years ago when I was doing a lot of computer symposiums, many of them had a special "War Stories" session where people would go up and tell stories about various computer problems they had solved. One of the presenters told the story of a time he was hired as a consultant by Purina to help debug a problem they were having at one of their labs where they tested various dog foods. It was a big warehouse building with dogs in cages stacked 4 or 5 high (don't get me started about that aspect). They had a computer controlled system that fed a measured amount of food to each of the dogs at specific times of the day. The problem was that in one area the dogs were getting overfed, and the lab techs couldn't figure out what was causing it. At random times during the night, the system would just dump more food in the bowls for just this group of dogs. The consultant guy studied the computer code that was controlling the auto-feeding system and found no problems. Then he checked out all the electro-mechanical parts and again found no issues. Eventually he decided to stay one night just to watch and listen to see if he could figure out what was going on.

The dog cages were stacked next to one of the wall boxes that held electronic controllers, and that night he found the cause. Turns out the dog in the end cage of the top row of cages in this section had discovered that if he raised his leg and peed on the box, food would get dispensed for him and all his buddies in the same cell block. The Purina folks installed a plastic shield over the box and moved the mastermind of the entire operation to a different cage. Problem solved.

I've read this one several times now, and it gets funnier each time that I read it. Thanks, Vaughn.

Every post to this forum topic has been very enjoyable. Please keep posting them, even if not exactly meeting the topic "Relating The Unrelatable". I think we're all enjoying reading these.

Charley

 

[Ryan Mooney]

I never climbed anything really tall. And certainly nothing as sketchy as that AM tower. We had some stuff on buildings maybe 40’ or so up and on a couple towers we were in the mid section so only 60-80’ or so up. Still was a pretty good job, it was in Hawaii so the best days I’d spend the morning writing embedded control software, then go out and do a test deployment which involved a climb and if I was lucky a nice rappel or two down and then knock off work and go hit the surf break.

When I was a teenager I was the tower guy when my grandpa put the windmill back together. That was a 60’ tower on top of a several 100’ high hill so it looked like a really really long way down. When they first put it up they had a crane truck in. But the control cable snapped during a big windstorm and let it turn into the wind which literally blew the blades off (they found one over a mile downwind). So he had to have the generator rewound and new blades built. I climbed up there then hoisted up a small jib crane and used that to hoist all the parts up while grandpa ran the ground work.

There’s an old series from the bbc about Fred Dibnah who was one of the last steeplejacks and climbed big old smokestacks with hoisted ladders to fix or remove them. He was a natural character and it’s a good set of shows that are fun to watch if you can find them.

 

[Darren Wright]

I used to work for one of the largest employers in the downtown area, they owned, rented, and occupied more buildings than any other in the city.

Every so often we’d get a call from our IBM rep saying that our system (which ran our email) was reporting a drive or network issue and they’d like to have someone come run diagnostics on the system. Oddly it was never our normal tech that would come, nor would he know much about the visit or why they came

After a couple of years of this I started to notice a pattern. The call would come in, a tech would show up to run diagnostics, then within a few days later the PoTUS would pay a visit to our downtown area.

On one rare occasion they didn’t come prior, but the PoTUS didn’t come downtown either. He did visit a large manufacturing facility in the area, who I knew was an IBM shop and someone that ran their systems. I spoke to him a week or so later and asked if IBM found anything wrong during their visit. He asked, “How did you know about that?”. To which I replied, “ I didn’t, it was just a hunch they had been by.”

 

[Paul Douglass]

My feet stay on the ground as much as possible.