Win a copy of Transfer Learning for Natural Language Processing (MEAP) this week in the Artificial Intelligence and Machine Learning forum!
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Tim Cooke
  • Paul Clapham
  • Devaka Cooray
  • Bear Bibeault
Sheriffs:
  • Junilu Lacar
  • Knute Snortum
  • Liutauras Vilda
Saloon Keepers:
  • Ron McLeod
  • Stephan van Hulst
  • Tim Moores
  • Tim Holloway
  • Piet Souris
Bartenders:
  • salvin francis
  • Carey Brown
  • Frits Walraven

Stories about when you messed something up.....

 
Marshal
Posts: 4856
317
IntelliJ IDE Python Java Linux
  • Likes 1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Today my boss pointed me at this Twitter post as a reminder that it's ok to tell people about times you messed up. I replied with this:

Tim on Twitter wrote:I once took down the network interface to test how a program handled total network outage ..... on a remote machine. I had to get the IT guy from another office to drive 2 hours out to the data centre to restart it. He wasn't happy about it, at all.


Do you have any stories to share?
 
Sheriff
Posts: 7051
184
Eclipse IDE Postgres Database VI Editor Chrome Java Ubuntu
  • Likes 4
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
A long time ago, I was working at a client's site.  I had to write a little utility that went like this:

* roll through file
* if such-and-such condition, delete this record


As you can probably guess, the if-condition was wrong and I wound up deleting the entire file, live, on site.  I could hear people saying, "Why isn't this working any more?" and the blood literally drained from my face.  Luckily I was able to get the file back in an hour or so.
 
Sheriff
Posts: 15519
263
Mac Android IntelliJ IDE Eclipse IDE Spring Debian Java Ubuntu Linux
  • Likes 4
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Back in the day when floppy disks were still a thing, I made the rookie mistake of trying to do some kind of mass update of data records in some dBase data files without making backups first. This was production data. And no, the users didn't keep backups either. And no, I couldn't just roll a transaction back. I thought I was f*ked. I called my supervisor and I got my ear chewed off on the phone (I was at a remote location about an hour from our main office). Luckily, I knew something about the dBase file format and that I could fix it with a utility program that I could write in C. Unfortunately, there wasn't a development machine where I was and I needed to fix the issue that day or else work would get held up -- not good because this was a hospital pharmacy. The company I worked for at the time was a food processing and packaging company, main product pineapples. There were only two such multinational companies in the Philippines so you have a 50-50 chance of guessing which one. I was doing a pharmacy inventory project for the plantation's hospital.

Anyway, I ended up dictating the C program to my colleague over the phone--this was before the internet, intranets, modems, etc.-- I don't know if there was a fax machine but in hindsight, the thought of using one never came to mind. So once he got it compiled and an executable built, he had to put the program on a floppy disk and give it to a driver who just happened to be coming back to the plantation from the main office. I spent all day massaging the files back to health. I think I delayed work for a day but the ladies in the pharmacy were always nice to me and they even bought me a nice dinner.

Anyhoo, big lesson learned there.
 
Marshal
Posts: 68904
275
  • Likes 3
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I took CodeRanch down for half the morning once because I made a mistake about defining a banned w‍ord.
 
Junilu Lacar
Sheriff
Posts: 15519
263
Mac Android IntelliJ IDE Eclipse IDE Spring Debian Java Ubuntu Linux
  • Likes 2
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Here's a non-work related story. It's about the time I messed up my dominant hand pinky finger which to now is still not back to 100% or will it ever. This is an important finger because as an Aikido practitioner, it's the one that I use the most. It's the dominant finger when gripping a sword or staff (the index finger is the least involved) and it's the dominant finger when apply wrist and other kind of joint locks. So a bad pinky finger is no good for me.

Anyway, I like to buy knives because that's one aspect of our training, knife techniques. So this one day I had a tanto-shaped 3" blade in hand. I also had a tire in the room that I used as a target for bokken (wooden sword) practice. I don't know what came into me but I thought to myself, "Huh, I wonder if I could cut into this tire with this tanto..." since the knife was relatively new and very sharp out of the box. So I went and did a reverse grip where my pinky finger is closest to the blade and proceed to try to jab the knife into the tire to see how far in it would penetrate. It didn't. At. All.

What my dumbass self did, however, was not put my thumb on the hilt of the knife to keep my hand from sliding down the blade. And that's exactly what happened. My hand slid halfway down the blade. No pain because it was just that sharp. When I let go, there was a gaping white gash on the middle joint of my pinky finger and the blood was starting to well up in it and ooze out. So I quickly put pressure on it and put a cotton swab over it. I ended up in the emergency room about an hour later (this was lunch time and I tried the local minute clinic first, again being a dummy). They stitched me up and sent me home. At that point they didn't think I had cut the tendon because I could move the finger fine. Turns out I did actually cut the tendon and two days later, it snapped while I was brushing my teeth, sending a searing pain up my arm and leaving my pinky finger dangling uselessly out from my clenched hand, like I was daintily holding a teacup or something.

I spent the next six months first having the finger immobilized, then getting surgery to reattach the tendon, and then going through physical therapy to get it back to as near original working order as possible. I'm only at 95% original and that's probably as good as it will ever be again. I still have the long zigzagging scar from the surgery where they made two opposing flaps in the skin of my pinky finger to get to the tendon. Once in a while (more often now as I get on in age) I'll wake up with that side of my arm all numb. Constant reminder of that one other time I was being a numbnut.
 
Campbell Ritchie
Marshal
Posts: 68904
275
  • Likes 2
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I reformatted my hard drive last night. I had a backup, so I thought I'd be OK, but it wouldn't work. “No backups found.”

Eventually I found what the problem was: I had misspelt the name of the backup folder. It worked rather better when I gave it the correct folder name.
 
Rancher
Posts: 886
22
Netbeans IDE Oracle MySQL Database Tomcat Server C++ Java
  • Likes 2
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Back in the early 90's in my first consulting gig out of college, I was working with a group that didn't believe in separate development and production environments, so they did both in the same database.  I put out out a big system for them and we were going through and optimizing.  At the request of the DBA we were removing any fields that were not essential due to severe space limitations on their server.  As you might have guessed, I dropped one.  I didn't remember that it was the field that tracked all their adjustments to their financial information.  Instantly the world was reset.  I contacted the DBA to have him restore a backup, and found out he didn't believe in backing up databases that had development going on in them.

The walk from my desk to the director's office was the longest walk I have taken in my life.  I told him what happened, and he just smiled and told me it goes faster the second time around.  It took his analysts a week to re-enter all the data.

Since then, when I am in charge of backups, there are always 3 repositories and usually 5, all on different devices and usually media types.
 
Les Morgan
Rancher
Posts: 886
22
Netbeans IDE Oracle MySQL Database Tomcat Server C++ Java
  • Likes 2
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
2nd story:

About 12 years ago, we had a project that was complete.  It was on a Sun Sunfire--fully loaded 4 dual core 2.8 GHz CPU's  with all the memory maxed out on each board, and went into a SAN for our storage.  The company had a big backup with the robotic arm that stored the tapes by project in carousels. We were running Oracle on Solaris 10, the backup device was on a Windows system.  The inevitable happened, the server took a nose dive and at the entire install.  NP, I thought, I have the paranoid 3 level backup plan in Oracle, I datapump and store to DVD, and I have a parallel copy on a dev box under my desk.

As it turned out, the big robotic backups were all, toast!  I put a help ticket in to Oracle and one to the backup support. 7 weeks we went through Oracle, Sun, the backup company, Dell, and Microsoft looking for what happened and how to restore any of the regular backups.  The client wanted one of the big robotic arm backups restored and didn't trust the other media.  I finally convinced them that I could get the db back from the DVD repository of the datapump.  As it would happen though, the DVD's had write errors to them, but originally had verified flawlessly.  I was able to finally restore the database from the dev copy, which I kept in sync with the production copy except for the scripts I was working on.

4 tape backups were bad in total--the weekly full, the daily incremental, cumulative, and the off site storage.  The DVD's were corrupt too for a total of 5 backups were toast, but all verified at creation.  People ask me why I'm paranoid and have multiple backups... I just smile and say, "It makes me sleep better at night."

Biggest finger pointing, "not my problem" fiasco I've ever seen.  To this day, nobody knows what happened, but they bought a different backup system and it's been good ever since.
 
Marshal
Posts: 25436
65
Eclipse IDE Firefox Browser MySQL Database
  • Likes 5
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Early in my career I was a Highly Priced Consultant. (Don't confuse that with Highly Paid Consultant.) I was at one of the customers' offices working on their lumber brokering system. Now this was back in the days when a disk drive operated inside a box about the size of a dishwasher. I was about to leave and head back to my office so I put my briefcase on top of one of the disk drives to put away the day's working papers. The briefcase was plastic and as I closed it a static spark jumped from my finger to the lock, which was metal. A green light on the disk drive turned red and I could hear voices being raised in the big room where the lumber traders worked.

I tiptoed out of the building as quickly as possible. I don't believe the disk drive was damaged.
 
Tim Cooke
Marshal
Posts: 4856
317
IntelliJ IDE Python Java Linux
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Campbell Ritchie wrote:I took CodeRanch down for half the morning once because I made a mistake about defining a banned w‍ord.


I think I remember that day ......
 
Tim Cooke
Marshal
Posts: 4856
317
IntelliJ IDE Python Java Linux
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I'm loving these stories. Well all apart from the one where Junilu wrecks his finger, that sounds very unpleasant.
 
Junilu Lacar
Sheriff
Posts: 15519
263
Mac Android IntelliJ IDE Eclipse IDE Spring Debian Java Ubuntu Linux
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Tim Cooke wrote:I'm loving these stories. Well all apart from the one where Junilu wrecks his finger, that sounds very unpleasant.


I was looking for pictures right after the first stitching up. Be happy I haven't found them. Yet.
 
Tim Cooke
Marshal
Posts: 4856
317
IntelliJ IDE Python Java Linux
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Oh no please don't.    I'm not coming back to this thread ever again.
 
lowercase baba
Posts: 12829
52
Chrome Java Linux
  • Likes 2
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I thought i heard once that Toy Story got deleted from Pixar's servers at one point.  Someone happened to have taken a recent copy home to look at(?) and that was the only copy left...If i recall right, it almost ruined the company...I'll see if i can find a link to it later...
 
Sheriff
Posts: 7602
521
Mac OS X VI Editor BSD Linux
  • Likes 1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Worse than mess up with something in prod.

About 25 years ago my godmother was writing her Master's Dissertation and had some problems with her Windows machine. So asked my help. Dissertation didn't tell me much as I didn't know what that is, but I knew well what to do with Windows.

I was new to computers, at that time I was mostly installing, reinstalling windows to my friends. I'm as a fan of windows always knew, if there are some problems with it, quickest solution is to format the hard drive (format c: /s) and re-install the system. Bulletproof solution, problems guaranteed to go away, whether it been virus or other problems. I had a collection of software which were commonly used across the systems, so I used to install them afterwards.

So I did. When all was done, up and running smoothly again, she asked where to find her dissertation document. I was blank and so she.

She called then a company with real engineers, but this time who specializes in data recovery. After some time she said all went OK. I think she lied to me, didn't want me to know.. I was a kid after all.

As far as I know she holds diploma in Mathematics and other one in Psychology. Really, maybe all as fine, unless these were from different batches than the one I was talking about.
 
Les Morgan
Rancher
Posts: 886
22
Netbeans IDE Oracle MySQL Database Tomcat Server C++ Java
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

fred rosenberger wrote:I thought i heard once that Toy Story got deleted from Pixar's servers at one point.  Someone happened to have taken a recent copy home to look at(?) and that was the only copy left...If i recall right, it almost ruined the company...I'll see if i can find a link to it later...



Not one where I messed up, but one place I was working they needed a utility to connect to a legacy app running on an IBM mainframe when they downsized to a PC based technology server and net.  They talked to IBM and contracted the work out to them.  They put one of their junior engineers on it and after 8 months of programming they set up a demo.  Something happened the morning of the demo and the kids machine crashed.  He had not backed it up, so the project was gone and so was 8 months of labor costs paid to IBM consulting services.  Everyone involved was red faced and it was quickly swept under the rug.
 
Les Morgan
Rancher
Posts: 886
22
Netbeans IDE Oracle MySQL Database Tomcat Server C++ Java
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

fred rosenberger wrote:I thought i heard once that Toy Story got deleted from Pixar's servers at one point.  Someone happened to have taken a recent copy home to look at(?) and that was the only copy left...If i recall right, it almost ruined the company...I'll see if i can find a link to it later...



Another time one of the companies I was working for had an imaging project.  There was a small client software developed and it was decided that they would have it burned to CDs and not pushed over the LAN.  The thinking at the time was, "the extra space on the CD will give them space for saving files".  Ya, exactly! First I heard of what they were doing I told them the CDs were dead space after anything was written--unlike a CD-RW which was later developed, and didn't turn out nearly as well as advertised. To my knowledge they still have a filing cabinet full of CD's that have that little client burned on them.  The client was pushed over the LAN once they realized their mistake, so the CDs where never used.
 
Les Morgan
Rancher
Posts: 886
22
Netbeans IDE Oracle MySQL Database Tomcat Server C++ Java
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

fred rosenberger wrote:I thought i heard once that Toy Story got deleted from Pixar's servers at one point.  Someone happened to have taken a recent copy home to look at(?) and that was the only copy left...If i recall right, it almost ruined the company...I'll see if i can find a link to it later...



Another, "they did it"... First programming job out of college the company had grown and was having a bigger building constructed to house more staff.  The team/individual that was assigned the task to get network gear went right down to the wire on the design specification for the purchase.  As it turned out, the people he had been talking to didn't understand the delivery schedule or possibly our side didn't understand the reps lingo, but when the cut the PO, it came out that the equipment they wanted was still 3 years out before it would hit the market.  Red faced they buried all the original work and PO and hurriedly bought existing technology.

Another instance--back when 486's came out, one of the techs decided he was going to update a unit to a 486-25 for all their machines.  he bought 29 motherboards and memory at around $1000 each plus memory costs. The new boards didn't fit the hole patter for the stand-offs in the case, but were made due instead of just getting a drill and making new holes.  After about a month the boards drooped due to the heat and lack of support.  They started to short out spectacularly--zaps and sparks and whiffs of smoke.  The original tech was sent to fix the problems.  He found what had happened, and dumped the 19 or so boards that he had not used yet into the dumpster--and quietly hand purchasing just get new computers for the 29 people he was going to upgrade.  One of the other techs regularly dumpster dove to see what he could get--19 brand new 486-25 boards--Jackpot!  The company found out and wanted the boards back from the 2nd tech that dumpster dove.  He refused to give them back, since they had been disposed of in a publicly accessible dumpster (law states anything thrown in to a trash receptacle becomes first come first serve unless the trash is secured, the dumpster was public accessible with no locks or fencing).  When they consulted legal, the whole thing was swept under the carpet, but the first tech quickly moved on to another location.

yet more: We bought 1/2 million dollar order of computers from Dell, great machines! Mine had a problem, I was the agency's senior developer and correlated development of all project technology integration.  The problem made my machine slow down to an agonizingly slow processing pace--I could literally work math problems faster than my computer could calculate the answers. Dell sent tech's out 9 times to fix the problem.  Each time I told them the machine slowed down after about 1/2 hour of use. They replaced everything except the case, memory, and video card (it was a high performance card not integrated graphics). They replaced the mother board multiple times, the power supply multiple times, and had basically hit a dead end.  Instead of just giving us a new machine, they took the stance there could't be anything wrong with it, and it was operator error. Well, we had a 1 million dollar order in with them and my boss called our purchasing department and said: "Hold the Dell order."  A senor tech was down at my station the next day.  He sat down with me as we talked for about a half hour before he even looked at the machine, and he actually listened.  He cracked the case open and started running the machine and listened to it for 10 minutes.  As it turned out, one of the high performance video card's cooling fans were defective and after about 10 minutes of operation, it heated up and slowed down--didn't stop, but just slowed down.  It was enough to change the sync rate on the bus and it put a huge drag on the entire system.  Everything appeared to be working, and was, but the overheating on the video card slowed the hardware down to a crawl.  The tech replaced the video card and all was well with that machine until we retired it years later.  Just goes to show: a million buck has a little bit of clout!

I've literally dozens of these types of "little problems" that I've seen in my career.
 
Les Morgan
Rancher
Posts: 886
22
Netbeans IDE Oracle MySQL Database Tomcat Server C++ Java
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Not really a "messed up", but not the way to do it:

Back in 2000 maybe 2001 we wanted to move to Linux and Java, but administrators being what they are were afraid to commit.  We got the official word back to stay in Windows and develop small apps with Java.  That didn't set well with any of us techies, and me being the senior programmer in change of technology, we had a little coup. Everything from that point was Java based, all new project were done in Java.  All in coming servers were Linux based or in the case of Sun, Solaris based machines. We did this for a year behind the scenes, and just didn't tell anybody.

After a year, we were in a planning meeting again, and they opened the floor to suggestions/needs for new technology directions.  I candidly asked: How has your up-time been this last year? and Doesn't it seem like our projects are coming out faster and having less maintenance issues?  Everyone heartily agreed, and I literally responded: "You're welcome.  We switched to Linux and Java last year and just didn't tell anyone until now."  There was quite a discussion that ensued, but I ended up with a promotion out of it, the technology team got to choose what was needed after that, and it was agreed that we would at least inform the administrators what we were really doing.  They are still Java/Linux based today (about 20 years later).
 
fred rosenberger
lowercase baba
Posts: 12829
52
Chrome Java Linux
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

fred rosenberger wrote:I thought i heard once that Toy Story got deleted from Pixar's servers at one point.  Someone happened to have taken a recent copy home to look at(?) and that was the only copy left...If i recall right, it almost ruined the company...I'll see if i can find a link to it later...


This article talks about a (probably) stray "rm -rf *" command that deleted 90% of Toy Story 2...
 
Ranch Foreman
Posts: 95
6
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Not my fault but still a fun story:
I was working in a call-center back in the days, customer service in the energy industry. We were using a fiber sponsored by the company we did the work for. At one day, I was just having a break, I noticed my supervisor and our IT guy in the "server room" (just an always locked room housing a 19" rack). My supervisor was on the phone with the company we worked for while holding his other hand on the fiber - our IT guy just stand next to him.
As I passed by the door I heard my supervisor asking over the phone: "So, should I pull it?" while about 100 agents had active calls. I don't know for sure but I guess the reply must have been some confirmation as he then suddenly pulled on the connector. In this very moment there were an almost in-sync "Hello?" from all of the active calls. Although he realized his mistake and plugged the fiber right back in it took almost an hour before the connection was back.
As I heard from others neither my supervisor nor our IT guy were punished as it was seen as "fault caused by the one on the phone" who obvious never had the permission to tell any of us to pull on some cable while active shift.
Our times back then were monday to friday from 8am to 8pm and free weekends. So what ever it was that lead to this was then delayed to next weekend.
But, and you may can already guess, the following monday morning started with some issues.

I don't think I have any more to share at this level but if one wants to hear I still have several F-ups to tell.
 
catch it before it slithers away! Oh wait, it's a tiny ad:
Two software engineers solve most of the world's problems in one K&R sized book
https://coderanch.com/wiki/718759/books/Building-World-Backyard-Paul-Wheaton
    Bookmark Topic Watch Topic
  • New Topic