Tinkering is all fun and games, until it’s 4 am, your vision is blurry, and thinking straight becomes a non-option, or perhaps you just get overly confident, type something and press enter before considering the consequences of the command you’re about to execute… And then all you have is a kernel panic and one thought bouncing in your head: “damn, what did I expect to happen?”.
Off the top of my head I remember 2 of those. Both happened a while ago, so I don’t remember all the details, unfortunately.
For the warmup, removing PAM. I was trying to convert my artix install to a regular arch without reinstalling everything. Should be kinda simple: change repos, install systemd, uninstall dinit and it’s units, profit. Yet after doing just that I was left with some PAM errors… So, I Rdd-ed libpam instead of just using --overwrite. Needless to say, I had to search for live usb yet again.
And the one at least I find quite funny. After about a year of using arch I was considering myself a confident enough user, and it so happened that I wanted to install smth that was packaged for debian. A reasonable person would, perhaps, write a pkgbuild that would unpack the .deb and install it’s contents properly along with all the necessary dependencies. But not me, I installed dpkg. The package refused to either work or install complaining that the version of glibc was incorrect… So, I installed glibc from Debian’s repos. After a few seconds my poor PC probably spent staring in disbelief at the sheer stupidity of the meatbag behind the keyboard, I was met with a reboot, a kernel panic, and a need to find another PC to flash an archiso to a flash drive ('cause ofc I didn’t have one at the time).
Anyways, what are your stories?
Not me, but one I saw… dude used chmod to lock down permissions across the board… including root… including the chmod command.
“What do I do?”
🤔
“Re-install?”
Yeah, a very unfortunate one: probably, the most painful to recover from. I’d just reinstall, honesty 😅 At least with mine I could simply add the necessary stuff from chroot or pacstrap and not spend a metric ton of time tracking all the files with incorrect permissions
There’s got to be other tools though that could change the file permissions on chmod, right? Though I suppose you’d need permission to use them and/or download them.
You can dump the permissions from the working system and restore them. Quite useful when working with archives that don’t support those attributes or when you run random stuff from the web 😁
Many distros offer a automated file/directory ownership restore feature on their liveOS
One that I can remember many years ago, classic trying to do something on a flash drive and
ddmy main hdd instead.Funny thing, since this was a 5400rpm and noticed relatively quick (say 1-2 minutes), I could ctrl-c the dd, make a backup of most of my personal files (being very careful not to reboot) and after that I could safely reformat and reinstall.
To this day it amazes me how linux managed to not crash with a half broken root file system (I mean, sure, things were crashing right and left, but given the situation, having enough to back up most things was like magic)
Many years ago I was dual booting Linux and Windows XP. I was having issues with the Linux install, and decided to just reinstall. It wasn’t giving me the option to reinstall fresh, only to modify the existing install.
So I had the bright idea to just rm -rf /
Surely it’ll let me do a fresh Linux install then.
Immediately after hitting enter I realized that my Windows partitions would be mounted. I did clearly the only sensible thing and pulled the plug.
I think I recovered all of my files. Kind of. I only lost all the file paths and file names. There was plenty to recover if I just sorted though 00000000.file, 00000001.file, 00000002.file, etc. Was 00000004.file going to be a Word document or a binary from system32 directory? Your guess is as good as mine!
sudo rm -f /lib /usr/share/backup/blah blah.tar.gz
Note the space.
Might be recoverable if you had a live distro ready. Otherwise, o7.
Oh no, this was back in the days when we loaded our distros by way of a stack of floppy disks.
Not quite catastrophic but:
I’m in the process of switching my main server over from windows to Linux
I went with Deb 12 and it all works smoothly but I don’t have enough room to back up data to change the drive formats so they’re still NTFS. I was looking at my main media HDD and thought “oh, I’ll at least delete those windows partitions and leave the main partition intact.”
I found out the hard way that NTFS partitions can’t just reclaim space like that. It shuffles all the data when you change the partition. It’s currently 23 hours into the job and it’s 33% done.
I did this to reclaim 30 MB of space on a 14 TB drive.
You mean you’ve removed the service partitions used by windows and grown the main one into the freed space? Than yes, it’s not the way. 'Cause creating a new partition instead of growing the existing one shouldn’t have touched the latter at all :/
Yes, I grew the existing one. Lesson learned I guess. 30.5 hours into it and it’s at 41%.
I thoroughly backup up my slow nvme before installing a new faster one. I actually didn’t even want to reuse the installation, just the files at /home.
So I mounted it at /mnt/backupnvme0n1, 2, etc and rsynced
The first few dry runs showed a lot of data was redundant, so I geniously thought “wow I should delete some of these”. And that’s when I did a classic sudo rm -rf in the /mnt root folder instead of /mnt/dirthathadthoseredundantfiles
stupid was when I wanted to test Linux Mint on an external SSD, and didn’t check that the bootloader wasn’t going to overwrite my internal drive’s.
So anyway I’m running Linux Mint now.
That’s an interesting way to distro hop for sure
I uninstalled Python.
I was playing around with Pygame of all things, and it wasn’t behaving as the (apparently out of date) documentation was saying it should, so I figured I’d just uninstall and reinstall Python.
EVERYTHING borked. APT wouldn’t even work.
Oh that’s a good one. It feels like it should be doable and then… BAM
Ha! Came to say this too!
I tried to uninstall Python because I was just trying to minimize junk on my computer and I usually code in Bash, Node or Java.
Tinkering is all fun and games, until it’s 4 am, your vision is blurry, and thinking straight becomes a non-option, or perhaps you just get overly confident, type something and press enter before considering the consequences of the command you’re about to execute… And then all you have is a kernel panic and one thought bouncing in your head: “damn, what did I expect to happen?”.
Nah, that’s when the fun really starts! ;)
The package refused to either work or install complaining that the version of glibc was incorrect… So, I installed glibc from Debian’s repos.
:D That one is a classic. Most distributions don’t include packagers from other distros because 99% of the time it’s a bad idea. But with Arch you can do whatever you want, of course
My two things:
- I’ve heard about some new coreutils (rm, cp, cat… this time the name really fits the contents :D) and I decided to test it out. Of course it was conflicting with my current coreutils package and I couldn’t just replace it because deleting the old package would break requirements. So without thinking I forced the package manager to delete it “I’ll install a new one in just a second”. Turns out it’s hard to install a package without cp, etc :D
- I don’t remember what I was doing but I overwrote the first bytes of hdd. Meaning my partition table disappeared. Nothing could be mounted, no partitions found. Seemingly a brick.
Turns out, if you run a rescue iso, ask it to try and recognize partitions and recreate the table without formatting, Linux will come back to life as if nothing happened
Nah, that’s when the fun really starts! ;)
Well, on the upside, it definitely works better than coffee or energy drinks :D
Also, nice save with the last one!
Before installing Arch on a USB flash drive, I disabled ext4 journaling in order to reduce disk reads and writes, being fully aware of the implications (file corruption after unexpected power loss). I was confident that I would never have to pull the plug or the drive without issuing a normal shutdown first. Unfortunately, there was one possibility I hadn’t considered: sometimes, there’s that one service preventing your PC from turning off, and at that stage there’s no way to kill it (besides waiting for systemd to time out, but I was impatient).
So I pulled the plug. The system booted fine, but was missing some binaries. Unfortunately, I couldn’t use pacman to restore them because some of the files it relied on were also destroyed.
This was not the last time I went through this. Luckily I’ve learned my lesson by now
fstab bind mount for /home that I misspelled, so I couldn’t login as myself.
fstab external hdd mount that didn’t have ignore flag so PC would pop if I booted while unplugged
Accidentally booting windows after a year and it overwrite my EFI boot entry.
The best I’ve see however was an acquaintance who accidentally set perms to own user on /usr/bin
So everything went from root:root to user:user which removed all the SUID/SGID bits as well so a bunch of bins broke lol.
Believe it or not, it was actually fairly easy to fix with chmod and chown
I have an embedded device that runs as root and has busy box. I accidentally ran chmod -x on one of the busy box sym links (strings I think) and it made all of the core utils un-executable. Unfortunately chmod is a core util. Every bash command was throwing wild errors and outputing gobbledygook. You really start to sweat when ls and cat stop working.
I had full disk image of the device and started deducing the issue. Luckily I could still execute non core utils and ssh/scp was working so I wrote a little program to restore permissions, uploaded/ran the binary, and learned nothing from the experience.
Years ago I was dual-booting with Ubuntu just to try out whatever this Linux thing was that all the nerds were talking about. Liked it and played around with it, but for whatever reason I wanted to go back to just Windows, I needed the space I had partitioned off or something, can’t remember why. So I just uninstalled or deleted the bootloader somehow (maybe I just deleted the Linux partition and expected the space to clear up like normal).
Go to restart the computer… oh shit. Ohshotohshitohshitohshit.
Just straight up overwriting boot sector and superblock of my hard drive thinking it’s the USB drive.
Udev tried to warn me, saying there’s no permission, and I just typed sudo without thinking.
Then after a second I remembered USB block devices are usually writable by users, but it was too late.
USB block devices containing mountable filesystems (on Desktop systems) can generally have those filesystems mounted and files written to them by regular users; But the block device itself stays only root writeable.
So, you need root privileges either way.
(Going from memory, but also decently confident)
Found out the hard way that if you edit /etc/sudoers with anything other than visudo you best be absolutely sure the syntax is correct, otherwise sudo will refuse to read it and you’ll be locked out.
Also learned to add -rf to the rm command at the end, after I re-read it to make sure it does what it should do. Something like rm /path -rf instead of rm -fr /path. That protects you from your fat fingers hitting the enter key half way through.
Been there with sudo. Fortunately,
sustill works, as well as going to another tty and logging in as root. Well, as long as the root login is enabled; otherwise that old hack withinit=/bin/bashmay work, unless you’ve prohibited editing kernel cmdline in the boitloader or decided on efistubIIRC the root account was disabled (with no password), so I resorted to my trusty SystemRescueCD pen to fix things. Never leave home without it.
Deleted my entire efi partition while trying to install some grub themes.
And then my backup didn’t work when I tried to restore it.
I have pretty colours now though, so it was all worth it :)
😁
Let’s see: Unintentionally making a proxy accessible to anyone online
Accidentally deallocating an ext4 partition and then having to run testdisk on it
Trying to manually create a grub entry and corrupting the bootloader
Installing a arch derivertive and having it silently overwrite grub
Installing puppy Linux and then trying to get it to use apt
Incorrect use of ppa’s on mint resulting in very old packages being installed
And many others besides
