I’m a little teapot 🫖

  • 1 Post
  • 268 Comments
Joined 1 year ago
cake
Cake day: September 27th, 2023

help-circle





  • seaQueue@lemmy.worldtoSelfhosted@lemmy.worldHDD randomly unmounting
    link
    fedilink
    English
    arrow-up
    1
    ·
    edit-2
    24 days ago

    I’m not sure how to get the N from session history, nor how to check my session history…

    journalctl --list-boots will list all sessions stored in the journal.

    The output is from yesterday, when the device stopped working correctly.

    I’m not familiar with linux kernel, but I can see there is definitely something wrong…

    The HDD (old) is attached to a USB hub (new), I tried switching port of the hub but the same issue happened again, if I try to mount it with sudo mount /mnt/2tb, it says it is already mounted:

    Those messages tell you what’s happening, there’s an unrecoverable error on the USB bus connecting the hard drive which is causing filesystem errors when writes fail. Diagnose that, lose the hub first and directly connect the drive to the pi, then try replacing the cable that attaches the drive if the error still occurs. I’d also check with people in the rpi community in case there are any known issues with USB on your model. There may be some pi specific USB firmware things you can do to increase reliability.

    You can also try disabling UASP for the drive in case BOT transfer somehow stabilizes the connection. You’ll lose performance but that helps with some USB storage bridges.

    Some USB storage bridges are just unreliable under Linux and crash under load, your last option is to buy another drive enclosure that’s tested and known to work correctly. I went through like 5 USB/NVMe enclosures looking for one that worked properly, that whole space is a compatibility mess.




  • seaQueue@lemmy.worldtoSelfhosted@lemmy.worldHDD randomly unmounting
    link
    fedilink
    English
    arrow-up
    12
    ·
    edit-2
    1 month ago

    Don’t just look at sdb hits in the log. Open up that entire session in journalctl kernel mode (journalctl -k -bN where N is the session number in session history) and find the context surrounding the drive dropping and reconnecting.

    You’ll probably find that something caused a USB bus reset or a similar event before the drive dropped and reconnected. if you find nothing like that try switching power supplies for the HDD and/or switching USB ports until you can move the drive to a different USB root port. Use lsusb -t and swap ports until the drive is attached beneath a different root port. You might have a neighboring USB device attached to the bus that’s causing issues for other devices attached to the same root port (it happens, USB devices or drivers sometimes behave badly.)

    Always look at the context of the event when you’re troubleshooting a failure like this, don’t just drill down on the device messages. Most of the time the real cause of the issue preceded the symptom by a bit of time.







  • Write a couple of your own toy services as practice. Write a one-shot that fires at a particular time during boot, a normal service that would run a daemon and a mount service that fires after its dependencies are loaded (like, say, a bind mount that sets up a directory under /run/foo after the backing filesystem is mounted - I do this to make fast ext4 storage available in some parts of the VFS tree while using a btrfs filesystem for everything else.) You can also write file watcher services that fire after changes to a file or directory, I use one of those to mirror /boot/ to /.boot/ on another filesystem so it’s captured by my system snapshots.

    I’d start by reading the docs so you have some ideas about what services can do, then you’ll find uses that you wouldn’t have thought of before.



  • I had to set one of these up for my SO a couple of years ago. I dropped EndeavourOS on it, installed btrbk and configured automatic snapshots on a schedule and before package installation/update in case she managed to bork things by pip installing things into system python.

    Fedora would probably work well too if you want a lower maintenance burden. I hesitate to suggest Ubuntu or Debian or their derivatives since you’ll probably want to be somewhat current with your Nvidia drivers.




  • We usually find solutions or workarounds to Nvidia driver issues within a day or two in the Arch community. The absolute worst case handling I’ve had to do was fork the Nvidia dkms package at the prior version (think nvidia-dkms-550) and run that until Nvidia themselves released a fixed version. Still pretty straightforward.

    The most helpful advice I can give to anyone running a distro maintained by folks with day jobs is “take system snapshots before updates” - do that and the worst case fix to any update problem like this is still really easy to handle, even if you’re 10 minutes out from a work call and an update just went wrong.