Raid6 Recovery on a ReadyNAS Ultra 6

Posted on Updated on

Ok, well this was the update I didn’t want to ever need to do. But, I’ve done it and so far it’s been sucessful.

About a month ago, I was sitting using my computer, and for no “apparent” reason, I got a message saying “Server is shutting down”. I thought that was weird, but it did as it said. My NetGear ReadyNAS Ultra 6 shut itself down, and didn’t want to come back up – at all.

I didn’t know what to do, so I let it sit for a while.

I’ve always kept the NAS up to date with the official stock OS of RAIDiator 4.X, so I knew it probably wasn’t going to be something weird of a customisation I’d done.

A long time ago, I’d read that this NAS had a VGA out port on the motherboard. I’d never looked at this before, but I thought that may be my go-to. So I pulled the guts out of the board to see what the problem was.

0-02-01-8f5ea694993c01899c1a59baf4c672f5b62b3c73a9f12917d0e2db4f7c55eb02_full.jpg

I figured a drive may have died, but I didn’t know which one, or how to diagnose the issue, because the NAS just wouldn’t boot. In the past I’d received emails telling me that drives were failing and that it was going to shut down, but this time… I didn’t get those messages.

So, I travelled to my local computer shop and bought a new 3TB WD Red drive, and a VGA breakout port… turns out while the computer geek who served me, didn’t notice (nor myself) that the pins on the AUS GEFORCE GT 710 card was actually 8 pins and not the required 15 pins. So… that was a waste of $40.

Instead I went to JayCar and collected all the bits I needed to make sure I could do this thing and potentially read any output of the VGA port on the board. At this stage I didn’t even know if I could do it…

0-02-01-ecfe1fb3675c99580a9113aff136182381bf790e606639edf64f57cdaaa67b35_full.jpg

So, I built all the bits and then when I booted the device… It showed up on the screen… BINGO! 🙂

0-02-01-8a3763833f531aca3ce23076f6e1c273291051ee7d3971a8ee58d9f337b5457d_full.jpg

I had a dead drive.

This NAS is just your standard computer with bit of custom work for the NAS stuff. The BIOS told me it was all normal and that the drive was dead. So… I took out my Seagate 3TB drive – yeah, you know the type… What ever you do in the future – stay clear of these drives ST3000DM001. Trust me. Just look them up in Google, and you’ll see what I’m getting at.

I replaced it with the WD Red drive and boom, it started up and started a resync.

0-02-01-2d99e24aa249ba75001b3c9ef794091da2773dd1460fbf9c0fbd9a3ca39dfcf2_full.jpg0-02-01-cbcd6e540f933d49d335e909f5395c9c9c2c724cd52bd26fc01298239643c6a1_full.jpg

I had to move the NAS to a different location and was a bit hesitant to switch it off during a resync, but, if a controlled shutdown is performed, a resync will continue safely.

Everything was going great.

I fixed up the notifications so I’d get alerted via email and also push alerts (via Pushover) and it was all good. Actually, while I’m on that, you should look in to Pushover.net for real-time alerts on your phone or device. It’s brilliant.

So, the big scary problem happened next.

I was tidying up around the place and a small power plug knocked the top of the NAS. The NAS beeped (which is weird, ’cause it beeps when the front door opens too) and I received a series of messages on my phone too…

Those messages were the very big concern.

***** File system check performed at Sun Oct  8 13:22:00 EST 2017 *****
fsck 1.42.12 (29-Aug-2014)
fsck.ext4: No such file or directory while trying to open /dev/c/c
Possibly non-existent device?

and

The paths for the shares listed below could not be found.  Typically, this occurs when the ReadyNAS is unable to access the data volume.
media
webroot
backup
[Sun Oct  8 13:22:17 EST 2017]

 

Oh dear… This meant bad news. really bad news.

So, I did some digging.

My Admin panel said “No Volumes”. I nearly freaked. My computer couldn’t located the shared drive paths either.

After doing some searching, I found a few answers I wasn’t sure about, but it couldn’t hurt.

I logged in via FTP. As suggested, there was nothing in the “/c/” path, or in the “/mount/c” path. So… this was troubling.

What I discovered which was worse is that RAID6 array had failed to mount, or load, or something. This was bad. 9TB of data missing…

This was as bad as it gets.

Or was it?

Below is a list of the commands I typed via SSH, and you can clearly see the results.

In case you don’t know how to SSH to your box, you should have already installed the Enable Root SSH Access” Addon. If you haven’t, then you’ll need to find a way in before you run these commands.

 
ReadyNas:~# cat /proc/mdstat
Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] 
md1 : active raid1 sda2[0] sdf2[8] sde2[7] sdd2[6] sdc2[9] sdb2[1]
 524276 blocks super 1.2 [6/6] [UUUUUU]
 
md0 : active raid1 sda1[0] sdf1[6] sde1[8] sdd1[7] sdc1[9] sdb1[2]
 4193268 blocks super 1.2 [6/6] [UUUUUU]
 
unused devices: <none>

ReadyNas:~# mdadm --examine
mdadm: No devices to examine

ReadyNas:~# cat /proc/mdstat 
Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] 
md1 : active raid1 sda2[0] sdf2[8] sde2[7] sdd2[6] sdc2[9] sdb2[1]
 524276 blocks super 1.2 [6/6] [UUUUUU]
 
md0 : active raid1 sda1[0] sdf1[6] sde1[8] sdd1[7] sdc1[9] sdb1[2]
 4193268 blocks super 1.2 [6/6] [UUUUUU]
 
unused devices: <none>

ReadyNas:~# mdadm --examine 
mdadm: No devices to examine

ReadyNas:~# mdadm --assemble --scan --assume-clean
mdadm:option --assume-clean not valid in assemble mode

ReadyNas:~# mdadm --assemble --scan 
mdadm: /dev/md/001AD40EDDAC:2 assembled from 5 drives and 1 rebuilding - not enough to start the array while not clean - consider --force.
mdadm: No arrays found in config file or automatically

ReadyNas:~# mdadm --scan 
mdadm: --scan does not set the mode, and so cannot be the first option.

ReadyNas:~# mdadm --assemble --scan
mdadm: /dev/md/001AD40EDDAC:2 assembled from 5 drives and 1 rebuilding - not enough to start the array while not clean - consider --force.
mdadm: No arrays found in config file or automatically

ReadyNas:~# mdadm --assemble --scan --force
mdadm: Marking array /dev/md/001AD40EDDAC:2 as 'clean'
mdadm: /dev/md/001AD40EDDAC:2 has been started with 5 drives (out of 6) and 1 rebuilding.

ReadyNas:~# cat /proc/mdstat 
Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] 
md127 : active raid6 sda3[0] sdf3[6] sde3[8] sdd3[7] sdc3[9] sdb3[2]
 11702178816 blocks super 1.2 level 6, 64k chunk, algorithm 2 [6/5] [UU_UUU]
 [>....................] recovery = 2.4% (71975936/2925544704) finish=1255.9min speed=37866K/sec
 
md1 : active raid1 sda2[0] sdf2[8] sde2[7] sdd2[6] sdc2[9] sdb2[1]
 524276 blocks super 1.2 [6/6] [UUUUUU]
 
md0 : active raid1 sda1[0] sdf1[6] sde1[8] sdd1[7] sdc1[9] sdb1[2]
 4193268 blocks super 1.2 [6/6] [UUUUUU]
 
unused devices: <none>

 

As you can see, the md127 listing, this meant that the RAID6 was back! And… It was ready to resync!!

So, I cleanly restarted the box, and it’s now resyncing the data and all the shares are  working again. All shares are back. And I’m super happy I still have my data! 😀

 

I hope this helps someone. ‘Cause it was difficult to know what to do, but it was fairly simple in the end.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s