How To: Seagate BlackArmor Data Recovery after DataVolume Failed Error

The Seagate BlackArmor 440 is a RAID Capable SMB NAS. Mine consisted of 4 1 TB Seagate drives in a RAID5 array. I highly DON’T recommend it. Check out the posts in the Seagate Community Forums for “Data Volume Usage Failed status”. This error message is what led me from irritation, to denial, to despair, to confusion and finally to triumph. The tl;dr version is that one day for no apparent reason, your RAID5 array with all your data on it will simply show up as FAILED in the BA440 even if all the disks show no problem. Something in the BA440 RAID software has crapped out and the whole point of RAID is now out the window.

Update: I found this quote on the BA’s Amazon review page. “Contacted support and they said “RAID 5 config stores all parity information on drive 1” So yes if you have an issue with Drive 1, your array is useless.

As those of you who use RAID5 know, we are supposed to be able to withstand a complete failure of 1 disk and still have complete data integrity. This is why I bought the BA440. It was used as a data warehouse for old files, especially media files. 95G of media files, 7G of old (but still required) accounting data, 5G of required regulatory documents. All of it was inaccessible. It’s important to note here that not a SINGLE disk showed to have failed. A failed disk would have been (theoretically) easy to fix. This was not that. In other words, this was a huge disaster. I completely agree with the review author’s comment “I have worked in IT for 15 years and I have never come across something quite so stupid.”

If you search the Seagate forums you will find 5 basic solutions.

  1. Upgrade the firmware. If you’re lucky, your array will be working after a reboot.
  2. Restore from a backup. As several forum posts mention, a RAID5 NAS is –used- as a backup in many, many instances, probably the majority of them. And no I did not have a backup of my backup.
  3. Send your drives to Seagate Data Recovery Services where they will be happy to recover the data for you for $3,000 to $20,000 (plus s&h). Uh, no thanks.
  4. Remove the drives, attach them to a windows system and use one of the several NAS recovery software tools. I tried the following.
      • NAS Data Recovery – Runtime Software
      • ReclaiMe Raid Recovery/FileRecovery – ReclaiMe.com
      • UFS Explorer RAID recovery – SysDev Labs

    Of these, the only one that I tried that seemed to work at all was the UFS Explorer. After a 20 hour scan, I was able to view the file structure and see that it would likely be able to recover many files. The trial version only allows viewing/copying of small files. The full version (155 Euros) unlocks larger files. I was able to open some PDFs and small word docs but can’t verify that it would have worked for all files.  Had I just had the basic BA440 setup, I probably would have paid for the full version of UFS and been able to recover most if not all of my data. There was a problem though. The 95G of media files were on an iSCSI drive hosted on the BA440. Using iSCSI allowed me to have a mapped drive letter pointed directly to the BA440 rather than having to deal with a cumbersome webinterface. It also sped up file transfer (which was atrocious even sped up…another reason to avoid the BA series). UFS had no idea what to do with that iSCSI data and couldn’t even see it.

  5. Get your linux-fu going and try to recover the data yourself.

This post is about how to mount your RAID5 array in Debian, create an iSCSI target so that Windows 7 can see it and get all of your data back like a superstar.

From here on this is a AS AT YOUR OWN RISK kind of tutorial. Nothing that follows should damage your data but I’m not a linux genius and you might not be either so you might want to image those disks before proceeding. Search for Clone RAID disk for details on how to do that. FWIW, I didn’t. ALSO, when you following along and the output of your system isn’t what I’m saying it should, I’m not sure I can help. You can ask but as noted, I’m not a linux expert. I’ll also note that what follows is a compilation of bits and pieces for 2 or 3 dozen various linux related websites. One of the frustrating things I found is that most of the sites had steps but didn’t really explain them (so a windows guy could understand at least) I hope that this post avoids that.

No matter if you are using method 4 or 5, the first hurdle is to get your disks attached to a machine so you can access the disks outside the BA enclosure. Remove each disk. Most tutorials will say label it, but I’m not sure it matters as 1) I won’t be using the BA again and 2) I don’t think it’s possible to repair the array without a reformat so the disk order is irrelevant. But labeling takes a second so you might as well.  I was fortunate to have 4 unused eSATA USB docks in the server room. If you don’t have any kicking around, they’re between $50-70 for a dual dock (so $100-ish total for 4 disks). You could also use eSATA cables to you motherboard if you have enough spares or buy an eSATA PCI expansion card. (also about $100 for 4 ports)

Now here, I should note I’m a windows guy and have only been playing with Linux for about 8 months. The command line doesn’t scare me though as I’m old enough to be a DOS guy too. I did happen to have a 10 year old Dell system that I had just installed Debian 7 (wheezy) on to use as a Radius Authentication Server. You can try using a live CD of various *nix distros but I had to install some packages and not sure how well that would work. The other draw back of a Live CD is that if you need to reboot or you lose power you need to re-do everything as nothing is saved. Everybody has a crappy old system stuffed in a closet, get it out & breathe new life into it.

At this point we’ll assume you have Debian 7 running and your 4 drives hooked up via USB. To keep this focused I’m going to assume you have at least –some- linux experience. You don’t need a lot but familiarity with the terminal console and basic how-to-get-things-done is required.

Open a terminal as root. This prevents you from having to sudo every command.

We need to install mdadm (Multi Disk array ADMin)

root@radiusserver: apt-get install mdadm

and the iSCSi target package and associated files

root@radiusserver: apt-get install iscsitarget
root@radiusserver: apt-get install iscsitarget-dkms

Note that the dkms package can take quite a while to install.

Some of the steps require us to make some configuration file edits. You can us vi or nano which are the nix geek method but I prefer a GUI so I use gedit. If you don’t have it you can

root@radiusserver: Apt-get install gedit

Once everything is installed we need to make sure the system can see the RAID drives.

root@radiusserver: cat /proc/mdstat

The output is as follows

Personalities : [raid1]
md3 : inactive sdd4[1](S) sde4[3](S) sdb4[0](S)
2921867757 blocks super 1.2

md2 : active (auto-read-only) raid1 sdb3[0] sde3[3] sdc3[2] sdd3[1]
521408 blocks [4/4] [UUUU]

md1 : active (auto-read-only) raid1 sdb2[0] sde2[3] sdc2[2] sdd2[1]
1044800 blocks [4/4] [UUUU]

md0 : active (auto-read-only) raid1 sdb1[0] sde1[3] sdc1[2] sdd1[1]
1043840 blocks [4/4] [UUUU]

unused devices: <none>

You’ll note that md3 is shown as inactive. These are my 4 RAID drives.

Next we’ll look to see what mdadm can find out about these disks:

root@radiusserver: mdadm --examine /dev/sd[adbe]4

This checks all devices in /dev/sd so /dev/sda4, /dev/sdb4, etc.

It finds the following arrays


/dev/sdb4:

Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 2dba9164:94ded8f1:1efe9130:56bae5be
Name : 3
Creation Time : Sat Sep 17 07:22:52 2011
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 1947911838 (928.84 GiB 997.33 GB)
Array Size : 2921867712 (2786.51 GiB 2991.99 GB)
Used Dev Size : 1947911808 (928.84 GiB 997.33 GB)
Data Offset : 272 sectors
Super Offset : 8 sectors
State : clean
Device UUID : b0d40a81:14eada47:e125cd7a:2c7bb3e5
Update Time : Wed Mar  5 10:58:46 2014
Checksum : a17ddcf9 - correct
Events : 84026
Layout : left-symmetric
Chunk Size : 64K
Device Role : Active device 0
Array State : AA.A ('A' == active, '.' == missing)

/dev/sdd4:

Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 2dba9164:94ded8f1:1efe9130:56bae5be
Name : 3
Creation Time : Sat Sep 17 07:22:52 2011
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 1947911838 (928.84 GiB 997.33 GB)
Array Size : 2921867712 (2786.51 GiB 2991.99 GB)
Used Dev Size : 1947911808 (928.84 GiB 997.33 GB)
Data Offset : 272 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 55774217:da69391e:7845a599:a732a26c
Update Time : Wed Mar  5 10:58:46 2014
Checksum : b3dad676 - correct
Events : 84026
Layout : left-symmetric
Chunk Size : 64K
Device Role : Active device 1
Array State : AA.A ('A' == active, '.' == missing)

/dev/sde4:

Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 2dba9164:94ded8f1:1efe9130:56bae5be
Name : 3
Creation Time : Sat Sep 17 07:22:52 2011
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 1947911838 (928.84 GiB 997.33 GB)
Array Size : 2921867712 (2786.51 GiB 2991.99 GB)
Used Dev Size : 1947911808 (928.84 GiB 997.33 GB)
Data Offset : 272 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 36b4b98c:90852ea4:17a9a4af:b24e9c94
Update Time : Wed Mar  5 10:58:46 2014
Checksum : ed40aeba - correct
Events : 84026
Layout : left-symmetric
Chunk Size : 64K
Device Role : Active device 3
Array State : AA.A ('A' == active, '.' == missing)

There’s a couple of things to note here.

  1. Array State : AA.A (‘A’ == active, ‘.’ == missing) – mdadm sees that disk 3 of the array is missing, which as noted above, it is.
  2. Array UUID : 2dba9164:94ded8f1:1efe9130:56bae5be
  3. There are 3 devices found, all 1 TB drives
    • /dev/sdb4
    • /dev/sdd4
    • /dev/sde4

    That’s because I followed Seagate’s data recovery advice and tried to reclaim a disk which made it unusable to my RAID array. Fortunately, unlike the BA440, Linux knows how to rebuild a RAID5 array with one of the 4 disks missing.

  4. State : clean This means that there’s nothing wrong with the array and a good chance we’ll be able to get all the data back.

The rest of the output we can ignore.

Now we have a look at the status of the RAID array

root@radiusserver: cat /proc/mdstat

Personalities : [raid1]
md3 : inactive sdd4[1](S) sde4[3](S) sdb4[0](S)
2921867757 blocks super 1.2

md1 : active (auto-read-only) raid1 sdb2[0] sde2[3] sdc2[2] sdd2[1]
1044800 blocks [4/4] [UUUU]

md3 is our unmounted RAID. You’ll note the devices from mdstat and mdadm –examine commands are the same.

At this point we need to install another package LVM2 (Local Volume Manager) This allows us to read the format of the RAID array that the BA440 uses.

root@radiusserver: apt-get install lvm2

Next thing we need to do is assemble the disk back into an array we can mount and use. We use mdadm and specify the device (from mdstat) and uuid (from mdadm –examine) of the array.

root@radiusserver: mdadm --assemble --force /dev/md3 --uuid=2dba9164:94ded8f1:1efe9130:56bae5be

(note there may be no output here. There should be. See below)

The next 2 commands activate the Volume Group

root@radiusserver: modprobe dm-mod
root@radiusserver: vgchange -ay
No volume groups found

Oops. You may find this error appears after the vgchange command. This means that the OS has control over our device (as happens when you add a USB device like our docking stations). We need to stop the device so we can apply the setting we want. Since /dev/md3 is the device we want to use:

root@radiusserver: mdadm -S /dev/md3
mdadm: stopped /dev/md3

Now if we rerun the –assemble command

root@radiusserver: mdadm --assemble --force /dev/md3 --uuid=2dba9164:94ded8f1:1efe9130:56bae5be

We see the following output:

mdadm: /dev/md3 has been started with 3 drives (out of 4).

Of course since this is -real- RAID5, a missing disk is no big deal.

Now rerun

root@radiusserver: modprobe dm-mod
root@radiusserver: vgchange -ay

We see

1 logical volume(s) in volume group 'vg0' now active

Now we run lvscan to make sure we can see the new LV status

root@radiusserver: lvscan
ACTIVE            '/dev/vg0/lv0' [2.72 TiB] inherit

Now we need to create a Mount Point (basically a directory pointer)

root@radiusserver: mkdir /media/DATAVOLUME

Now we actually mount the volume so we can use it.

root@radiusserver: mount /dev/vg0/lv0 /media/DATAVOLUME

Note: If you get an error when when trying to mount the volume (/dev/lg*/lv* does not exist), you may be running into a block size issue. This is a known issue when running a Live OS version of Debian 7.6 from a USB stick. The Ext4 file system on the BA has a block size of 65536 and the Live OS can only read a block size of 4096. The solution is to run

apt-get install fuseext2

Note #2: Please see the comment from Vlad about using fusext2 in this situation. You may encounter issues when reading large files, such as the iSCSI target LUN. Again this only seems to apply if you are using the Live OS. I did not encounter any issues from the full install.

Then run

fuseext2 -o ro -o sync_read /dev/mapper/vg*-lv* /mnt

(where * is the appropriate volume id)

This seems to be a limitation of the Live OS and not a standard install but since many will try this from a Live OS (especially Windows users), it’s important to know this potential issue.

Thanks to Keaton Forest in the comments for this workaround.
Now list the contents of the mounted directory

root@radiusserver: ls /media/DATAVOLUME

And you should see a list of files on the DATAVOLUME of the RAID array.

aquota.user  iscsi_device  jewab     Public        webserverBackup
Download     IT_Archives   PDFArchives     __raclient  wiki

Now if you start up the Nautilus file browser

root@radiusserver: gksudo Nautilus

and you should be able to browse the “normal” non iSCSI files on the just mounted volume. Copy them to a network drive or local drive other that the RAID array. If you didn’t use iSCSI, you data recovery is complete. Time for a beer.

If you did have iSCSI set up on the BB440, you’ve got more work to do.

First, if you try to access the directory iscsi_device, you will receive a permission denied error. This directory contains the actual iSCSI target (or targets if you have more than one) so we need to take ownership of the directory by running:

root@radiusserver:/etc/iscsi# chown -R <your username> /media/DATAVOLUME/iscsi_device

Now list the contents of that directory

root@radiusserver: ls /media/DATAVOLUME

In my case is have 2 iSCSI targets

iSCSI-1 SQLBackup

I’m only going to recover the data from iSCSI-1 so that’s what I’ll focus on from now on. If you need to recover multiple devices, you would just repeat these steps

Now run

root@radiusserver: sudo gedit /etc/default/iscsitarget

This is likely empty. Add the following line to make enable the system to create an iSCSI target:

ISCSITARGET_ENABLE=true

Next edit the iSCSI target configuration file

root@radiusserver: sudo gedit /etc/iet/ietd.conf

Don’t modify anything except add the following to the bottom of the file

Target iqn.2014-03.local.mynet:storage.Lun0
       Lun 0 path=/media/DATAVOLUME/iscsi_device/iSCSI-1,TYPE=fileio

This defines the Target name which is the iqn line. This is what will show up in the Discovered Targets window of the iSCSI Initiator in Windows. The second line defines the LUN (Logical Unit Number), giving it a number (0), the full path to the directory (remember that the path is CaseSensitive! That messed me up for a while) and specifies the type (fileio). FileIO is used for directories, there other types but we don’t need them.

Note: If you want to set up a second or more Target you would change the target id, Lun and path

Target iqn.2014-03.local.mynet:storage.Lun1
       Lun 1 path=/media/DATAVOLUME/iscsi_device/SQLBackup,TYPE=fileio

Now we need to allow connections to the target

root@radiusserver: sudo gedit /etc/initiators.allow

and add the following to the bottom of the file (it may be empty)

192.168.150.20 iqn.2014-03.local.mynet:storage.Lun0

Now all we need to do is restart the iSCSI service:

root@radiusserver:/etc/iscsi# sudo service iscsitarget restart
[ ok ] Removing iSCSI enterprise target devices: :.
[ ok ] Stopping iSCSI enterprise target service: :.
[ ok ] Removing iSCSI enterprise target modules: :.
[ ok ] Starting iSCSI enterprise target service:.
. ok

Now we can check to make sure the target is running and available

root@radiusserver:/etc/iscsi# iscsiadm -m discovery -t st -p 192.168.150.20

(change the ip above to your local ip address)

Then we see our target

192.168.150.20:3260,1 iqn.2014-03.local.mynet:storage.Lun0

Note here. If your Debian system has a firewall set up (iptables) you will need to allow access for port 3260. If you’ve set this up for recovery, the default Debian install does NOT use a firewall so you shouldn’t need to do anything.

Now all we need to do is head back to our windows machine and fire up the iSCSI initiator as admin. Enter the ip of the target (in the example 192.168.150.20) and hit quick connect. Your target should show up in the Discovered targets section. Click the Volumes and Devices tab and make sure there is something listed in the Volume/Mount point/device section. If not, click Auto Configure. Close the iSCSI tool and head to Explorer. You should see a new drive letter which points to the iSCSi target and you should now be able to move your files to a network or local drive.

Hey, you’re done! That calls for a celebration.

What next?

Once your data is moved off the RAID drives (and verified good!), you could reinstall them in the BA device, reformat and copy the data back, under the hopes that this won’t happen again. Not for me sorry. Fool me once…yada yada.. (of course we now know how to fix it if it happens again…)

You could void your warranty and install a version of linux to replace the OS. Search for “install linux blackarmor” for options. Note the possibility of bricking the unit is reasonably high. I might try this as I certainly won’t be going back to using the BA’s native software. It’s why I’m writing this.

Lastly, you could tuck the debian system you just built with the 4 external drives in a closet or server room and use that as your RAID5 file storage. It’s probably what I’m going to do. NOTE: I’ve added a second post outlining how to do this.

I hope this helps some of you to recover your data and saves you some grey hair!

Advertisements

44 Responses to How To: Seagate BlackArmor Data Recovery after DataVolume Failed Error

  1. James Wong says:

    Dear sir,
    I have the same problem as what you describe. Unfortunately I am not as technical as you are. Do you have any other way to solve the problem?
    James Wong

    • JayB says:

      Unfortunately, if you can’t manage the tools I’ve outlined, you may need to use a data recovery service. You might try http://freedatarecovery.us/ He is apparently donation driven and may be able to help at a reasonable cost. WARNING: I have not used his services and can’t be sure that he is legitimate.

      • Stephen says:

        Thank you JayB for suggesting my remote data recovery service. I’m Stephen Haran, the operator of http://www.FreeDataRecovery.us

        Linux mdadm software RAID recovery is my speciality. And yes I work from donations so the cost is low and not an issue. I’m probably a bit late to help James but I invite anyone interested in my service to stop by my site for a quick chat and/or demo.

        I’ve been running this service for three years now and have many satisfied clients from around the world and I would be happy to help.

  2. DarkSwiss says:

    Hi there
    Thank you for your post! I have the same problem and i don’t know what i should do with this useless BA. Disk1 had an error and now, the Volume can not be found.

    As you wrote in your post, you used UFS Explorer RAID recovery Tool. This what not useful for you because of your iSCSI Targets. I don’t use iSCSI so is it possible to recover the RAID array with UFS Explorer RAID? If I connect the Disks to a computer, does my mainboard need RAID support?

    Thank you for your help

    • JayB says:

      If you don’t use iSCSI or a have spare computer on which you can install linux, then UFS is probably your best bet. You’ll still need the 2 or 4 sata docks (depending on which BA you have) but your chances of recovery are pretty good. Your MoBo does not need to support RAID as all UFS does is treat each disk separately and reconstruct the array itself.

      Good luck!

      • DarkSwiss says:

        Hi, I know its a bit late but i just want to tell that i could successfully access my RAID with UFS Explorer RAID recovery. So if there’s someone without linux know-how UFS Explorer will be a good solution. The software costs about 100 EURO but you can test it before you have to buy it. All functions are available for free, so you can check if the RAID Volume is mountable. With free version you just can restore files <2kbytes. After you purchase the product, filesize is unlimited.

        In my case, the lost volume was rebuilt in a few seconds and i got all my data restored. I had 4TB data on the volume. I think 100 Euro was worth it.

      • JayB says:

        Great news! I’d have probably gone that route too had I not had the iSCSI issue to deal with. Thanks for letting me know how it went 🙂

      • DarkSwiss says:

        I was lucky that my old PC’s mainboard had 3 additional SATA slots and a power supply that had 3 additional connectors free. The rest was simple: When I started the software, the virtual RAID was built automatically and then, with UFS’s builtin explorer, I connected to my new PC’s samba share, where i could copy all my data. Then it was only a matter of time until the copy process of 4TB data was finished. 🙂

        Thanks again for your help!

  3. Joey says:

    I have this same Seagate 4-bay NAS and drive 3 experienced a failure, but I was able to bring it back online after simply re-seating it. However, all of my data seems to be lost. In the past I’ve been able to use a SATA > USB hard drive harness to view the contents of a “lost” drive in Windows. Is there an equivalent SATA > USB harness that I could use within Linux to view the contents of Drive 1 of my array? Would I be able to view the contents of the drive if I was to use a SATA > USB converted and attach it to a Linux system? Thanks for your help!

    • JayB says:

      Couple of things. Loss of drive 3 shouldn’t have affected RAID, it’s issues with drive 1 that generally seem to be the RAID killer (but who knows with these things). I’d go through the data recovery/rebuild procedure as recommended by Seagate. If that’s not possible or doesn’t work, then you would want to try the recovery procedures outlined in my post. The USB/Sata docks are in fact what you need whether you use the windows or linux methods. You will need 4 docks (well you can do it with 3) to be able to access the data. Remember that your data is spread across ALL drives. Looking at drive 1 (or any drive) isn’t going to work. You need to be able to mount at least 3 disks of the RAID array.

  4. jesus says:

    i have the same problem, the only diference is
    Array State : A..A (‘A’ == active, ‘.’ == missing)

    i dont use iscsi, and “No volume groups found” message , can help me?

    sorry for mi english thanks

  5. JayB says:

    First thing I would try is to power off the unit, and reseat the 2 drives. There does seem to be a problem with them coming loose. Power back up and hopefully that solves your problem. If that doesn’t work, try the steps outlined in the post.

    A..A means that 2 of your 4 drives can’t be seen by the unit. If 2 of your drives are -actually- bad (and not just bad as reported by the seagate unit), then that’s a bad thing. a 4 disk RAID5 array requires a minimum of 3 drives to recover the data. If 2 of your drives are gone, so is your data 😦

    Good luck.

  6. Jesus says:

    NAS was working fine and accessing all the information, fault number 3 hard drive
    change and work well as a whole day,
    the next day the number Disk 2 fails, the disks reported me they were good, reassemble it, but missed the volume, all 4 discs are good …

  7. Jesus says:

    JayB have you a phone number or chat?

  8. Jesus says:

    i can do anything?, please help me…

  9. JayB says:

    Unfortunately, I can’t really provide any more support than what is in the blog post. If you can’t or won’t try what’s outlined there, I can’t really help. I just don’t really have the time. You might try Stephen @ http://www.FreeDataRecovery.us This is what he does and apparently he can do most of it remotely,

  10. Jesus says:

    Thanks for your answer, and try the method you mentioned but I did not succeed, because only 2 units recognize me, I’ll deal with the reference you mentioned me, greetings

  11. Thank you , I had problem my NAS 400 Blackarmor, this tutorial is very good !!!!!

  12. Alright. I have all 4 drives of my Raid 5 connected to a PC with a Live Debian 7.6 running off a USB stick. I have root access and installed mdadm, lvm2. I can see that all my disks are reading fine (AAAA) but I’m stuck at the point where I mount the Raid 5 onto a directory pointer.

    root@radiusserver: lvscan
    ACTIVE ‘/dev/vg8/lv8’ [5.45 TiB] inherit

    When I try to mount the raid to a directory path it says its in read only mode and then says special device /dev/vg8/lv8 does not exist.

  13. I solved it!!!

    The problem was that the Ext4 file system had a block size of 65536. My Debian live OS could only read 4096 block sizes at biggest. So i googled for a solution and found a package called fuseext2 that can emulate the block size for the OS.

    First installed it with apt-get install fuseext2

    Then I typed:
    fuseext2 -o ro -o sync_read /dev/mapper/vg8-lv8 /mnt

    Then I was able to access the files by typing:
    cd /mnt
    ls

    I hope this helps anyone else having similar problems. I was recovering a raid 5 Ext 4 configuration from a NAS 440 Black Armor.

    • JayB says:

      Thanks for posting that! Hope it helps someone else. I wonder if that’s a limitation of the live OS? As I mentioned in the article there are some issues with live OS’s. I any case I’ll add a pointer in the article to this. Thanks again.

    • Jason Bolton says:

      I have been having problems getting Debian installed so I used CentOS, but have hit this block size issue. Have you any experience of this on CentOS 6?

  14. Hello again.

    Well this time round the same Seagate NAS as before has had one drive fail, whereas the last time I had a problem no drives had failed but the NAS configuration had become corrupt.

    At first look at the NAS > Storage > Volumes/RAID, it showed that the HDD2 had failed and the Raid 5 was failed. I then swapped out the damaged drive for a brand new one as it was still under warranty. I checked the NAS > Storage > Disk Manager section and it showed HDD2 was foreign. There was a button to adopt it so I clicked it. At this point I was very irritated that there was no place to be found where I could repair the raid 5 from within the NAS configuration page.

    I followed the same steps as before (Debian 7.6 off Live USB) and was able to access my data with only 3 out of 4 drives functioning. Big smile after that.

    Now I was at the point where I had copied all the required data onto an external USB HDD and I was going to clean the 4 drives and reformat the Raid 5 on the NAS. So I went to NAS > Storage > Volumes/RAID and to my complete surprise there was a little action button to recover the Raid. So I gave it a shot just for fun. Its sitting at 25% complete at the moment and it looks like it might even work.

    I wanted to let you folks know that there is this feature hidden in the system. But I think the NAS needs to adopt the new drive first and then reboot before it will show the repair raid action.

    Regardless I would not trust it without having a spare backup of the data in case something goes wrong.

  15. Michael says:

    This has just saved my NAS!!! Well the data, but you get the drift! Many, many thanks to you!

  16. Sorry late response: So I waited for the Raid 5 to rebuild after clicking the action button in the storage manager. The system rebooted after 100%. When I went to check the shared folders there were none. And when I navigated to the Seagate NAS from a windows machine there seemed like there were no files or folders on the system. This puzzled me because I could see that there was space being used but no files to see. So i started re-creating the same shared folders as there were before. Interestingly as the folders were created, they were populated with the original files. So it did actually recover the files, but it did not keep the shared folder permissions. Needless to say that is a shit system and I will be migrating all the files onto a Linux file sharing server.

  17. Chris says:

    Thank you so much for this article and the details. I reached the 2nd assemble command and received a different message. paraphrased: md3 rebuilt, not enough to start array.
    I then took the 4 drives back to the BA440 and they started back up. I am now waiting for the “Delayed status” to go away, shutting this NAS down, ordering a different NAS (not BA) and immediately copying everything to new NAS. It’s a shame they are able to sell it as RAID5 when it actually isn’t full RAID5.

  18. mtb1ab says:

    Thanks for the article. I haven’t been successful yet – trying to assemble a degraded array. However when I look at the array status for each of the drives one of the drives is different. They all show to be part of the same raid 5 array, they all show a clean status, but the first drive shows a status of A.A. (two missing drives). The other two drives show the array status as AAA. (one missing drive). The drive that shows two missing drives is device 2. Any ideas? That may be a drive that has some bad sectors. However, I am able to see it and have successfully imaged all 3 drives. Please help if you have any ideas.

  19. Thanks for writing this up!

    fuseext2 is actually a bad option here, and may lead to corrupted data (on read, not on disk)! It does not support ext4 extents, which the filesystem on the NAS will use. Any large files, such as the iscsi target LUN, will be using extents and will read as corrupted data using fuseext2 (silently, even).

    Instead, a more modern fuse module that supports full ext4 is available here: https://github.com/gerard/ext4fuse . It needs to be built from source, but it’s straightforward to do so. I needed one small patch to make it cope with large files properly; that’s pushed to https://github.com/vvuk/ext4fuse, and I have a pull request open with the author. So:

    wget https://github.com/vvuk/ext4fuse/archive/master.zip
    unzip master.zip
    cd ext4fuse-master
    apt-get install libfuse-dev
    CFLAGS=”-O2″ make
    sudo ./ext4fuse -o ro,sync_read /dev/mapper/vg*-lv* /mnt/foo

    Additionally, at that point, you can also optionally just mount the iscsi target directly instead of actually going through iscsi… the steps are basically:

    First, look at the full partition map:
    losetup -f /mnt/foo/iscsi_device/iscsitarget
    parted /dev/loop0
    > unit B
    > p
    > quit
    Take note of the partition you want to access’s start offset and size.

    Then remove the loopback device:
    losetup -d /dev/loop0

    And then create another loop, but explicitly limited to the partition:
    losetup -f –offset 12345 –sizelimit 12345 /mnt/foo/iscsi_device/iscsitarget

    Then mount the partition as normal:
    mount /dev/loop0 /mnt/data

  20. JayB says:

    Thanks for the comment Vlad and the procedure. I’ll make a note in the article. I will say that I didn’t have any issues reading the iSCSI lun and was able to retrieve all my data. Perhaps I got lucky 🙂 Maybe it depends on the size?

    Update: I assume I didn’t have issues because I used the full install and this problem is limited to the Live OS install. Just another reason to avoid the Live OS.

  21. Dear JayB,
    Thank you for your post above; it was very informative and helped prepare me for the worst when I had to replace my Drive 1 on my BA 400 series NAS. However, none of the issues you described arose, and everything went very smoothly with no issues at all, even though after reading your problems, I was expecting major dramas.
    My BA NAS is a 400 series. I bought it second-hand and it came with 3 x 1.5 TB HDDs, so maybe I should call it a 430. The three HDDs were configured as RAID 5.

    Maybe you can tell me why no issues arose for me.

    This is what happened.
    My NAS 430 had 3 x 1.5 TB Seagate drive in it, in bays 1, 2 and 3, and they were configured as RAID 5.
    I used it on my home network i) as a file server, and ii) as a convenient place to store drive images of other computers while I worked on them.
    About a year ago (after I’d used the BA NAS for about three years) the drive in Bay 3 failed. In anticipation of increasing the size of the BA NAS at some time in the future, I bought a new and larger Seagate 2 TB drive to replace the failed 1.5 TB drive with. I installed the new HDD in Bay 4 and removed the failed drive from Bay 3. I don’t remember why I installed the new HDD in Bay 4 instead of Bay 3, but that is what I did. The RAID 5 re-built itself without issues.

    My drive arrangement then became 1.5T – 1.5T – 0 – 2.0T as RAID 5.

    Also in 2014, I upgraded the firmware to version 4000.1411.

    Then last week, June 2015, the 1.5 T drive in Bay 2 failed.
    I pulled it and tested it on another PC with SeaTools and the report said the drive had failed, with the message “This drive is not responding to commands”.

    I decided to buy two new 2 TB drives, and replace both the remaining 1.5 TB drives, to upgrade the BA NAS to 6 TB, in RAID 5. I also planned to move the drive in Bay 4 back to Bay 3, (but one step at a time).
    I placed a new 2 TB drive in Bay 2, but the NAS would not recognize its presence, so I turned the power off, then on again to re-boot the NAS, after which it did recognize the new HDD. Then, via the Seagate GUI, I “Claimed the drive”, then asked the NAS to Recover the RAID 5 setup. It did the job flawlessly over the next 10 hours.

    My drive arrangement was now 1.5T – 2.0T – 0 – 2.0T in RAID 5.

    My next task was to replace the 1.5TB drive in Bay 1 with a new 2.0 TB drive. But then I read your article which described Drive 1 as being critical and having all the parity information on it. I was seriously concerned that if I removed Drive 1, and replaced it with a new empty drive, that the RAID would not work, and I would lose all my data. So to be safe, I made a full backup of all my data to an external USB before I started.

    I proceeded slowly in the following steps:
    I turned off my NAS,
    • I removed Drive 1,
    • I turned my BA NAS back on, running on only two HDDs, being one x 2.0 TB drive in Bay 2 and one x 2.0 TB drive in Bay 4.
    • I checked the BA NAS from my computer using Windows Explorer (Win XP), and I could see all my files even without Drive 1 installed.
    • I checked the BlackArmor Discovery GUI and I could see that my DataVolume was intact but running in degraded mode, and that under Disk Manager, only Drives 2 and 4 were listed, as I expected.
    • I turned off the BA NAS.
    • I installed a brand-new, empty, 2.0 TB Seagate drive in Bay 1.
    • I turned the BA NAS back on, and watched to check that the hardware found the new drive, and that it booted up correctly, which it did.
    • I returned to my computer and checked the BA with Windows Explorer again, and I could still see all my files.
    • I started the BA NAS Discovery/Management GUI and went into Disk Manager.
    • I clicked on the icon for Claim Disk and the NAS claimed the new 2.0 TB HDD correctly.
    • I went into Volumes and clicked on the icon for Recovering the Volume.
    • The Recovery process started and completed without a hitch over the next 10 hours, during which time I was still able to use my computer to access files on the BA NAS, and to save new files to it, even while it was going through the Recovery process.
    • The entire procedure proceeded smoothly and flawlessly. I was impressed and relieved.

    So now I am wondering:
    • If Drive 1 has all the critical parity data on it, how was it that I was able to remove Drive 1, and still access all my files, and have the NAS work, albeit in Degraded mode, just running on the HDDs in Bays 2 and 4 only?
    • And how was it able to re-build the RAID, and incorporate the new blank HDD that I installed in Bay 1 into the RAID, without any data on Drive 1, unless the file-system data is full and complete on Drives 2 and 4?

    Again, my BA NAS is running Firmware version 4000.1411. Could it be that this firmware upgrade solves all the problems that led to the issues that you experienced?

    By the way, I have a second, spare BA NAS 440 that is not being utilized at the moment. It has 4 x 1.0 TB Western Digital drives in it. I also have 4 spare 200 GB Western Digital drives that I could put into it to experiment with (or I could use the 1 TB drives too in any combination).
    So if you have any suggestions, or tests that I could carry out to investigate this further, I’m happy to try things just for curiosity. And because I have the spare empty drives, and I have lots of spare data that I could put onto them for testing, I can afford to scramble the RAID completely without fear of losing anything important.

    Thanks again for your post, it has helped me understand the process of recovering data from RAIDs next time that I might need to.

    I have already done some successful data recovery from a DEC (Digital Equipment Corp) server, running NT4, with seven SCSI HDDs in a RAID array, and without the password.
    And I made an unsuccessful data recovery attempt, from an ancient HP server with three SCSI drives in a RAID array. But they’re another story.

  22. Aaron says:

    Thank you so much. This was an amazing tutorial. It saved my life, or at least a lot of time.

  23. Sonic2k says:

    Found your article here, in search of answers when my NAS died.

    The software, UFS Explorer Professional, got all my data back. Very happy about that.

    This issue with the product, is due to bugs in crappy firmware. Mine died, while, I kid you not, I was listening to music from it… my player application suddenly stopped and reported “file not found”

    And then I noticed all the shares were gone… vanished, poof!!

    Couldn’t login to the thing, so I did the reset procedure. This broke it even further… the web interface totally full of XML errors…

    Getting rid of this piece of junk and investing in proven, reliable NAS platforms.

  24. man00ver says:

    Another victim here, using a Seagate 4-drive Business Storage unit when drive 1 failed. I rescued the files using 3 USB/SATA adapters, a USB hub, and a Windows software called “Zero Assumption Recovery” (ZAR X) with its $70 USD license and robust web instructions. Just wanted your readers to know there’s reasonably-priced hope for the non-Linux user who falls prey to this trap. Thanks for your article!

    • JayB says:

      Thanks for the tip!

      • Another, related problem has begun to appear with this product. This is the issue of disk errors.
        The disk drive in mine, has bad sectors, and somewhat large sections of them in close proximity to each other (2TB drive)

        Instead of reporting the volume as degraded, or indicating some error to the user, the system just tries to read/write the location ad infinitum and eventually crashes. The crashed state is quite distinct (the power button does nothing, and the buttons on the front panel do nothing, the stuff on the LCD remains static).

        The only way to recover from this is to power-cycle the unit, and if you have it set-up to e-mail you status reports, it will send you a mail saying there was an abnormal error condition.

        I would LOVE to wipe the stock firmware and install a flavour of Linux on this thing. But nobody has really done this.

  25. Acquadio says:

    Hi,
    thanks for the guide, I have almost done it! There is just one more step to do: access the iSCSI disks from network. I get iSCSI target running and windows box is connecting, but the LUNs are not attached, so I can´t see any disks. Worst of all, the volume I need most is formatted as ReFS so I can´t mount it directly in Linux. Is there some other way? Is it possible to copy the LUN files to some other computer and try to mount them from there?
    But again, thanks a lot for this guide, I was at least able to get the non-iSCSI data.

    Tomas

  26. JayB says:

    Awesome. Thanks for posting back. I’m sure it will help someone. 😁

  27. kortex says:

    Great post. Thank you for sharing. I was able to recover the data from a single HDD of my RAID1.

  28. digitania says:

    Wanted to add my thanks for this post. I have a BA440 that reported a drive failure in its RAID5 set. Installed a new drive and had the BA report *another* drive failure during the rebuild, which left me with an inaccessible data volume. In addition, Drive 1 was the first failed drive on my unit, which left the BlackArmor admin GUI unavailable for me when Drive 2 reported a failure. Using your post as a guideline, I was able to build a Debian system and install the four drives in my gaming rig — which had plenty of available SATA ports — and behold! I found that only one drive was actually failed, got LVM to mount the array as a VG, and there was my data! I got through it without too much difficulty, and am currently recovering my data onto a clean device.

    The fuseext2 step was necessary for me even with a non-live Debian Jessie (8.4.0) install, fyi.

    I can’t thank you enough for this. Seagate should be ashamed for their crap product and crappier support. This experience has pretty well cured me of buying another Seagate product for my home use. And, of course, this was the one with my personal stuff on it — not the one that runs my home ESXi lab cluster, which has run without a hitch since I installed it, and wouldn’t have mattered as much if it blew chunks…

    • JayB says:

      So happy I was able to help. 2 years after I wrote this post, it’s still my most read piece…so ya, Seagate should be ashamed. Thanks the meeting me know 🙂

  29. Andy Pickering says:

    Thanks so much for this! Managed to recover all required info from our little office NAS using the Debian approach. Just for your SEO – the model of my NAS was a STBP8000200 (Seagate Business Storage 4 bay) – never to be used again! Anyway – thanks again for the excellent write up!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: