How To: Seagate BlackArmor Data Recovery after DataVolume Failed Error

The Seagate BlackArmor 440 is a RAID Capable SMB NAS. Mine consisted of 4 1 TB Seagate drives in a RAID5 array. I highly DON’T recommend it. Check out the posts in the Seagate Community Forums for “Data Volume Usage Failed status”. This error message is what led me from irritation, to denial, to despair, to confusion and finally to triumph. The tl;dr version is that one day for no apparent reason, your RAID5 array with all your data on it will simply show up as FAILED in the BA440 even if all the disks show no problem. Something in the BA440 RAID software has crapped out and the whole point of RAID is now out the window.

Update: I found this quote on the BA’s Amazon review page. “Contacted support and they said “RAID 5 config stores all parity information on drive 1” So yes if you have an issue with Drive 1, your array is useless.

As those of you who use RAID5 know, we are supposed to be able to withstand a complete failure of 1 disk and still have complete data integrity. This is why I bought the BA440. It was used as a data warehouse for old files, especially media files. 95G of media files, 7G of old (but still required) accounting data, 5G of required regulatory documents. All of it was inaccessible. It’s important to note here that not a SINGLE disk showed to have failed. A failed disk would have been (theoretically) easy to fix. This was not that. In other words, this was a huge disaster. I completely agree with the review author’s comment “I have worked in IT for 15 years and I have never come across something quite so stupid.”

If you search the Seagate forums you will find 5 basic solutions.

1. Upgrade the firmware. If you’re lucky, your array will be working after a reboot.
2. Restore from a backup. As several forum posts mention, a RAID5 NAS is –used- as a backup in many, many instances, probably the majority of them. And no I did not have a backup of my backup.
3. Send your drives to Seagate Data Recovery Services where they will be happy to recover the data for you for $3,000 to $20,000 (plus s&h). Uh, no thanks.
4. Remove the drives, attach them to a windows system and use one of the several NAS recovery software tools. I tried the following.
   • NAS Data Recovery – Runtime Software
   • ReclaiMe Raid Recovery/FileRecovery – ReclaiMe.com
   • UFS Explorer RAID recovery – SysDev Labs

   Of these, the only one that I tried that seemed to work at all was the UFS Explorer. After a 20 hour scan, I was able to view the file structure and see that it would likely be able to recover many files. The trial version only allows viewing/copying of small files. The full version (155 Euros) unlocks larger files. I was able to open some PDFs and small word docs but can’t verify that it would have worked for all files.  Had I just had the basic BA440 setup, I probably would have paid for the full version of UFS and been able to recover most if not all of my data. There was a problem though. The 95G of media files were on an iSCSI drive hosted on the BA440. Using iSCSI allowed me to have a mapped drive letter pointed directly to the BA440 rather than having to deal with a cumbersome webinterface. It also sped up file transfer (which was atrocious even sped up…another reason to avoid the BA series). UFS had no idea what to do with that iSCSI data and couldn’t even see it.

5. Get your linux-fu going and try to recover the data yourself.

This post is about how to mount your RAID5 array in Debian, create an iSCSI target so that Windows 7 can see it and get all of your data back like a superstar.

From here on this is a AS AT YOUR OWN RISK kind of tutorial. Nothing that follows should damage your data but I’m not a linux genius and you might not be either so you might want to image those disks before proceeding. Search for Clone RAID disk for details on how to do that. FWIW, I didn’t. ALSO, when you following along and the output of your system isn’t what I’m saying it should, I’m not sure I can help. You can ask but as noted, I’m not a linux expert. I’ll also note that what follows is a compilation of bits and pieces for 2 or 3 dozen various linux related websites. One of the frustrating things I found is that most of the sites had steps but didn’t really explain them (so a windows guy could understand at least) I hope that this post avoids that.

No matter if you are using method 4 or 5, the first hurdle is to get your disks attached to a machine so you can access the disks outside the BA enclosure. Remove each disk. Most tutorials will say label it, but I’m not sure it matters as 1) I won’t be using the BA again and 2) I don’t think it’s possible to repair the array without a reformat so the disk order is irrelevant. But labeling takes a second so you might as well.  I was fortunate to have 4 unused eSATA USB docks in the server room. If you don’t have any kicking around, they’re between $50-70 for a dual dock (so $100-ish total for 4 disks). You could also use eSATA cables to you motherboard if you have enough spares or buy an eSATA PCI expansion card. (also about $100 for 4 ports)

Now here, I should note I’m a windows guy and have only been playing with Linux for about 8 months. The command line doesn’t scare me though as I’m old enough to be a DOS guy too. I did happen to have a 10 year old Dell system that I had just installed Debian 7 (wheezy) on to use as a Radius Authentication Server. You can try using a live CD of various *nix distros but I had to install some packages and not sure how well that would work. The other draw back of a Live CD is that if you need to reboot or you lose power you need to re-do everything as nothing is saved. Everybody has a crappy old system stuffed in a closet, get it out & breathe new life into it.

At this point we’ll assume you have Debian 7 running and your 4 drives hooked up via USB. To keep this focused I’m going to assume you have at least –some- linux experience. You don’t need a lot but familiarity with the terminal console and basic how-to-get-things-done is required.

Open a terminal as root. This prevents you from having to sudo every command.

We need to install mdadm (Multi Disk array ADMin)

root@radiusserver: apt-get install mdadm

and the iSCSi target package and associated files

root@radiusserver: apt-get install iscsitarget

root@radiusserver: apt-get install iscsitarget-dkms

Note that the dkms package can take quite a while to install.

Some of the steps require us to make some configuration file edits. You can us vi or nano which are the nix geek method but I prefer a GUI so I use gedit. If you don’t have it you can

root@radiusserver: Apt-get install gedit

Once everything is installed we need to make sure the system can see the RAID drives.

root@radiusserver: cat /proc/mdstat

The output is as follows

Personalities : [raid1]
md3 : inactive sdd4[1](S) sde4[3](S) sdb4[0](S)
2921867757 blocks super 1.2

md2 : active (auto-read-only) raid1 sdb3[0] sde3[3] sdc3[2] sdd3[1]
521408 blocks [4/4] [UUUU]

md1 : active (auto-read-only) raid1 sdb2[0] sde2[3] sdc2[2] sdd2[1]
1044800 blocks [4/4] [UUUU]

md0 : active (auto-read-only) raid1 sdb1[0] sde1[3] sdc1[2] sdd1[1]
1043840 blocks [4/4] [UUUU]

unused devices: <none>

You’ll note that md3 is shown as inactive. These are my 4 RAID drives.

Next we’ll look to see what mdadm can find out about these disks:

root@radiusserver: mdadm --examine /dev/sd[adbe]4

This checks all devices in /dev/sd so /dev/sda4, /dev/sdb4, etc.

It finds the following arrays

/dev/sdb4:

Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 2dba9164:94ded8f1:1efe9130:56bae5be
Name : 3
Creation Time : Sat Sep 17 07:22:52 2011
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 1947911838 (928.84 GiB 997.33 GB)
Array Size : 2921867712 (2786.51 GiB 2991.99 GB)
Used Dev Size : 1947911808 (928.84 GiB 997.33 GB)
Data Offset : 272 sectors
Super Offset : 8 sectors
State : clean
Device UUID : b0d40a81:14eada47:e125cd7a:2c7bb3e5
Update Time : Wed Mar  5 10:58:46 2014
Checksum : a17ddcf9 - correct
Events : 84026
Layout : left-symmetric
Chunk Size : 64K
Device Role : Active device 0
Array State : AA.A ('A' == active, '.' == missing)

/dev/sdd4:

Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 2dba9164:94ded8f1:1efe9130:56bae5be
Name : 3
Creation Time : Sat Sep 17 07:22:52 2011
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 1947911838 (928.84 GiB 997.33 GB)
Array Size : 2921867712 (2786.51 GiB 2991.99 GB)
Used Dev Size : 1947911808 (928.84 GiB 997.33 GB)
Data Offset : 272 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 55774217:da69391e:7845a599:a732a26c
Update Time : Wed Mar  5 10:58:46 2014
Checksum : b3dad676 - correct
Events : 84026
Layout : left-symmetric
Chunk Size : 64K
Device Role : Active device 1
Array State : AA.A ('A' == active, '.' == missing)

/dev/sde4:

Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 2dba9164:94ded8f1:1efe9130:56bae5be
Name : 3
Creation Time : Sat Sep 17 07:22:52 2011
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 1947911838 (928.84 GiB 997.33 GB)
Array Size : 2921867712 (2786.51 GiB 2991.99 GB)
Used Dev Size : 1947911808 (928.84 GiB 997.33 GB)
Data Offset : 272 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 36b4b98c:90852ea4:17a9a4af:b24e9c94
Update Time : Wed Mar  5 10:58:46 2014
Checksum : ed40aeba - correct
Events : 84026
Layout : left-symmetric
Chunk Size : 64K
Device Role : Active device 3
Array State : AA.A ('A' == active, '.' == missing)

There’s a couple of things to note here.

1. Array State : AA.A (‘A’ == active, ‘.’ == missing) – mdadm sees that disk 3 of the array is missing, which as noted above, it is.
2. Array UUID : 2dba9164:94ded8f1:1efe9130:56bae5be
3. There are 3 devices found, all 1 TB drives
   • /dev/sdb4
   • /dev/sdd4
   • /dev/sde4

   That’s because I followed Seagate’s data recovery advice and tried to reclaim a disk which made it unusable to my RAID array. Fortunately, unlike the BA440, Linux knows how to rebuild a RAID5 array with one of the 4 disks missing.

4. State : clean This means that there’s nothing wrong with the array and a good chance we’ll be able to get all the data back.

The rest of the output we can ignore.

Now we have a look at the status of the RAID array

root@radiusserver: cat /proc/mdstat

Personalities : [raid1]
md3 : inactive sdd4[1](S) sde4[3](S) sdb4[0](S)
2921867757 blocks super 1.2

md1 : active (auto-read-only) raid1 sdb2[0] sde2[3] sdc2[2] sdd2[1]
1044800 blocks [4/4] [UUUU]

md3 is our unmounted RAID. You’ll note the devices from mdstat and mdadm –examine commands are the same.

At this point we need to install another package LVM2 (Local Volume Manager) This allows us to read the format of the RAID array that the BA440 uses.

root@radiusserver: apt-get install lvm2

Next thing we need to do is assemble the disk back into an array we can mount and use. We use mdadm and specify the device (from mdstat) and uuid (from mdadm –examine) of the array.

root@radiusserver: mdadm --assemble --force /dev/md3 --uuid=2dba9164:94ded8f1:1efe9130:56bae5be

(note there may be no output here. There should be. See below)

The next 2 commands activate the Volume Group

root@radiusserver: modprobe dm-mod
root@radiusserver: vgchange -ay
No volume groups found

Oops. You may find this error appears after the vgchange command. This means that the OS has control over our device (as happens when you add a USB device like our docking stations). We need to stop the device so we can apply the setting we want. Since /dev/md3 is the device we want to use:

root@radiusserver: mdadm -S /dev/md3
mdadm: stopped /dev/md3

Now if we rerun the –assemble command

root@radiusserver: mdadm --assemble --force /dev/md3 --uuid=2dba9164:94ded8f1:1efe9130:56bae5be

We see the following output:

mdadm: /dev/md3 has been started with 3 drives (out of 4).

Of course since this is -real- RAID5, a missing disk is no big deal.

Now rerun

root@radiusserver: modprobe dm-mod
root@radiusserver: vgchange -ay

We see

1 logical volume(s) in volume group 'vg0' now active

Now we run lvscan to make sure we can see the new LV status

root@radiusserver: lvscan
ACTIVE            '/dev/vg0/lv0' [2.72 TiB] inherit

Now we need to create a Mount Point (basically a directory pointer)

root@radiusserver: mkdir /media/DATAVOLUME

Now we actually mount the volume so we can use it.

root@radiusserver: mount /dev/vg0/lv0 /media/DATAVOLUME

Note: If you get an error when when trying to mount the volume (/dev/lg*/lv* does not exist), you may be running into a block size issue. This is a known issue when running a Live OS version of Debian 7.6 from a USB stick. The Ext4 file system on the BA has a block size of 65536 and the Live OS can only read a block size of 4096. The solution is to run

apt-get install fuseext2

Note #2: Please see the comment from Vlad about using fusext2 in this situation. You may encounter issues when reading large files, such as the iSCSI target LUN. Again this only seems to apply if you are using the Live OS. I did not encounter any issues from the full install.

Then run

fuseext2 -o ro -o sync_read /dev/mapper/vg*-lv* /mnt

(where * is the appropriate volume id)

This seems to be a limitation of the Live OS and not a standard install but since many will try this from a Live OS (especially Windows users), it’s important to know this potential issue.

Thanks to Keaton Forest in the comments for this workaround.
Now list the contents of the mounted directory

root@radiusserver: ls /media/DATAVOLUME

And you should see a list of files on the DATAVOLUME of the RAID array.

aquota.user  iscsi_device  jewab     Public        webserverBackup
Download     IT_Archives   PDFArchives     __raclient  wiki

Now if you start up the Nautilus file browser

root@radiusserver: gksudo Nautilus

and you should be able to browse the “normal” non iSCSI files on the just mounted volume. Copy them to a network drive or local drive other that the RAID array. If you didn’t use iSCSI, you data recovery is complete. Time for a beer.

If you did have iSCSI set up on the BB440, you’ve got more work to do.

First, if you try to access the directory iscsi_device, you will receive a permission denied error. This directory contains the actual iSCSI target (or targets if you have more than one) so we need to take ownership of the directory by running:

root@radiusserver:/etc/iscsi# chown -R <your username> /media/DATAVOLUME/iscsi_device

Now list the contents of that directory

root@radiusserver: ls /media/DATAVOLUME

In my case is have 2 iSCSI targets

iSCSI-1 SQLBackup

I’m only going to recover the data from iSCSI-1 so that’s what I’ll focus on from now on. If you need to recover multiple devices, you would just repeat these steps

Now run

root@radiusserver: sudo gedit /etc/default/iscsitarget

This is likely empty. Add the following line to make enable the system to create an iSCSI target:

ISCSITARGET_ENABLE=true

Next edit the iSCSI target configuration file

root@radiusserver: sudo gedit /etc/iet/ietd.conf

Don’t modify anything except add the following to the bottom of the file

Target iqn.2014-03.local.mynet:storage.Lun0
       Lun 0 path=/media/DATAVOLUME/iscsi_device/iSCSI-1,TYPE=fileio

This defines the Target name which is the iqn line. This is what will show up in the Discovered Targets window of the iSCSI Initiator in Windows. The second line defines the LUN (Logical Unit Number), giving it a number (0), the full path to the directory (remember that the path is CaseSensitive! That messed me up for a while) and specifies the type (fileio). FileIO is used for directories, there other types but we don’t need them.

Note: If you want to set up a second or more Target you would change the target id, Lun and path

Target iqn.2014-03.local.mynet:storage.Lun1
       Lun 1 path=/media/DATAVOLUME/iscsi_device/SQLBackup,TYPE=fileio

Now we need to allow connections to the target

root@radiusserver: sudo gedit /etc/initiators.allow

and add the following to the bottom of the file (it may be empty)

192.168.150.20 iqn.2014-03.local.mynet:storage.Lun0

Now all we need to do is restart the iSCSI service:

root@radiusserver:/etc/iscsi# sudo service iscsitarget restart
[ ok ] Removing iSCSI enterprise target devices: :.
[ ok ] Stopping iSCSI enterprise target service: :.
[ ok ] Removing iSCSI enterprise target modules: :.
[ ok ] Starting iSCSI enterprise target service:.
. ok

Now we can check to make sure the target is running and available

root@radiusserver:/etc/iscsi# iscsiadm -m discovery -t st -p 192.168.150.20

(change the ip above to your local ip address)

Then we see our target

192.168.150.20:3260,1 iqn.2014-03.local.mynet:storage.Lun0

Note here. If your Debian system has a firewall set up (iptables) you will need to allow access for port 3260. If you’ve set this up for recovery, the default Debian install does NOT use a firewall so you shouldn’t need to do anything.

Now all we need to do is head back to our windows machine and fire up the iSCSI initiator as admin. Enter the ip of the target (in the example 192.168.150.20) and hit quick connect. Your target should show up in the Discovered targets section. Click the Volumes and Devices tab and make sure there is something listed in the Volume/Mount point/device section. If not, click Auto Configure. Close the iSCSI tool and head to Explorer. You should see a new drive letter which points to the iSCSi target and you should now be able to move your files to a network or local drive.

Hey, you’re done! That calls for a celebration.

What next?

Once your data is moved off the RAID drives (and verified good!), you could reinstall them in the BA device, reformat and copy the data back, under the hopes that this won’t happen again. Not for me sorry. Fool me once…yada yada.. (of course we now know how to fix it if it happens again…)

You could void your warranty and install a version of linux to replace the OS. Search for “install linux blackarmor” for options. Note the possibility of bricking the unit is reasonably high. I might try this as I certainly won’t be going back to using the BA’s native software. It’s why I’m writing this.

Lastly, you could tuck the debian system you just built with the 4 external drives in a closet or server room and use that as your RAID5 file storage. It’s probably what I’m going to do. NOTE: I’ve added a second post outlining how to do this.

I hope this helps some of you to recover your data and saves you some grey hair!