The Problem

I had a RAID 5 failure on an old Linux Server with special hardware connected. A RAID 5 as you probably know is a redundant disk array of hard disks. Disks are usually connected to a RAID Controller. The controller in my server is a 3ware 9500S-4LP. This controller is outdated but still available to buy at some web shops. Sometimes it is not possible to migrate outdated systems to new hardware like in this specific case.

On my server three disks were attached to the controller. I used the tw_cli which was available for downloading on the 3ware homepage in the past. Recently I found the binaries here:

https://www.thomas-krenn.com/de/download/hide_component.1/frame.only_content/hide_category.1/hide_product.1/product.2924.html

I downloaded the tw_cli package, extracted it, executed it and then to get an overview I did:

//server> show 

Ctl	Model 		Ports	Drives	Units	NotOpt	RRate	VRate	BBU ------------------------------------------------------------------------ 
c8 	9500S-4LP 	4 	3 	1 	1 	4 	4 	-

I used the “show” command to get the basic controller information.

Then, the next command showed detailed information about the controller state.

//server> info c8 u0 
Unit 	UnitType Status 	%Cmpl 	Port 	Stripe 	Size(GB) Blocks 
-------------------------------------------------------------------------
u0 	RAID-5 	DEGRADED 	- 	- 	64K 	745.037 1562456064 
u0-0 	DISK 	DEGRADED 	- 	p3 	- 	372.519 781228032 
u0-1 	DISK 	OK 		- 	p2 	- 	372.519 781228032 
u0-2 	DISK 	OK 		- 	p0 	- 	372.519 781228032

I saw that it was degraded because of a failed disk.

The Fix

I had to stop all the processes which were executed on the RAID 5 volume. Then I had to umount the RAID 5 volume from the file system. The next step was to remove the degraded disk on the port 3 with this command:

//server>maint remove c8 p3

Then I shut the server down. I saw on the controller hardware on which SATA Port the p3 Disk was connected. I replaced the disk with a newer, bigger one. The cache of the new disk was bigger than on the old one. That is important!

//server>maint rescan c8

This was the next important step – “rescan” for the newly installed disk. And then to start recreation of the RAID I finally did:

//server>maint rebuild c8 u0 p3

After a few hours i got this:

//server> info c8 u0 
Unit 	UnitType 	Status 	%Cmpl 	Port 	Stripe 	Size(GB) Blocks ----------------------------------------------------------------------- 
u0 	RAID-5 		REBUILDING 42 	- 	64K 	745.037	1562456064 
u0-0 	DISK 		DEGRADED 	- p3 	- 	372.519	781228032 
u0-1 	DISK 		OK 		- p2 	- 	372.519	781228032
u0-2 	DISK 		OK 		- p0 	- 	372.519	781228032

The rebuild took only a few hours.

If the Rebuild doesn’t start and you get an error message, it could be a good alternative to start the rebuild inside the controller BIOS.

Finally the server was as good as it was before 🙂