building on your own a large data storage ...

On Sat, 7 Jul 2007 19:15:12 +0000 (UTC), Andre Majorel
<cheney.TakeThisOut@halliburton.com> wrote:


>> Should I got for ATA or SATA drives and why?
>
>SATA is better because 1) it doesn't have the master/slave
>issues of IDE, i.e. if you have 4 SATA ports on your
>motherboard, you *can* do a 4-disk RAID-5 array and 2)
>motherboards with 8 SATA ports are easy to find.

Mostly irrelevant, ether what he has can do the rail level
required, or he can buy what's needed. Also, it could be a
bad idea to rely on a RAID5 dependant on a particular
motherboard chipset, instead of an add-on card (which is far
less likely to fail)>

It has very little to do with SATA vs PATA. Certainly
master/slave assignments are no issue at all, being able to
jumper a drive is fairly trivial in the grand scheme of
things.
 >> Stay informed about: building on your own a large data storage ... 

On 2007-07-08, kony <spam.DeleteThis@spam.com> wrote:
> On Sat, 7 Jul 2007 19:15:12 +0000 (UTC), Andre Majorel
><cheney.DeleteThis@halliburton.com> wrote:
>
>>> Should I got for ATA or SATA drives and why?
>>
>>SATA is better because 1) it doesn't have the master/slave
>>issues of IDE, i.e. if you have 4 SATA ports on your
>>motherboard, you *can* do a 4-disk RAID-5 array and 2)
>>motherboards with 8 SATA ports are easy to find.
>
> Mostly irrelevant, ether what he has can do the rail level
> required, or he can buy what's needed.

But at what price ? I haven't found any cheap 4-IDE-port cards.

> Also, it could be a bad idea to rely on a RAID5 dependant on a
> particular motherboard chipset, instead of an add-on card
> (which is far less likely to fail)>

Or not as easy to replace, anyway. But your warning applies to
hardware RAID-5 and we are discussing _software_ RAID-5. With
software RAID, whatever RAID-5 features the chipset may have
don't matter because we simply don't use them.

> It has very little to do with SATA vs PATA. Certainly
> master/slave assignments are no issue at all, being able to
> jumper a drive is fairly trivial in the grand scheme of
> things.

I was referring to another master/slave issue : that IDE devices
cannot share a port without fighting.

If you put 2 devices on the same IDE port and try to do I/O with
both, the total transfer rate is less than the transfer rate you
would get when using only one device at a time. Not a little
less ; reportedly so much less that it's unacceptable.

--
André Majorel <URL:http://www.teaser.fr/~amajorel/>
(Counterfeit: qysop.DeleteThis@curb.com icecaloh.DeleteThis@coupon.com)
"Duty, honor, country" -- Douglas MacArthur
"Travail, famille, patrie" -- Philippe Pétain
 >> Stay informed about: building on your own a large data storage ... 

On 2007-07-07, Arno Wagner <me.DeleteThis@privacy.net> wrote:
> In comp.sys.ibm.pc.hardware.storage Andre Majorel
> <cheney.DeleteThis@halliburton.com> wrote:
>> ["Followup-To:" header set to comp.sys.ibm.pc.hardware.storage.]
>> On 2007-07-04, lbrtchx.DeleteThis@hotmail.com <lbrtchx.DeleteThis@hotmail.com> wrote:
>>
>>> Any suggestions of the type of boxes/racks I should use?
>
>> 3ware make 3-disks-in-2-5.25"-spaces trays. They are expensive
>> and the fans they use die after about a year. When a fan goes
>> bad, the tray helpfully warns you about it by beeping loudly and
>> constantly. The fans are not the easiest to find (60 mm or some
>> such). Be prepared to hear a lot of beeping.
>
>> I made my own trays. It was a lot of work and they look ugly but
>> it was cheap and they do the job. The ventilation is superior to
>> commercial trays (120 mm fan than moves a lot of air quietly and
>> reliably).
>
> There is a 4-in-3 mounting with 120mm fan made by Coolermaster.
> Quite cheap, about 22 Swiss Franks, which is something like
> 15 USD/EUR. Ugly, but cools well and perfect for making disk-packs
> that fit in standard PC cases. I have one of these in a server
> running something like 3 years now and the fan is still fine.
> It does fit in 3 standard 5 1/4" bays if you remove the plastic
> side-rails. They are just screwed on. Mounting holes also match
> any standard case. The fan should direcly blow in outside air.
>
> Here is a link (the coolermaster website is broken both
> in Opera and Firefox....). Sorry, it is German:
>
> http://www.pcp.ch/Cooler-Master-4-in-3-Device-Modul-1a12170083.htm

It's the Cooler Master STB-3T4-E1 :

http://www1.coolermaster.com/index.php?url_place=product&p_serial=STB-3T4-E1

There's also the STB-3T4-E3-GP :

http://www1.coolermaster.com/index.php?url_place=product&p_serial=STB-3T4-E3-GP

Excellent. I paid that much for the fan alone. I wish I had
known about it one year ago. Well, maybe not.

My DIY ones hold 5 disks instead of 4. You can pop out the fan
and remove/add disks from the front. The disks are in vertical
position which will provide a little bit of natural convection
if the fan jams.

On the plus side, the STB-3T4-E3-GP is shock mounted. And it
looks like you can install it without flattening the tabs in the
PC case.

How is laid out the STB-3T4-E1 ? Do you install the disks from
the back as with the STB-3T4-E3-GP ? Are the disks secured, or
can they fall out the back if you tilt the PC backwards ?

--
André Majorel <URL:http://www.teaser.fr/~amajorel/>
(Counterfeit: jul.DeleteThis@tailgate.com sobocam.DeleteThis@castigate.com)
"Duty, honor, country" -- Douglas MacArthur
"Travail, famille, patrie" -- Philippe Pétain
 >> Stay informed about: building on your own a large data storage ... 

Andre Majorel <cheney.DeleteThis@halliburton.com> wrote in
news:slrnf91a1n.2pn.cheney@atc5.vermine.org:

[snip]

> But at what price ? I haven't found any cheap 4-IDE-port cards.

But there are cheap 2 port cards, so a motherboard with 3 pci slots could
have 8 drives, all as master, or 16 drives if you use master/slave.

A dedicated storage box wouldn't need the slots for any other cards.
 >> Stay informed about: building on your own a large data storage ... 

Previously bealoid <signup.RemoveThis@bealoid.co.uk> wrote:
> Andre Majorel <cheney.RemoveThis@halliburton.com> wrote in
> news:slrnf91a1n.2pn.cheney@atc5.vermine.org:

> [snip]

>> But at what price ? I haven't found any cheap 4-IDE-port cards.

> But there are cheap 2 port cards, so a motherboard with 3 pci slots could
> have 8 drives, all as master, or 16 drives if you use master/slave.

> A dedicated storage box wouldn't need the slots for any other cards.

You still can get a 4-port SATA controller for the price of a
2-port ATA controller and (at least here) SATA drives are not
more expensive. Plus the cableing is far superiour, which becomes
an issue with several drives. Also note that SATA cables can be
up to 1m long, which may actually be needed.

Arno
 >> Stay informed about: building on your own a large data storage ... 

Previously Andre Majorel <cheney DeleteThis @halliburton.com> wrote:
> On 2007-07-07, Arno Wagner <me DeleteThis @privacy.net> wrote:
>> In comp.sys.ibm.pc.hardware.storage Andre Majorel
>> <cheney DeleteThis @halliburton.com> wrote:
>>> ["Followup-To:" header set to comp.sys.ibm.pc.hardware.storage.]
>>> On 2007-07-04, lbrtchx DeleteThis @hotmail.com <lbrtchx DeleteThis @hotmail.com> wrote:
>>>
>>>> Any suggestions of the type of boxes/racks I should use?
>>
>>> 3ware make 3-disks-in-2-5.25"-spaces trays. They are expensive
>>> and the fans they use die after about a year. When a fan goes
>>> bad, the tray helpfully warns you about it by beeping loudly and
>>> constantly. The fans are not the easiest to find (60 mm or some
>>> such). Be prepared to hear a lot of beeping.
>>
>>> I made my own trays. It was a lot of work and they look ugly but
>>> it was cheap and they do the job. The ventilation is superior to
>>> commercial trays (120 mm fan than moves a lot of air quietly and
>>> reliably).
>>
>> There is a 4-in-3 mounting with 120mm fan made by Coolermaster.
>> Quite cheap, about 22 Swiss Franks, which is something like
>> 15 USD/EUR. Ugly, but cools well and perfect for making disk-packs
>> that fit in standard PC cases. I have one of these in a server
>> running something like 3 years now and the fan is still fine.
>> It does fit in 3 standard 5 1/4" bays if you remove the plastic
>> side-rails. They are just screwed on. Mounting holes also match
>> any standard case. The fan should direcly blow in outside air.
>>
>> Here is a link (the coolermaster website is broken both
>> in Opera and Firefox....). Sorry, it is German:
>>
>> http://www.pcp.ch/Cooler-Master-4-in-3-Device-Modul-1a12170083.htm

> It's the Cooler Master STB-3T4-E1 :

> http://www1.coolermaster.com/index.php?url_place=product&p_serial=STB-3T4-E1

> There's also the STB-3T4-E3-GP :

> http://www1.coolermaster.com/index.php?url_place=product&p_serial=STB-3T4-E3-GP

> Excellent. I paid that much for the fan alone. I wish I had
> known about it one year ago. Well, maybe not.

> My DIY ones hold 5 disks instead of 4. You can pop out the fan
> and remove/add disks from the front. The disks are in vertical
> position which will provide a little bit of natural convection
> if the fan jams.

And building something yourself successfully is allways
a source of satisfaction!

> On the plus side, the STB-3T4-E3-GP is shock mounted. And it
> looks like you can install it without flattening the tabs in the
> PC case.

Only if you have three consecutive slots without them.
I did not have to flatten any, since it seems server cases
do not have the tabs anyways.

> How is laid out the STB-3T4-E1 ? Do you install the disks from
> the back as with the STB-3T4-E3-GP ? Are the disks secured, or
> can they fall out the back if you tilt the PC backwards ?

Actually both seem to be the same, i.e. disks are put in from the
backside and secured with 4 screws each. For this you need to remove
the whole units, remove the sides and then you have a 3.5" wide drive
cage in your hands with the fan at the front. The STB-3T4-E3-GP merely
seems to have a front-cover added.

Arno
 >> Stay informed about: building on your own a large data storage ... 

"Andre Majorel" <cheney RemoveThis @halliburton.com> wrote in message news:slrnf91a1n.2pn.cheney@atc5.vermine.org
> On 2007-07-08, kony <spam RemoveThis @spam.com> wrote:
> > On Sat, 7 Jul 2007 19:15:12 +0000 (UTC), Andre Majorel <cheney RemoveThis @halliburton.com> wrote:
> >
> > > > Should I got for ATA or SATA drives and why?
> > >
> > > SATA is better because 1) it doesn't have the master/slave
> > > issues of IDE, i.e. if you have 4 SATA ports on your
> > > motherboard, you *can* do a 4-disk RAID-5 array and 2)
> > > motherboards with 8 SATA ports are easy to find.
> >
> > Mostly irrelevant, wether what he has can do the raid level
> > required, or he can buy what's needed.
>
> But at what price ? I haven't found any cheap 4-IDE-port cards.
>
> > Also, it could be a bad idea to rely on a RAID5 dependant on a
> > particular motherboard chipset, instead of an add-on card
> > (which is far less likely to fail)>
>
> Or not as easy to replace, anyway. But your warning applies to
> hardware RAID-5 and we are discussing _software_ RAID-5.
> With software RAID,

> whatever RAID-5 features the chipset may have don't matter

> because we simply don't use them.

Then it better not have them if they can't be shut off.

>
> > It has very little to do with SATA vs PATA. Certainly
> > master/slave assignments are no issue at all, being able to
> > jumper a drive is fairly trivial in the grand scheme of things.
>
> I was referring to another master/slave issue : that IDE devices
> cannot share a port without fighting.

They are not fighting. You simply can't send a command to both
the drives and then let them fight for who's first with the data.
That would allow the late drive to transfer from cache, without the
extra latency, doubling singledrive speed.
Instead you send a command to one drive, wait for it to finish and
then send a command to the other one, and wait for that to finish.
You only get single drive speed and have double the average latency
as opposed to a single drive. On the other hand you get twice the
transfer length to offset the double latency.

>
> If you put 2 devices on the same IDE port and try to do I/O with both,

> the total transfer rate is less than the transfer rate you would get

That is not supported by theory. *Total* transferrate should be
the same: twice as much data at twice as much time.
Single (one) drive transfer rate though will obviously be cut in half,
but that's not what you said.

> when using only one device at a time.

As if there is any other way.

> Not a little less ; reportedly so much less that it's unacceptable.

Yes, if your transfer length is too small latency has a devestating
effect on the average transfer rate. The maximum transfer for a
single command (256kB) only takes ~3.5ms to transfer at 70MB/s.
Set against the total time of say 15ms that will slice your average
random read speed to 16MB/s, even for a single drive.

If you fooloshly alternate drive requests on a one command basis
then you will convert an initially sequential read in a random read.
Maybe that's the problem, though even then read ahead caching on
the drive should fix that.
 >> Stay informed about: building on your own a large data storage ... 

On 2007-07-07, Arno Wagner <me RemoveThis @privacy.net> wrote:
> In comp.sys.ibm.pc.hardware.storage Andre Majorel
> <cheney RemoveThis @halliburton.com> wrote:
>
>> - understanding the trade-offs (partioning an array of disks vs.
>> making an array of partitions, using LVM or not, optimum
>> granularity) and
>
> I strongly suggest using partitions of type 0xfd, because then the
> kernel will auto-assemble the array on system start. For complete
> disks you need some start-script or other, which also means you cannot
> have the root-partition on the array. In additions these start-scripts
> sometimes are unreliable.

That's what I did too. Partition all the disks identically then
create array 1 from /dev/sd{a,b,c,d}1, array 2 from
/dev/sd{a,b,c,d}2, etc. The root and swap are on the first two
arrays, which are RAID-1 so as to eliminate the need for initrd.
The data is on RAID-5 arrays.

Another reason for preferring arrays of partitions to partitions
of arrays is that, when mdadm detects an error anywhere in any
component device, the whole array is marked as degraded. It
takes a long time to resync a 750 GB array.

Also, when you add a disk to an array of disks, you are
dependent on mdadm's --grow feature. Whereas with arrays of
partitions, you have the option of copying the data from array 5
to array 6, stopping and re-creating array 5 and copying the
data from array 6 back to array 5. And so on until all you have
grown all the arrays. Primitive and slow but robust.

>> One very important thing about RAID that too many people
>> overlook : don't make an N-disk array from N disks of the same
>> make and model bought the same day. Our sysadmin at work did and
>> both drives on a RAID-1 array failed within days of each
>> other...
>
> Hehe. For high-reliability applications that is very good advice.
> You may even want a 3- or 4- way RAID 1 (which Linux can do)
> for these with 3 or 4 different disks.

Definitely. People tend to assume they need identical disks.
They don't. I have a 4-disk RAID-5 array made of disks from 4
different manufacturers. There might be a cost in terms of
performance but when you get 125 MB/s on reads, you can't
complain too much. <g>

# time dd bs=10M count=1000 if=/dev/md105 of=/dev/null
1000+0 records in
1000+0 records out

real 1m18.976s
user 0m0.000s
sys 0m20.897s

Disk sizes are standardised nowadays. For Hitachi, Samsung,
Seagate and Western Digital, 200 GB = 24321 x 255 x 63 , 250 GB =
30401 x 255 x 63, 500 GB = 60801 x 255 x 63... Only Maxtor are
a little bit bigger. Partition them all to the figures above and
be done with it.

> For ordinary use, I would say it is enough to have a cold
> spare ready and rebuild the array immediately. You can also
> take tha array down until you have the spare. But don't
> continue to use it in degraded state.

The hapless sysadmin said he didn't get mdadm's alert email. No
idea whether that is a bug in mdadm or an error on his part.

--
André Majorel <URL:http://www.teaser.fr/~amajorel/>
(Counterfeit: wuwas RemoveThis @dateline.com xodeduzud RemoveThis @indolent.com)
"Duty, honor, country" -- Douglas MacArthur
"Travail, famille, patrie" -- Philippe Pétain
 >> Stay informed about: building on your own a large data storage ...