Anyone Familiar with Home Networks/File Servers?

lumberjack

lumberjack

Young man on the go
Joined
Mar 6, 2005
Messages
9,752
Location
Mississippi
It's not a current priority, but it's not too far down the line assuming work keeps plodding along.

What I am trying to accomplish:
Highly expandable (capacity) file server. 10-15TB starting out with ideally no limit.
Some variety of disk failure protection (parity drive, RAID, etc)
Wireless connectivity, mainly for automatically backing up devices (phones, iPads, laptops, etc). In the future this could also serve as a media server
Wired connection for working with stored video. USB 2 need not apply.. think USB 3 or Thunderbolt
Serve as the hub for business stuff, accessible from the iPhone/iPad/Laptop from abroad via the internet (VPN style) so office work I do in the field is transmitted back and it cuts down on time in the actual office. Example, customer calls, I grab the iPad/laptop and input their information and the bid time. This generates an entry in the customer database and schedules the bid on my calendar, setting reminders automatically. I bid the job at the appointed time, input the data on site, a quote/contract is generated for email or printing, even if it's a handshake job. On the acceptance of the bid, the job goes to a work list, and on completion the invoice is generated with the option to print it and and the envelope or email it.


One way I'm thinking of going about it:
High spec iMac. I can do business data entry, movie editing (faster with the direct connection), design work, etc on the computer. The computer also handles the "network" side of the equation, both VPN and home based. Basically a fully functional, high powered (consumer level) workstation that doubles as a server.
Drobo (or equivalent) stand alone direct connect file servers. Independently expandable, connectable via Thunderbolt (6 devices per iMac Thunderbolt port). There are downsides to Drobo itself, but I want easy expansion of the array. The iMac could array multiple Drobos into a single drive (from the user's perspective) or other configurations as required.


I could do the same thing with a Mac Mini, but I lose the high power of the iMac. I could build an UnRaid (or similar) file server, but I lose the fast connection options (ethernet only AFAIK) which would really suck compiling large raw video files for editing into a video on my laptop or iMac (if I get one).


Is this anyone's forte?
 
  • Thread Starter Thread Starter
  • #5
gf beranek said:
What do you need all that for just to do tree work?
Click to expand...

I don't do just tree work. I do tree work, site work, and growing into mulching and mowing I hope. I'm also a dealer for Valley Tool in California as well as Branch Manager Attachments. I'm a dealer rep for Top Notch equipment and I am trying to develop and grow a YouTube channel to support and reflect all that. With the general marketing as well the YouTube channel, I take tens of thousands of pictures a year as well as quite a few TB of video. From the raw footage I edit videos to publish on YouTube or Facebook. I archive pictures/video/messages/emails for quite some time, my last video I published was a quick before/after from a job we did in the summer of 2012. I'm a bit of a digital pack rat in that regard. I need to consolidate and organize, so I can use it more easily. I have an idea for another movie at the moment, but I have to go through 4 external hard drives to compile the raw footage for the video.

Business wise, I need to get on top of the "office" portion of the business and streamline it so I keep up with it. That's just good business and the lack of it is a stumbling block for growth. I also need to advance my advertising stance. It's horribly lax compared to the growth I need to get where I want to be.

Personally, I want to have an automated backup system from the peripheral devices so I don't have to fret when something gets lost, stolen, destroyed, or damaged. After a wedding last year I downloaded the cameras to my computer. The next morning I went to boot the computer and I had the Mac version of a fatal error. Complete wipe and reinstall of the OS. Luckily I travel with a backup computer and I was able to restore to the moment after I downloaded the camera. I also had a copy of the media on the backup drive as well.

Soooo, long story short, I'm trying to catch up and future proof for a bit.
 
I just got home and read it quick, I can definitely give you some suggestions on direction with storage specifically. I'll try to post up after the kids go to bed or worst case tomorrow from work. :thumbup:
 
I still don't have sufficient hard-drive space, Leon. I may have to follow suit with Carl on that.

I recall my first 18 gig drive, back in 03, was near a thousand bugs, and it was only enough to hold an hours worth of digital video. Storage has really come down in the years since. Terabytes for the same price today. and that is good.
 
I've read a bit about NAS...network attached storage. Apparently an efficient way to be able to expand storage as you grow. Often used in medical imaging for storing large patient image files.
 
Carl so far you seem like you're on the right track.

If I were going to do something similar at my own place, I'd split things up. 1 resource heavy machine, and one NAS solution.

I'd have a main server that was CPU/RAM heavy, say at least a dual 8 core processors, and at least 64g of ram. Personally I'd try to get a 10gb network card even though it'd be fairly expensive. On this machine, I'd run some type of Hypervisor (vmware/xen/etc). This way you can run any operating system you want, and have multiples of each. You can allocate CPU and RAM as necessary and even back up entire OS images. That way if you have a virus, or something goes wrong, you can just blow away the VM and start from scratch.

You'd obviously need drive space allocated for those VM's. That's the reason for the 10g network card. If you get a NAS solution that does iSCSI, you can just setup a RAID of LUNs for each VM. The disks would need to be fast: SATA III (or SAS) with at least 10k rpm.

A setup like that will give cost you a bit but give you flexibility for quite a while. By the time you outgrew a setup like that you'd likely be big enough to hire someone to manage the IT side of your business.

On a smaller scale, with slower storage, one of those drobo setups looks pretty good. You could get larger (but slower) storage for less money. Say 4 4TB 10k rpm disks setup in a RAID 5. That'd give you 12gb of faster (striped) space with some redundancy from the parity. With that setup you'd probably want to run some smaller SSD's on the server the drobo is attached to. I'd do 2 80gb SSD's in RAID1. You could still run virtual machines on that computer as well, but obviously wouldn't be able to put the OS installs on an iSCSI LUN.

In terms of thunderbolt vs usb 3.0, I don't have any experience with either. I'd go with whichever you think will be supported longer. I know Intel developed thunderbolt and it's heavily backed by Apple, but USB has been around a while and obviously isn't going anywhere. The bottleneck in IO with either would still likely be the hard drives on the other end.

If I missed anything (I'm sure I must have) let me know. I can clarify anything too. Some of the terms sound complicated but they're just fancy acronyms for simple concepts. I can break anything down more for anyone who has questions so please don't hesitate to ask what something means in layman's terms. I think this stuff is neat and like it when people are interested in it.

Most of the storage stuff we do at work is more Enterprise level stuff. Our NAS stuff is about 1.5 petabytes spread across about a dozen redundant nodes that all talk over 40gbps infiband and multiple 10g connections. It's pretty cool stuff.
 
Adam_P said:
Carl so far you seem like you're on the right track.

If I were going to do something similar at my own place, I'd split things up. 1 resource heavy machine, and one NAS solution.

I'd have a main server that was CPU/RAM heavy, say at least a dual 8 core processors, and at least 64g of ram. Personally I'd try to get a 10gb network card even though it'd be fairly expensive. On this machine, I'd run some type of Hypervisor (vmware/xen/etc). This way you can run any operating system you want, and have multiples of each. You can allocate CPU and RAM as necessary and even back up entire OS images. That way if you have a virus, or something goes wrong, you can just blow away the VM and start from scratch.

You'd obviously need drive space allocated for those VM's. That's the reason for the 10g network card. If you get a NAS solution that does iSCSI, you can just setup a RAID of LUNs for each VM. The disks would need to be fast: SATA III (or SAS) with at least 10k rpm.

A setup like that will give cost you a bit but give you flexibility for quite a while. By the time you outgrew a setup like that you'd likely be big enough to hire someone to manage the IT side of your business.

On a smaller scale, with slower storage, one of those drobo setups looks pretty good. You could get larger (but slower) storage for less money. Say 4 4TB 10k rpm disks setup in a RAID 5. That'd give you 12gb of faster (striped) space with some redundancy from the parity. With that setup you'd probably want to run some smaller SSD's on the server the drobo is attached to. I'd do 2 80gb SSD's in RAID1. You could still run virtual machines on that computer as well, but obviously wouldn't be able to put the OS installs on an iSCSI LUN.

In terms of thunderbolt vs usb 3.0, I don't have any experience with either. I'd go with whichever you think will be supported longer. I know Intel developed thunderbolt and it's heavily backed by Apple, but USB has been around a while and obviously isn't going anywhere. The bottleneck in IO with either would still likely be the hard drives on the other end.

If I missed anything (I'm sure I must have) let me know. I can clarify anything too. Some of the terms sound complicated but they're just fancy acronyms for simple concepts. I can break anything down more for anyone who has questions so please don't hesitate to ask what something means in layman's terms. I think this stuff is neat and like it when people are interested in it.

Most of the storage stuff we do at work is more Enterprise level stuff. Our NAS stuff is about 1.5 petabytes spread across about a dozen redundant nodes that all talk over 40gbps infiband and multiple 10g connections. It's pretty cool stuff.
Click to expand...

Wow!
 
  • Thread Starter Thread Starter
  • #24
Adam_P said:
Carl so far you seem like you're on the right track.

If I were going to do something similar at my own place, I'd split things up. 1 resource heavy machine, and one NAS solution.

I'd have a main server that was CPU/RAM heavy, say at least a dual 8 core processors, and at least 64g of ram. Personally I'd try to get a 10gb network card even though it'd be fairly expensive. On this machine, I'd run some type of Hypervisor (vmware/xen/etc). This way you can run any operating system you want, and have multiples of each. You can allocate CPU and RAM as necessary and even back up entire OS images. That way if you have a virus, or something goes wrong, you can just blow away the VM and start from scratch.

You'd obviously need drive space allocated for those VM's. That's the reason for the 10g network card. If you get a NAS solution that does iSCSI, you can just setup a RAID of LUNs for each VM. The disks would need to be fast: SATA III (or SAS) with at least 10k rpm.

A setup like that will give cost you a bit but give you flexibility for quite a while. By the time you outgrew a setup like that you'd likely be big enough to hire someone to manage the IT side of your business.

On a smaller scale, with slower storage, one of those drobo setups looks pretty good. You could get larger (but slower) storage for less money. Say 4 4TB 10k rpm disks setup in a RAID 5. That'd give you 12gb of faster (striped) space with some redundancy from the parity. With that setup you'd probably want to run some smaller SSD's on the server the drobo is attached to. I'd do 2 80gb SSD's in RAID1. You could still run virtual machines on that computer as well, but obviously wouldn't be able to put the OS installs on an iSCSI LUN.

In terms of thunderbolt vs usb 3.0, I don't have any experience with either. I'd go with whichever you think will be supported longer. I know Intel developed thunderbolt and it's heavily backed by Apple, but USB has been around a while and obviously isn't going anywhere. The bottleneck in IO with either would still likely be the hard drives on the other end.

If I missed anything (I'm sure I must have) let me know. I can clarify anything too. Some of the terms sound complicated but they're just fancy acronyms for simple concepts. I can break anything down more for anyone who has questions so please don't hesitate to ask what something means in layman's terms. I think this stuff is neat and like it when people are interested in it.

Most of the storage stuff we do at work is more Enterprise level stuff. Our NAS stuff is about 1.5 petabytes spread across about a dozen redundant nodes that all talk over 40gbps infiband and multiple 10g connections. It's pretty cool stuff.
Click to expand...


Thunderbolt offers dual 10Gb channels. Thunderbolt 2 offers allows combining the channels for 20Gb one way. That's plenty. USB 3 is up to 5Gb, 3.1 is 10Gb. Either is fine as Mac supports both. Thunderbolt has some advantages in that it also carries video output, so, for example, you could connect a monitor to the computer with a Thunderbolt cable, then 5-6 additional devices, be they monitors or hard drives.


RAID arrays aren't expandable while maintaining the data in the array. Then again, striping would make that hard to scale. Drobo (or similar) is inviting because when it becomes time to expand I can buy another, then combine the volumes on the iMac if desired. The scary thing to me about RAID is a controller failure.

At the moment I don't anticipate much in the way of file serving with the exception of backing up the miscellaneous devices in and around the house. If I digitize the DVD/BlueRays, it's possible it would act as a media server.


The iMac I've currently specced is a 3.5GHz i7, 1TB SSD, 32GB ram, NVIDIA GeForce GTX 780M 4GB GDDR5 with a second monitor. That would be fine for video editing. If the project was big enough I could get my laptop (2.7 GHz i7, 16GB RAM, 768GB SSD) involved to on the initial encoding and analyzing.

I have a 6TB Lacie drive set up as RAID 0 (Thunderbolt) for my fast drive. That actually may be enough, now that I think of it. If so, an unRAID setup intrigues me. You could have 23 data drives, one cache drive, one parity drive. The attraction is you can add drives as you go, you're not locked in to having to set up the array at the beginning. If one drive fails, the parity takes care of that, if another drive drops while rebuilding, rebuild fails (obviously). You only lose the data on those two discs as there is no striping. The unRAID server would serve as an archive, the Lacie (or equivalent(s)) could serve as a point to pull from the archive, then start processing. The biggest downside is it's not plug in and play, I'd have to build the server. The other downside is the slower transfer speed as compared to RAIDs that stripe, but if it's just an archive that's not a big deal. I could set up a transfer for the footage I need of the current project before I go to bed. Once its on the RAID 0 drive(s), speed isn't an issue.


I do want to stick with Mac as the whole purpose of the network is to streamline the business's office work. The software I use will have to be Mac based, perhaps Quick Books. iPhone and iPad apps would be a relatively high priority.

If the server side becomes a burden on the iMac, I could get a Mac Mini to take over the server side of things, but that's down the road and an easy expansion.
 
The specs on the iMac sound pretty sweet.

So in regards the the RAID stuff not being expandable while maintain data, do you just mean that you can't access data while it's restriping? If you don't want to restripe you can always create a new array when you expand, the same as adding a drobo to expand.

In terms of the failure point with the controller, that's definitely valid. The higher end controllers have excellent quality and absolutely worst case you could always buy another. Like I said, I don't have any experience with the drobo units, but I'd imagine the controllers are internal, and if it failed you'd be mailing the unit back to them to repair. Mega downtime. I'd worry that the units are going to be a fad and in a few years won't be as supported. When you go with a controller and standard disk setup you end up with some more flexibility long term. You could also look into RAID10 instead of RAID5, or even consider RAID1 for critical data and a RAID0 for fast scratch space (like editing videos before you commit the final cut to the RAID1).

I have 0 familiarity with the unRAID system, but if you're doing mostly reading rather than writing it seems ok. If you do try that let us know how it works out for you. I definitely wouldn't mind setting up a media server at home someday and that would be a good option.
 
You must log in or register to reply here.
Back
Top