Disclaimer: I am not responsible for any loss of data, security breaches, fraud, omissions, errors, misconfigurations, service interruptions, slowdowns, freezes, breakdowns, or diminishments of any kind resulting from, caused by or attributed to the following or any content on this site.
Please backup before you proceed, and don't run any command without knowing what it is for.
And PLEEAAASE use a UPS!
You also need to know that the logical name (/dev/sdX) are NOT always the same. It can quickly change doing a reboot! Always check their serial numbers before you format them.
apt-get install mdadm parted lshw -y
RAID | Min. devices | Max. device failure | Space available | Speed gain* |
---|---|---|---|---|
RAID0 | 2 | 0 | devicesize*devices | (devices)x read & write |
RAID1 | 2 | devices-1 | devicesize | (devices)x read |
RAID5 | 3 | 1 | devicesize*(devices-1) | (devices-1)x read |
RAID6 | 4 | 2 | devicesize*(devices-2) | (devices-2)x read |
RAID10 | 4 && (devices % 2)==0 | devices/2 | devicesize*(device/2) | (device)x read & (device/2) write |
* I found it on the internet. A few sites disagree on it, but this is the most common calculation.
Filesystem | Max. file size | Max. volume size |
---|---|---|
ext2 | 2TB | 32TB |
ext3 | 2TB | 32TB |
ext4 | 16TB | 16TB (32bit/DEFAULT) / 1EB ('-O 64bit') |
NTFS | 256TB | 256TB |
XFS | 8 EB | 16 EB |
Quick FAQ:
Q: Durrr, just use hardware RAID!?
A: NO! When the RAID controller dies after 20 years or whatever, just imagine the price of getting an identical one then!!
Don't be like me and changing the parition table on the wrong drives.
lshw -class disk
In this (tested) example, we use 3x6TB disks to create a RAID5 with.
PLEASE consider RAID6 if you have 4 or more disks!!
And RAID is NOT the same as backup! It will help to some degree, but you NEED a backup! (Hint: Crashplan, Amazon Glacier, etc.)
Since the discs are larger than 2TB, we need to switch the partition table to GPT instead of MBR.
WARNING: THIS WILL DESTROY ALL CONTENT ON THE CHOSEN DISKS!!
lshw -class disk && lsblk #Be sure you choose the correct drives! This can seriously fuck everything up if don't.
parted --script /dev/sda mklabel gpt
parted --script /dev/sdb mklabel gpt
parted --script /dev/sdd mklabel gpt
parted -a optimal /dev/sda mkpart primary 0% 100%
parted -a optimal /dev/sdb mkpart primary 0% 100%
parted -a optimal /dev/sdd mkpart primary 0% 100%
mdadm --create /dev/md0 --level=5 --raid-devices=3 /dev/sda1 /dev/sdb1 /dev/sdd1 # Create RAID5 with these three disks.
cat /proc/mdstat
mkfs -t ext4 -E lazy_itable_init=1 -O 64bit,sparse_super,filetype,resize_inode,dir_index,ext_attr,has_journal,extent,huge_file,flex_bg,uninit_bg,dir_nlink,extra_isize /dev/md0
mkdir /mnt/md0
mount -t ext4 /dev/md0 /mnt/md0
chown -R mathias:mathias /mnt/md0
echo "/dev/md0 /mnt/md0 ext4 defaults 0 0" >> /etc/fstab
We have quickly realised, that your porn collection needs more than the 11TB you got from 3x6TB!
Now it's time to expand it with another 6TB disk.
WARNING: THIS WILL DESTROY ALL CONTENT ON THE CHOSEN DISK!!
lshw -class disk && lsblk #FIND AND BE 100% SURE YOU GET THE CORRECT /dev/sXX!!!!!! CHECK USING DISK SERIAL NUMBERS!
parted --script /dev/sde mklabel gpt
parted -a optimal /dev/sde mkpart primary 0% 100%
mdadm --add /dev/md0 /dev/sde1
mdadm --detail /dev/md0 # You can now see, a spare has been added.
mdadm --grow /dev/md0 --raid-devices=4 # Replace 4 with the (new) total amount of discs..
watch -n1 'mdadm --detail /dev/md0 && echo "\n" && cat /proc/mdstat'
# I recommend waiting until it's done reshaping..
systemctl stop smbd
umount /dev/md0
e2fsck -f /dev/md0 -C 0 # Check for errors, if you're in a hurry this is not required.
resize2fs /dev/md0 -p
e2fsck -f /dev/md0 -C 0 # Also not needed.. Feel free to skip this step unless you're paranoid..
mount -t ext4 /dev/md0 /mnt/md0
systemctl start smbd
mdadm --detail /dev/md0 && df -h /mnt/md0
We have quickly realised, how important your porn collection is. And we have therefore bought another 6TB disk (“device”) to make 2 disk failure possible.
WARNING: THIS WILL DESTROY ALL CONTENT ON THE CHOSEN DISK!!
lshw -class disk && lsblk #FIND AND BE 100% SURE YOU GET THE CORRECT /dev/sXX!!!!!! CHECK USING DISK SERIAL NUMBERS!
parted --script /dev/sdf mklabel gpt
parted -a optimal /dev/sdf mkpart primary 0% 100%
mdadm --add /dev/md0 /dev/sdf1
mdadm --detail /dev/md0 # You can now see, a spare has been added.
mdadm --grow /dev/md0 --level=6 --raid-devices=5 --backup-file=/root/mdadmbackup_md0_raid5
Our huge porn collections seems to be in danger, and we must therefor replace it as soon as possible with a new drive. So we can get the it all back to into an stable state again.
First, we remove the drive(s) from the RAID with this:
mdadm --detail /dev/md0 && echo "\n" && cat /proc/mdstat # Identify the broken drive, it's usually marked with a (F) for failed
lshw -class disk # Find the serial number for the broken drive, if it's still online. And remove it from the system.
mdadm /dev/md0 -r failed # Remove the failed drives, output will be like: mdadm: hot removed 8:65 from /dev/md0
mdadm /dev/md0 -r detached # OR run this, if the drive didn't fail, but was just detached.
Now it's time to add the new drive(s)
WARNING: THIS WILL DESTROY ALL CONTENT ON THE NEW CHOSEN DISK!!
lshw -class disk && lsblk #FIND AND BE 100% SURE YOU GET THE CORRECT /dev/sXX!!!!!! CHECK USING DISK SERIAL NUMBERS!
parted --script /dev/sdg mklabel gpt
parted -a optimal /dev/sdg mkpart primary 0% 100%
mdadm --add /dev/md0 /dev/sdg1
watch -n1 'mdadm --detail /dev/md0 && echo "\n" && cat /proc/mdstat' # Now just sit back, and watch it while it's recovering. This step is not required, but cool to look at.
Want to see some fancy stats for the rest of the day? Alrighty!
watch -n1 'mdadm --detail /dev/md0 && echo "\n" && cat /proc/mdstat'
If that “Failed Devices : 0” change to 1 or more, it's okay to panic.. BECAUSE YOU NEED TO PANIC AND GET A NEW DRIVE ASAP!!!! When you have the new drive, look at the section just above this one.
parted --script /dev/sdc mklabel gpt
parted --script /dev/sdf mklabel gpt
parted --script /dev/sdg mklabel gpt
parted -a optimal /dev/sdc mkpart primary 0% 100%
parted -a optimal /dev/sdf mkpart primary 0% 100%
parted -a optimal /dev/sdg mkpart primary 0% 100%
mdadm --create /dev/md1 --level=5 --raid-devices=3 /dev/sdc1 /dev/sdf1 /dev/sdg1
cat /proc/mdstat
cryptsetup -y -v luksFormat /dev/md1 # The encryption setup
cryptsetup luksOpen /dev/md1 md1luks # Run this after each boot
mkfs -t ext4 -E lazy_itable_init=1 -O 64bit,sparse_super,filetype,resize_inode,dir_index,ext_attr,has_journal,extent,huge_file,flex_bg,uninit_bg,dir_nlink,extra_isize /dev/mapper/md1luks # The bigger the longer, have patience.
mkdir /mnt/md1
mount -t ext4 /dev/mapper/md1luks /mnt/md1 # Run this after each boot
chown -R mathias:mathias /mnt/md1
cryptsetup -y -v luksFormat /dev/sdc # The encryption setup
cryptsetup luksOpen /dev/sdc porn # Run this after each boot
mkfs -t ext4 -E lazy_itable_init=1 -O 64bit,sparse_super,filetype,resize_inode,dir_index,ext_attr,has_journal,extent,huge_file,flex_bg,uninit_bg,dir_nlink,extra_isize /dev/mapper/porn
mkdir /mnt/pornmount
mount -t ext4 /dev/mapper/porn /mnt/pornmount # Run this after each boot
umount /mnt/pornmount
cryptsetup luksClose porn
cryptsetup -v status md1luks
This is the script I run, manually after boot. I've disabled samba, so I know when things are down, and to avoid mistakes.
set -e
cryptsetup luksOpen /dev/md1 md1luks
mount -t ext4 /dev/mapper/md1luks /mnt/md1
systemctl start smbd
apt-get install samba samba-common libcups2
mv /etc/samba/smb.conf /etc/samba/smb.conf.bak
nano /etc/samba/smb.conf
The smb.conf consists of multiple parts (all in the same file), first we need the [global] settings tag.
I have fine tuned this to be fast as possible! If you know any other tweak that makes it faster, let me know :)
[global]
workgroup = WORKGROUP
server string = Samba Server %v
netbios name = debian
security = user
map to guest = bad user
dns proxy = no
mangled names = no
socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=131072 SO_SNDBUF=131072
strict allocate = Yes
read raw = Yes
write raw = Yes
strict locking = No
min receivefile size = 16384
use sendfile = true
aio read size = 16384
aio write size = 16384
Then you can choose what you want next, based on what you want to do.
As long you don't add any other shares, that requires login. The client wont be asked to login with something.
mkdir /mnt/md0/anonymous
chmod -R 0755 /mnt/md0/anonymous
chown -R nobody:nogroup /mnt/md0/anonymous
[Anonymous]
path = /mnt/md0/anonymous
browsable =yes
writable = yes
guest ok = yes
read only = no
mkdir /mnt/md0/mathias
chmod -R 0700 /mnt/md0/mathias
chown -R mathias:mathias /mnt/md0/mathias
[mathias]
path = /mnt/md0/mathias
valid users = mathias
guest ok = no
writable = yes
browsable = yes
create mask = 0700
directory mask = 2700
mkdir /mnt/md0/entertainment
chmod -R 0770 /mnt/md0/entertainment
chown -R mediauser:smbgroup /mnt/md0/entertainment
[entertainment]
path = /mnt/md0/entertainment
valid users = @smbgroup
guest ok = no
writable = yes
browsable = yes
force user = mediauser
force group = smbgroup
First we to create a group
groupadd smbgroup
useradd mediauser -G smbgroup
usermod -a mathias -G smbgroup
Samba can't read the passwords set in Linux, and I also don't recommend having your top secret password floating over the network.
Luckily it's easy to add them using:
smbpasswd -a mathias
apt-get install nfs-kernel-server portmap
nano /etc/exports # Modify and insert the lines below
# /mnt/md0/nfs/esxi 10.20.40.0/24(rw,sync,no_root_squash,no_subtree_check)
# /mnt/md0/sharedfolder 10.20.30.40(rw,sync,no_root_squash,no_subtree_check)
exportfs -ra
systemctl restart nfs-kernel-server
apt-get install nfs-common
mkdir /mnt/localnfsfolder
nano /etc/fstab # Modify and insert the lines below - BUT BE CAREFUL, THIS FILE CAN BREAK YOUR SYSTEM!
# 10.20.30.123:/mnt/md0/sharedfolder /mnt/localnfsfolder nfs rw,async,hard,intr,noexec 0 0
mount -a # or: mount /mnt/localnfsfolder
Get the UID and GID from the folder you wish to mount: ls -n /mnt/md0/sharedfolder
Add the UID and GID to the registry first, so you don't have to restart the Client For NFS service after install.
REG ADD HKLM\Software\Microsoft\ClientForNFS\CurrentVersion\Default /v AnonymousUid /t REG_DWORD /d 1000
REG ADD HKLM\Software\Microsoft\ClientForNFS\CurrentVersion\Default /v AnonymousGid /t REG_DWORD /d 1001
Turn Windows features on or off > Client for NFS
mount -o anon 10.20.30.123:/mnt/md0/sharedfolder z:
The following things might help you, I haven't fully tested them yet. But I have all of the following applied.
If you're doing an iperf3 from Windows, and it just seems wrong, remember to do parallel connections. Because Windows is Windows.. (iperf3 -c iphere -P 20)
Find the PCI id (in my case 80861528)and turn up for the bandwidth on it - I had no speed increase by this:
setpci -v -d 8086:1528 e6.b=2e
Add MTU 9000:
allow-hotplug eth2
iface eth2 inet static
address 10.11.12.13
netmask 255.255.255.0
mtu 9000
Apply some fancy things to /etc/sysctl.conf, if MTU9000 doesn't achieves the results you want (reload with sysctl -p):
net.ipv4.tcp_reordering = 16
net.ipv4.tcp_fack = 0
net.ipv4.tcp_dsack = 0
net.ipv4.tcp_allowed_congestion_control = htcp reno highspeed scalable lp
net.ipv4.tcp_congestion_control = highspeed
net.ipv4.tcp_low_latency = 1
net.ipv4.tcp_slow_start_after_idle = 0
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_adv_win_scale = 4
net.ipv4.tcp_mem = 16777216 33554432 67108864
net.ipv4.tcp_rmem = 8388608 16777216 33554432
net.ipv4.tcp_wmem = 8388608 16777216 33554432
net.ipv4.udp_mem = 2097152 8388608 16777216
net.ipv4.udp_rmem_min = 262144
net.ipv4.udp_wmem_min = 262144
net.core.rmem_default = 4194304
net.core.wmem_default = 8388608
net.core.rmem_max = 16777216
net.core.wmem_max = 33554432
net.core.optmem_max = 4194304
net.core.somaxconn = 8192
net.core.netdev_max_backlog = 3000000
You seriously need a backup solution. I used to use Crashplan as my primary. But now they don't exists for home use anymore, so this section is just kept in case you go for their business version. And because it looks cool to have a lot of content on this page.
But please, do not set it to backup your movie collection or something that you can easily get again. It would be really sad if they started to prevent NAS's from using their service!
apt-get install libXtst-dev
wget https://download.code42.com/installs/linux/install/CrashPlan/CrashPlan_4.5.2_Linux.tgz
tar zvfx CrashPlan_4.5.2_Linux.tgz
cd crashplan-install/
#enter
#enter
#q
#yes (enter)
#enter
#enter
#enter
#enter
#enter
#enter
#enter
#enter
#enter
#enter
# Make sure you have X11 forwarded, also please don't run this as root (I did it anyway).
/usr/local/bin/CrashPlanDesktop
# If the gui doesn't start, check /usr/local/crashplan/log/ui_output.log for more info.
I should write something useful here, but that must be some other day…
Until then, run rsync -a /source/folder /destination/folder to sync locally, if you want something to look at, then run rsync -av –progress /source/folder /destination/folder. To create a complete copy, you might want to add rsync -a –delete /source/folder /destination/folder to your cron, however, if you accidentally delete stuff at source. You're doomed. Do a test simulation with –dry-run
Oh yes, lets introduce some more possible security flaws to our system! <3
Your own personalized cloud! For the clients I recommend that you use Syncthing-GTK instead of Syncthing. It's much better when you have a gui, and want to be able to sync file changes as soon as they happen.
apt-get install curl
curl -s https://syncthing.net/release-key.txt | apt-key add -
echo "deb http://apt.syncthing.net/ syncthing release" | tee /etc/apt/sources.list.d/syncthing.list
apt-get update
apt-get install syncthing
useradd syncthing -G smbgroup -d /mnt/md0/syncthing
mkdir /mnt/md0/syncthing
chmod -R 0770 /mnt/md0/syncthing
chown -R syncthing:smbgroup /mnt/md0/syncthing
[Unit]
Description=Syncthing
After=network-online.target
[Service]
Type=simple
User=syncthing
Group=smbgroup
UMask=007
ExecStart=/usr/bin/syncthing
Restart=on-failure
# Configures the time to wait before service is stopped forcefully.
TimeoutStopSec=300
[Install]
WantedBy=multi-user.target
systemctl start syncthing
systemctl status syncthing
systemctl enable syncthing
And append the following to /etc/samba/smb.conf:
[syncthing]
path = /mnt/md0/syncthing
valid users = mathias
guest ok = no
writable = yes
browsable = no
create mask = 0750
directory mask = 2750
force user = syncthing
force group = smbgroup
chmod +x /etc/init.d/syncthing
/etc/init.d/syncthing start
sed -i 's/<address>127.0.0.1:8384<\/address>/<address>0.0.0.0:8384<\/address>/g' /mnt/md0/syncthing/.config/syncthing/config.xml
/etc/init.d/syncthing restart # Now visit your.nas.ip.adr:8384 and create login info
/etc/init.d/samba restart
Emby is a super awesome alternative to Plex. I would actually say it's better than Plex!
It use it to sync my Kodi watch status, and auto scrobble what I'm watching to Trakt.tv
#These two lines are not really needed, but will avoid future warning messages.
wget http://download.opensuse.org/repositories/home:emby/Debian_8.0/Release.key
apt-key add - < Release.key
echo 'deb http://download.opensuse.org/repositories/home:/emby/Debian_8.0/ /' >> /etc/apt/sources.list.d/emby-server.list
apt-get update
apt-get install emby-server -y --force-yes
usermod -g smbgroup emby
systemctl start emby-server # http://ip.of.your.nas:8096/
systemctl status emby-server
systemctl enable emby-server
A super great torrent client! With a neat web interface and full proxy support. You can even connect the desktop client directly too it if you don't want an webinterface.
apt-get update
apt-get install deluge deluged deluge-web
usermod -g smbgroup debian-deluged
update-rc.d deluge-daemon remove
[Unit]
Description=Deluge Bittorrent Client Daemon
After=network-online.target
[Service]
Type=simple
User=debian-deluged
Group=smbgroup
UMask=007
ExecStart=/usr/bin/deluged -d
Restart=on-failure
# Configures the time to wait before service is stopped forcefully.
TimeoutStopSec=300
[Install]
WantedBy=multi-user.target
[Unit]
Description=Deluge Bittorrent Client Web Interface
After=network-online.target
[Service]
Type=simple
User=debian-deluged
Group=smbgroup
UMask=007
ExecStart=/usr/bin/deluge-web
Restart=on-failure
[Install]
WantedBy=multi-user.target
sed -i 's/ENABLE_DELUGED=0/ENABLE_DELUGED=1/g' /etc/default/deluged
systemctl start deluged
systemctl status deluged
systemctl enable deluged
systemctl start deluge-web # http://your.nas.ip.here:8112 default password: deluge - after password change, do a F5 and connect to daemon
systemctl status deluge-web
systemctl enable deluge-web