Difference between revisions of "Diskruimte"

From Cncz
Jump to navigation Jump to search
m (Changing the en description of unix user rights to be grammatically correct and gender neutral.)
 
(31 intermediate revisions by 5 users not shown)
Line 1: Line 1:
 
[nl]
 
[nl]
De diskruimte op [[Hardware_servers|C&CZ servers]] kan door gebruikers benaderd worden vanaf allerlei [[Hardware_servers|C&CZ servers en werkplekken]], maar ook vanaf andere werkplekken of zelfs vanaf thuis met WinSCP of VPN.
+
= Diskruimte =
 +
De diskruimte op [[Hardware_servers|C&CZ servers]] kan door gebruikers benaderd worden vanaf allerlei [[Hardware_servers|C&CZ servers en werkplekken]], maar ook vanaf andere werkplekken of zelfs vanaf thuis met WinSCP of [[Vpn|VPN]].
 
Van vrijwel alle schijven die onder beheer van C&CZ staan, worden regelmatig [[Backup|backups]] gemaakt, zodat bij kleine of grote calamiteiten de data niet verloren gaan.
 
Van vrijwel alle schijven die onder beheer van C&CZ staan, worden regelmatig [[Backup|backups]] gemaakt, zodat bij kleine of grote calamiteiten de data niet verloren gaan.
 
[/nl]
 
[/nl]
 
[en]
 
[en]
The diskspace on [[Hardware_servers|C&CZ servers]] can be used from all kind of [[Hardware_servers|C&CZ servers and personal computers]], but also from other PCs or even from home with WinSCP or VPN. Almost all disks that are managed by C&CZ, are being [[Backup|backupped]] regularly, in order to be able to restore data in case of small or large calamities.
+
= Diskspace =
 +
The diskspace on [[Hardware_servers|C&CZ servers]] can be used from all kind of [[Hardware_servers|C&CZ servers and personal computers]], but also from other PCs or even from home with WinSCP or [[Vpn|VPN]]. Almost all disks that are managed by C&CZ, are being [[Backup|backed up]] regularly, in order to be able to restore data in case of small or large calamities.
 
[/en]
 
[/en]
 
==Home directories==
 
==Home directories==
Line 41: Line 43:
 
[/nl]
 
[/nl]
 
[en]
 
[en]
Long ago the (Unix) home directory of a user, except for a few protected areas, was readable for all users of the server. Today the home directory of a user can be accessed by the user himself. The user can change the access rights. C&CZ checks for home directories that are writable by other users.
+
Long ago the (Unix) home directory of a user, except for a few protected areas, was readable for all users of the server. Nowadays a user's home directory is only accessible to the user themself. The user can change the access rights. C&CZ checks for home directories that are writable by other users.
 
[/en]
 
[/en]
  
==[Netwerk schijven][Network shares]==
+
=== [Toegang via NFS][Access through NFS] ===
 +
 
 
[nl]
 
[nl]
Aparte schijfruimte (diskruimte) voor groepen/afdelingen/projecten: er zijn enkele [http://nl.wikipedia.org/wiki/Redundant_Array_of_Independent_Disks RAID-arrays] waarop partities gehuurd kunnen worden voor een periode van 3 jaar. De prijs voor nieuwe schijven of verlenging van oude schijven is per juli 2018 intern FNWI:
+
Een homedirectory (U: schijf) kan op Linux aangekoppeld worden via [[Mount_Homedisk|NFS/Kerberos (English only)]].
 
[/nl]
 
[/nl]
 
[en]
 
[en]
Diskspace for groups/institutions/projects: there are a few [http://en.wikipedia.org/wiki/RAID_array RAID arrays] with partitions that can be rented for a period of 3 years. The price for new discs or a new 3 year extension of an older disc is per July 2018 for FNWI departments:   
+
Mounting a home (U:) drive on Linux via [[Mount_Homedisk|NFS/Kerberos]].
 +
[/en]
 +
 
 +
==[Functionaliteit en prijs van netwerkschijven][Functionality and costs of network shares]==
 +
=== [RAID server schijven][RAID server shares] ===
 +
[nl]
 +
Aparte schijfruimte (diskruimte) voor groepen/afdelingen/projecten: er zijn enkele fileservers met [http://nl.wikipedia.org/wiki/Redundant_Array_of_Independent_Disks RAID opslag] waarop partities gehuurd kunnen worden voor een periode van 3 jaar. De prijs voor nieuwe schijven of verlenging van oude schijven is per juli 2018 intern FNWI:
 +
[/nl]
 +
[en]
 +
Diskspace for groups/institutions/projects: there are a few fileservers with [http://en.wikipedia.org/wiki/RAID_array RAID storage] with partitions that can be rented for a period of 3 years. The price for new discs or a new 3 year extension of an older disc is per July 2018 for FNWI departments:   
 
[/en]
 
[/en]
  
Line 64: Line 76:
 
| align="left" | € 80 per [jaar][year]
 
| align="left" | € 80 per [jaar][year]
 
| align="left" | € 20 per [jaar][year]
 
| align="left" | € 20 per [jaar][year]
 +
|-
 +
| align="left" | > 400 GB up to 1 TB (no daily backup?)
 +
| align="left" | € ??? per TB/[jaar][year]
 +
| align="left" | € 50 per TB/[jaar][year]
 +
|-
 +
| align="left" | > 1 TB (no backup?)
 +
| align="left" | N/A
 +
| align="left" | Have a look at Ceph storage
 +
|}
 +
 +
[nl]
 +
Hoewel zelfs de goedkoopste versie aanmerkelijk duurder is dan het aanschaffen van 1 harde schijf voor 1 PC, is het vaak zinvol vanwege de betrouwbaarheid (redundante schijven, backup, onderhoudscontract) en bedrijfszekerheid (stabiele server). Een of meer mappen op zo'n partitie kunnen op Windows PCs beschikbaar worden als een schijf, door het maken van een netwerkverbinding. Op Unix/Linux computers kunnen ze via NFS aangekoppeld worden. De mogelijkheid om te lezen en/of te schrijven in zo'n map kan beperkt worden tot een groep logins. Die groep kan door de afdeling zelf beheerd worden via de [[Dhz|Doe-Het-Zelf website]].<br> C&amp;CZ heeft op die servers een service contract afgesloten en beschikt over reserve-apparatuur, waardoor een storing relatief snel opgelost kan worden. Omdat de schijven in een RAID-set opgenomen zijn, veroorzaakt het uitvallen van 1 of zelfs 2 schijven, geen storing voor gebruikers. Van de partities worden ook (dagelijks en incrementeel) [[Backup|backups]]  gemaakt, zodat zelfs bij uitval van de hele serverruimte de data nog (ooit) teruggezet kunnen worden.
 +
[/nl]
 +
[en]
 +
Although even the cheapest version is much more expensive than buying 1 disk for 1 PC, it often makes sense, because of the reliability (redundant disks, backup, support contract) and security (stable server).  One or more folders on such a partition can be mapped as a network drive on Windows PCs or NFS-mounted on Unix/Linux hosts. The ability to read and/or write files on these folders can be limited to a group of logins. That group can be managed by the department on the [[Dhz|Do-It-Yourself website]].<br> C&amp;CZ has service contracts for these servers and has spares on site, so a failure can be resolved quite fast. Because the disks are part of a RAID set, the failure of 1 single disk or even 2 disks, will not give a disruption of service for users. The partitions are [[Backup|backed up]] (daily and incremental). Even in the case when the whole server room is lost, data can (eventually) be restored.
 +
[/en]
 +
 +
=== Ceph Storage ===
 +
 +
[nl]
 +
Vanaf november 2019 kunnen we bijna oneindige storage aanbieden voor FNWI middels ons Ceph storage cluster. Door hoe Ceph werkt is er een afweging tussen snelheid en redundantie. Met de extra redundantie opties kunnen we zelfs betere beschikbaarheid bieden dan bij RAID-6 systemen. De fysieke opslag servers zijn verdeeld over 3 locaties (datacenters).
 +
'''NB Ceph volumes hebben geen backups, de volumes zijn doorgaans te groot om te kunnen backuppen.'''
 +
[/nl]
 +
[en]
 +
Starting November 2019 we can provide almost unlimited storage for the Faculty of Science using our Ceph storage cluster. The way Ceph works there is a tradeoff for performance and redundancy. Also it is possible to improve redundancy above single server RAID-6 level, with the additional redundancy options. The physical storage servers are spread accros three locations (datacenters).
 +
'''NB Ceph volumes have no backups, the volumes tend to be too large to backup.'''
 +
[/en]
 +
==== [Keuzes in redundantie][Choices in redundancy] ====
 +
[nl]
 +
Ceph heeft de mogelijkheid om data op verschillende manieren op te slaan (per "pool" te kiezen), standaard worden blokken 3 maal opgeslagen, zodat bij verlies van ́é́én blok er nog twee kopiëen over blijven. Inmiddels is het ook zo dat de 3copy pool blijft werken bij uitval van een heel datacenter. In het begin hadden we twee locaties, waardoor we voor betere betrouwbaarheid een 4copy pool hebben gemaakt, maar die voegt met drie locaties weinig toe aan de betrouwbaarheid.
 +
 +
Naast het opslaan van kopiëen kent Ceph ook nog "Erasure Coding" (EC) als vorm van redundantie. Het voordeel van deze manier is dat je minder redundantie overhead hebt, door het gebruik van een algoritme zoals bijvoorbeeld RAID-6 toe te passen. Het nadeel van EC is dat de overhead voor kleine bestanden erg groot is. Er zijn verschillende EC pools; EC8+3: goedkoop, maar bij vernietiging van 1 heel datacenter is alle data weg (heel onwaarschijnlijk dat dat gebeurt), bij tijdelijke uitval van een datacenter is de data veilig, maar even niet beschikbaar. De EC5+4 pool blijft beschikbaar en schrijfbaar bij uitval van 1 datacenter en bij vernietiging van een datacenter is de data nog veilig.
 +
 +
'''Ceph Erasure coding heeft een grote overhead bij kleine files, de prijzen in de tabel hieronder zijn gebaseerd op de optimale overhead, die pas benaderd wordt met files groter dan 4 megabytes.'''
 +
[/nl]
 +
[en]
 +
Ceph has different options for storing data (configurable per "pool"). By default, Ceph stores data with 3 copies, so when one copy is lost, the remaining two still have redundancy. Now, because we have three locations, the 3copy pool will remain available when one whole datacenter becomes unavailable. When we started, we created a 4copy pool, so it would remain available when a location was off-line, but with three locations, this adds little to the redundancy.
 +
 +
Besides storing copies of the data blocks, Ceph can use "Erasure Coding" (EC) as alternative way of providing redundancy. The advantage is that much less overhead is required for secure storage, but the disadvantage is high overhead for storing small files. We have several different EC pools; EC8+3, the cheapest, but when one datacenter is destroyed, all the data is lost (very unlikely!), when one datacenter becomes temporarily unavailable, the data is still safe, but off-line. Our EC5+4 pool remains available when a whole datacenter is offline or lost, the data remains safe as long as two datacenters are working well.
 +
 +
'''Ceph Erasure coding has a high overhead for smaller files, the prices mentioned below are based on the optimal storage overhead, which can be approximated when files stored are at least 4 megabytes or larger.'''
 +
[/en]
 +
 +
'''NB, 1 TB is 1.000.000.000.000 bytes'''
 +
 +
{| class="wikitable"
 +
| align="left" | '''Pool'''
 +
| align="left" | '''[waarom][why]'''
 +
| align="left" | '''[prijs][price] per TB per [jaar][year] [zonder][without] backup'''
 +
|-
 +
| align="left" | Erasure coding 8+3 (*)
 +
| align="left" | [goedkoop][cheap]
 +
| align="left" | &euro; 50 (was 45)
 +
|-
 +
| align="left" | '''Erasure coding 5+4'''
 +
| align="left" | [goedkoop + extra redundantie][cheap + additional redundancy]
 +
| align="left" | &euro; 60
 +
|-
 +
| align="left" | '''3 copy'''
 +
| align="left" | [snellere][faster] r+w
 +
| align="left" | &euro; 100
 +
|-
 +
| align="left" | 4 copy (**)
 +
| align="left" | [snellere r+w + extra redundantie][faster r+w + additional redundancy]
 +
| align="left" | &euro; 135
 
|}
 
|}
  
 
[nl]
 
[nl]
Hoewel zelfs de goedkoopste versie aanmerkelijk duurder is dan het aanschaffen van 1 harde schijf voor 1 PC, is het vaak zinvol vanwege de betrouwbaarheid (redundante schijven, backup, onderhoudscontract) en bedrijfszekerheid (stabiele server). Een of meer mappen op zo'n partitie kunnen op Windows PCs beschikbaar worden als een schijf, door het maken van een netwerkverbinding. Op Unix/Linux computers kunnen ze via NFS aangekoppeld worden. De mogelijkheid om te lezen en/of te schrijven in zo'n map kan beperkt worden tot een groep logins.<br> C&amp;CZ heeft op die RAID-arrays garantie afgesloten en beschikt over reserve-apparatuur, waardoor een storing snel opgelost kan worden. Omdat het RAID-arrays zijn, veroorzaakt het uitvallen van 1 enkele schijf geen storing voor de gebruikers. Van de partities worden ook (dagelijks en incrementeel) [[Backup|backups]]  gemaakt, zodat zelfs bij uitval van de hele computerruimte de data nog (ooit) teruggezet kunnen worden.
+
'''* EC8+3 prijs''' is van 45 naar 50 euro gegaan per 1 januari 2022, dit is vanwege het gebruik, waarbij blijkt dat er toch erg veel kleine files opgeslagen zijn, wat in Ceph tot flinke overhead zorgt.
 +
[/nl]
 +
[en]
 +
'''* EC8+3 price''' is raised from 45 to 50 euro as of January 1st 2022, due to the way the storage is used with large amounts of very small files. This creates a significant overhead in Ceph.
 +
[/en]
 +
 
 +
[nl]
 +
'''**4copy'''. De 4copy pool bleef beschikbaar toen we 2 datacenters hadden, als een van de twee datacenters uitviel, de 3copy pool blijft nu ook beschikbaar bij het uitvallen van een heel datacenter, daarmee is het voordeel van de 4copy pool grotendeels weg. De 4copy pool heeft wel het voordeel van een extra kopie, maar dat maakt niet uit bij het uitvallen van een heel datacenter.
 +
[/nl]
 +
[en]
 +
'''**4copy''' The 4copy pool used to have the advantage, when we had only two locations, that it would remain usable when one datacenter location failed. Now the 3copy pool will remain active when one datacenter breaks, because we have three locations. Of course, the 4copy pool will have the additional benefit of an extra copy, but this has no advantage when a whole datacenter breaks.
 +
[/en]
 +
 
 +
 
 +
[nl]
 +
De Ceph storage kan in overleg worden aangeboden als Windows/Samba share, NFS share en als S3 object store. Object store is fundamenteel anders dan een normaal bestandssysteem, dus als S3 gewenst is, kan de de data niet ook als Windows share of NFS share gebruikt worden.
 +
 
 +
De performance eigenschappen van Ceph zijn anders dan normale netwerk opslag op losse servers; de schrijfsnelheden zijn over het algemeen groter dan de leessnelheid en nog meer dan bij traditionele opslag zijn kleine bestanden funest voor de doorvoersnelheid.
 
[/nl]
 
[/nl]
 
[en]
 
[en]
Although even the cheapest version is much more expensive than buying 1 disk for 1 PC, it often makes sense, because of the reliability (redundant disks, backup, support contract) and security (stable server).  One or more folders on such a partition can be mapped as a network drive on Windows PCs or NFS-mounted on Unix/Linux hosts. The ability to read and/or write files on these folders can be limited to a group of logins.<br> C&amp;CZ has bought warranty for these RAID arrays and has spares on site, so a failure can be resolved quite fast. Because it is a RAID-array, the failure of 1 single disk will not give a disruption of service for users. The partitions are [[Backup|backed up]] (daily and incremental). Even in the case when the whole computer room is lost, data can (eventually) be restored.
+
The Ceph storage can be used as Windows/Samba share, NFS share or S3 object store. Object store differs fundamentally from a normal filesystem, so data stored in a Windows or NFS share cannot be accessed using the S3 protocol.
 +
 
 +
The performance properties of Ceph are different from traditional single server storage; write speed usually exceeds read speed and lots of small files is killing for throughput, even worse than on traditional storage.
 
[/en]
 
[/en]
 +
 
=== [Naamgeving][Naming] ===
 
=== [Naamgeving][Naming] ===
 
{| class="wikitable"
 
{| class="wikitable"
Line 95: Line 192:
 
[en]
 
[en]
 
Most of the shared disks can be read and written by a specific group of users. The owners of this group can administer  on the [[Dhz|Do-It-Yourself website]] which accounts are a member of this group.
 
Most of the shared disks can be read and written by a specific group of users. The owners of this group can administer  on the [[Dhz|Do-It-Yourself website]] which accounts are a member of this group.
 +
[/en]
 +
 +
=== [Aanvragen][Requests] ===
 +
 +
[nl]
 +
Bij een aanvraag van een of meer netwerkschijven zal vermeld moeten worden:
 +
* gewenste naam/namen van de schijf/schijven
 +
* gewenste grootte (max ca. 500GB bij backup)
 +
* evt. gewenste backup-schema's om kosten te beperken (Daily/Monthly/Yearly)
 +
* Science loginnaam van een eigenaar
 +
* evt. Science loginnaam van een lid
 +
* kostenplaats of projectcode voor de kosten voor de eerste drie jaar.
 +
[/nl]
 +
[en]
 +
A request for one or more network discs should contain:
 +
* requested name of the disc(s)
 +
* requested size (max ca. 500GB with backup)
 +
* possibly requested backup schedules to lower the price (Daily/Monthly/Yearly)
 +
* Science loginname of an owner
 +
* possibly Science loginname of a member
 +
* charge account (kostenplaats) or project code for the costs in the first three years.
 
[/en]
 
[/en]
  
Line 105: Line 223:
  
 
Merk op dat ook in dat geval oude bestanden verwijderd worden.
 
Merk op dat ook in dat geval oude bestanden verwijderd worden.
 +
 +
Zorg ervoor dat de tijdstempels van bestanden zijn bijgewerkt wanneer u een bestand naar deze share kopieert. Sommige kopieerprogramma's (zoals rsync) behouden de originele tijdstempels en oudere bestanden worden dan verwijderd. Om tijdstempels bij te werken, kunt u de volgende opdracht gebruiken:
 +
find . -exec touch {} +
  
 
Maak alsjeblieft eerst een sub-map aan met b.v. je Science-loginnaam, zet de tijdelijke bestanden vervolgens in die map.
 
Maak alsjeblieft eerst een sub-map aan met b.v. je Science-loginnaam, zet de tijdelijke bestanden vervolgens in die map.
  
Voor bestanden samen kleiner dan 250GB is ook [https://wiki.science.ru.nl/cncz/index.php?title=Nieuws&setlang=nl#.5BSURFdrive:_nu_250_GB_per_gebruiker.5D.5BSURFdrive:_now_250_GB_per_user.5D Surfdrive] een alternatief. Voor eenmalig versturen van bestanden tot 500GB is [https://www.surffilesender.nl/ SURFfilesender] geschikt.
+
Voor bestanden samen kleiner dan 250GB is ook [https://wiki.science.ru.nl/cncz/index.php?title=Nieuws&setlang=nl#.5BSURFdrive:_nu_250_GB_per_gebruiker.5D.5BSURFdrive:_now_250_GB_per_user.5D Surfdrive] een alternatief. Voor het versturen van bestanden tot 500GB is [https://www.surffilesender.nl/ SURFfilesender] geschikt.
 
[/nl]
 
[/nl]
 
[en]
 
[en]
 
Every now and then you want to send one or more large files (more than a few tens of MBs) to someone else within the Faculty, mail is unsuited for those large files.
 
Every now and then you want to send one or more large files (more than a few tens of MBs) to someone else within the Faculty, mail is unsuited for those large files.
 
To make this easy, one can use a network share, where one can store large files temporarily in order to have someone else copy the files from this location. Note that this is explicitly meant for temporary storage, we do not make backups of this share, every day we remove files older than 21 days old.
 
To make this easy, one can use a network share, where one can store large files temporarily in order to have someone else copy the files from this location. Note that this is explicitly meant for temporary storage, we do not make backups of this share, every day we remove files older than 21 days old.
 +
When copying files to this share, make sure the file timestamps are updated. Some copy programs (like rsync) maintain the original timestamps and older files will be deleted. To update timestamps, you can use the following command:
 +
find . -exec touch {} +
  
 
This share can also be used to store temporary files only readable for yourself by using a different name for the share.
 
This share can also be used to store temporary files only readable for yourself by using a different name for the share.

Latest revision as of 14:36, 8 September 2022

Diskspace

The diskspace on C&CZ servers can be used from all kind of C&CZ servers and personal computers, but also from other PCs or even from home with WinSCP or VPN. Almost all disks that are managed by C&CZ, are being backed up regularly, in order to be able to restore data in case of small or large calamities.

Home directories

Every user with a Science login has or is entitled to an amount of disc space of a few Gigabytes on a server. This disc space is called the "home-directory" on Unix/Linux computers and the "H- or U-drive" on Windows-computers. The location of this homedirectory (which server) can be viewed on the Do-It-Yourself website.

Naming

Username guest204
SMB (Windows/...) name \\home1.science.ru.nl\guest204

or
\\home2.science.ru.nl\guest204
check on DIY

URL (Apple/Linux/Android/...) name smb://home1.science.ru.nl/guest204

or
smb://home2.science.ru.nl/guest204
check on DIY

NFS (C&CZ beheerd Linux) name /home/guest204

Access rights

Long ago the (Unix) home directory of a user, except for a few protected areas, was readable for all users of the server. Nowadays a user's home directory is only accessible to the user themself. The user can change the access rights. C&CZ checks for home directories that are writable by other users.

Access through NFS

Mounting a home (U:) drive on Linux via NFS/Kerberos.

Functionality and costs of network shares

RAID server shares

Diskspace for groups/institutions/projects: there are a few fileservers with RAID storage with partitions that can be rented for a period of 3 years. The price for new discs or a new 3 year extension of an older disc is per July 2018 for FNWI departments:

size incl. backup without backup
ca. 200 GB € 40 per year € 10 per year
ca. 400 GB € 80 per year € 20 per year
> 400 GB up to 1 TB (no daily backup?) € ??? per TB/year € 50 per TB/year
> 1 TB (no backup?) N/A Have a look at Ceph storage

Although even the cheapest version is much more expensive than buying 1 disk for 1 PC, it often makes sense, because of the reliability (redundant disks, backup, support contract) and security (stable server). One or more folders on such a partition can be mapped as a network drive on Windows PCs or NFS-mounted on Unix/Linux hosts. The ability to read and/or write files on these folders can be limited to a group of logins. That group can be managed by the department on the Do-It-Yourself website.
C&CZ has service contracts for these servers and has spares on site, so a failure can be resolved quite fast. Because the disks are part of a RAID set, the failure of 1 single disk or even 2 disks, will not give a disruption of service for users. The partitions are backed up (daily and incremental). Even in the case when the whole server room is lost, data can (eventually) be restored.

Ceph Storage

Starting November 2019 we can provide almost unlimited storage for the Faculty of Science using our Ceph storage cluster. The way Ceph works there is a tradeoff for performance and redundancy. Also it is possible to improve redundancy above single server RAID-6 level, with the additional redundancy options. The physical storage servers are spread accros three locations (datacenters). NB Ceph volumes have no backups, the volumes tend to be too large to backup.

Choices in redundancy

Ceph has different options for storing data (configurable per "pool"). By default, Ceph stores data with 3 copies, so when one copy is lost, the remaining two still have redundancy. Now, because we have three locations, the 3copy pool will remain available when one whole datacenter becomes unavailable. When we started, we created a 4copy pool, so it would remain available when a location was off-line, but with three locations, this adds little to the redundancy.

Besides storing copies of the data blocks, Ceph can use "Erasure Coding" (EC) as alternative way of providing redundancy. The advantage is that much less overhead is required for secure storage, but the disadvantage is high overhead for storing small files. We have several different EC pools; EC8+3, the cheapest, but when one datacenter is destroyed, all the data is lost (very unlikely!), when one datacenter becomes temporarily unavailable, the data is still safe, but off-line. Our EC5+4 pool remains available when a whole datacenter is offline or lost, the data remains safe as long as two datacenters are working well.

Ceph Erasure coding has a high overhead for smaller files, the prices mentioned below are based on the optimal storage overhead, which can be approximated when files stored are at least 4 megabytes or larger.

NB, 1 TB is 1.000.000.000.000 bytes

Pool why price per TB per year without backup
Erasure coding 8+3 (*) cheap € 50 (was 45)
Erasure coding 5+4 cheap + additional redundancy € 60
3 copy faster r+w € 100
4 copy (**) faster r+w + additional redundancy € 135

* EC8+3 price is raised from 45 to 50 euro as of January 1st 2022, due to the way the storage is used with large amounts of very small files. This creates a significant overhead in Ceph.

**4copy The 4copy pool used to have the advantage, when we had only two locations, that it would remain usable when one datacenter location failed. Now the 3copy pool will remain active when one datacenter breaks, because we have three locations. Of course, the 4copy pool will have the additional benefit of an extra copy, but this has no advantage when a whole datacenter breaks.


The Ceph storage can be used as Windows/Samba share, NFS share or S3 object store. Object store differs fundamentally from a normal filesystem, so data stored in a Windows or NFS share cannot be accessed using the S3 protocol.

The performance properties of Ceph are different from traditional single server storage; write speed usually exceeds read speed and lots of small files is killing for throughput, even worse than on traditional storage.

Naming

Volume name sharename
SMB (Windows/..) name \\sharename-srv.science.ru.nl\sharename
URL (Apple/Linux/Android/...) name smb://sharename-srv.science.ru.nl/sharename
NFS (C&CZ-beheerd Linux) name /vol/sharename

Access rights

Most of the shared disks can be read and written by a specific group of users. The owners of this group can administer on the Do-It-Yourself website which accounts are a member of this group.

Requests

A request for one or more network discs should contain:

  • requested name of the disc(s)
  • requested size (max ca. 500GB with backup)
  • possibly requested backup schedules to lower the price (Daily/Monthly/Yearly)
  • Science loginname of an owner
  • possibly Science loginname of a member
  • charge account (kostenplaats) or project code for the costs in the first three years.

Temporary shared diskspace

Every now and then you want to send one or more large files (more than a few tens of MBs) to someone else within the Faculty, mail is unsuited for those large files. To make this easy, one can use a network share, where one can store large files temporarily in order to have someone else copy the files from this location. Note that this is explicitly meant for temporary storage, we do not make backups of this share, every day we remove files older than 21 days old. When copying files to this share, make sure the file timestamps are updated. Some copy programs (like rsync) maintain the original timestamps and older files will be deleted. To update timestamps, you can use the following command:

find . -exec touch {} +

This share can also be used to store temporary files only readable for yourself by using a different name for the share. Note that also in this case, old files will be removed.

Please create a subdirectory with your name first, and put your files in that directory.

For files totaling less than 250GB, also Surfdrive is an alternative. For sending files up to 500GB SURFfilesender can be used.

Naming

Volume name temp
SMB (Windows/..) name \\temp-srv.science.ru.nl\share

or
\\temp-srv.science.ru.nl\onlyme

URL (Apple/Linux/Android/...) name smb://temp-srv.science.ru.nl/share

or
smb://temp-srv.science.ru.nl/onlyme

NFS (C&CZ-beheerd Linux) name /vol/temp

Access rights

  • Readable by all users: share
  • Only readable for the owner: onlyme

Lijn.png