Difference between revisions of "Recente Storingen"

From Cncz
Jump to: navigation, search
([EduroamCAT niet bruikbaar met Science accounts][EduroamCAT not working with Science accounts])
m ([Netwerkswitch van serverruimte stroomloos][Server room network switch powerless])
(37 intermediate revisions by 3 users not shown)
Line 37: Line 37:
  
 
<startFeed />
 
<startFeed />
=== [DNS-problemen vanaf buiten met ru.nl][DNS problems from outside with ru.nl] ===
+
 
<itemTags>medewerkers,studenten,websites</itemTags>
+
=== [Windows 7 computers disabled in B-FAC domain][Windows 7 computers disabled in B-FAC domain] ===
 +
<itemTags>medewerkers,docenten</itemTags>
 
[nl]
 
[nl]
   Begin        : 2021-02-21
+
   Begin        : 2021-03-24
   Eind          :  
+
   Eind          : na upgrade naar ander OS
   Getroffen    : iedereen die iets in ru.nl wil benaderen van buiten de campus
+
   Getroffen    : gebruikers van Windows 7 in het B-FAC domein
  
De externe DNS-servers van ru.nl werken niet goed, waardoor ook o.a. science.ru.nl soms niet gevonden kan worden: DNS-namen onder ru.nl resolven dan niet naar een IP-adres. We hebben enkele TTL's (time-to-lives) vergroot om te proberen de overlast iets minder te maken. Deze kleine TTL's waren bedoeld om in het geval van problemen met deze servers de service snel te kunnen verhuizen, maar dragen nu bij aan de overlast. Wie VPN gebruikt heeft na het starten van de VPN hier geen probleem meer mee, omdat dan de interne DNS-servers gebruikt worden. We hopen dat men de DNS-problemen met ru.nl snel kan verhelpen.
+
I.v.m. het verscherpen van de beveiliging worden de laatste Windows 7 machines per 24-03-20221 in het Active Directory Domain B-FAC gedisabled.
 +
Verzoek is al sinds lang om de betreffende machines naar een meer up-to-date OS te upgraden.
 +
Zie evt. eerdere aankondigingen over [https://wiki.cncz.science.ru.nl/Nieuws#.5BMicrosoft_Windows_10_upgrade.5D.5BMicrosoft_Windows_10_upgrade.5D Windows 10]
 +
en [https://wiki.cncz.science.ru.nl/Nieuws_archief#.5BWindows_7_stopt_januari_2020:_Upgrade_nu.21.5D.5BWindows_7_ends_January_2020:_Upgrade_now.21.5D het einde van Windows 7].
 
[/nl]
 
[/nl]
 
[en]
 
[en]
   Begin        : 2021-02-21
+
   Begin        : 2021-03-24
   End          :
+
   End          : after upgrade to other OS
   Affected      : everyone trying to access something in ru.nl from off-campus.
+
   Affected      : users of Windows 7 in the B-FAC domain
  
The external DNS servers for ru.nl have problems, which result in science.ru.nl a.o. not being found. DNS names within ru.nl then will resolve to an IP address. We enlarged some TTLs (Time-To+lives) to try to lessen the problem. These small TTLs were meant to be able to move a service to a new server in case of problems, but now they just make the problem bigger. After starting VPN you won't notice this problem, because the internal DNS servers that you use then are not affected. We hope the the administrators of ru.nl can fix this soon.
+
Because of security issues the last remaining Windows 7 machines wil be disabled, effective 24-03-2021, as member of the Active Directory Domain B-FAC.
 +
Please upgrade these computers to a more up-to-date OS.
 +
See also previous announcements on [https://wiki.cncz.science.ru.nl/Nieuws#.5BMicrosoft_Windows_10_upgrade.5D.5BMicrosoft_Windows_10_upgrade.5D Windows 10]
 +
and
 +
[https://wiki.cncz.science.ru.nl/Nieuws_archief#.5BWindows_7_stopt_januari_2020:_Upgrade_nu.21.5D.5BWindows_7_ends_January_2020:_Upgrade_now.21.5D the end of Windows 7].
 
[/en]
 
[/en]
  
Line 78: Line 86:
  
 
<startFeed />
 
<startFeed />
 +
===[Netwerkswitch van serverruimte stroomloos][Server room network switch powerless] ===
 +
<itemTags>medewerkers,studenten,docenten</itemTags>
 +
[nl]
 +
  Begin        : 2021-10-12 11:50
 +
  Eind          : 2021-10-12 12:05
 +
  Getroffen    : Gebruikers van een van de vele servers achter deze switch
 +
 +
Twee van de modules van een belangrijke switch in de belangrijkste C&CZ serverruimte werden stroomloos tijdens het voorbereiden van gepland onderhoud. Hierdoor raakte ca. 75% van de servers in deze ruimte hun netwerkverbinding kwijt. Door het omzetten naar nieuwe PDU's kon de storing tot ca. 15 minuten beperkt worden.
 +
[/nl]
 +
[en]
 +
  Begin        : 2021-10-12 11:50
 +
  End          : 2021-10-12 12:05
 +
  Affected      : Users of one of the many servers behind this switch
 +
 +
Two modules of an important switch in the main C&CZ server room lost power during the preparation of planned maintenance. This disconnected ca. 75% of the servers in the room from the network. Moving the modules to new PDU's kimited the downtime to ca. 15 minutes.
 +
[/en]
 +
 +
=== [Licentieserver probleem][License server problem] ===
 +
<itemTags>medewerkers,docenten</itemTags>
 +
[nl]
 +
  Begin        : 2021-10-11 04:40
 +
  Eind          : 2021-10-11 08:26
 +
  Getroffen    : Gebruikers van een van de licenties van deze server
 +
 +
Een fout in de beheersoftware zorgde ervoor dat bij de herstart van de licentieserver geen enkel licentieproces goed opstartte. Pas na reparatie waren de licenties weer beschikbaar.
 +
[/nl]
 +
[en]
 +
  Begin        : 2021-10-11 04:40
 +
  End          : 2021-10-11 08:26
 +
  Affected      : Users of one of the licenses of this server
 +
 +
An error in the management software prevented all license processes from starting correctly at the reboot of the license server. After fixing this error, all licenses were available again.
 +
[/en]
 +
 +
=== [Fileserver 'flock' overbelast][Fileserver 'flock' overloaded] ===
 +
<itemTags>medewerkers,docenten</itemTags>
 +
[nl]
 +
  Begin        : 2021-09-17 14:30
 +
  Eind          : 2021-09-17 15:30
 +
  Getroffen    : Gebruikers van een van de ca. 100 netwerkschijven van deze server
 +
 +
Vooraf geteste cursussoftware veroorzaakte bij het gebruik door 100 studenten een te grote belasting op de fileserver. Alle gebruikers van deze fileserver hadden hier last van.
 +
[/nl]
 +
[en]
 +
  Begin        : 2021-09-17 14:30
 +
  End          : 2021-09-17 15:30
 +
  Affected      : Users of one of the
 +
 +
Course software that had been tested caused an overload of the fileserver when it was used by 100 students. The performance of the fileserver was impaired for all users of network shares of this server.
 +
[/en]
 +
 +
=== [VPN server unreachable][VPN onbereikbaar] ===
 +
<itemTags>medewerkers,docenten</itemTags>
 +
[nl]
 +
  Begin        : 2021-04-24
 +
  Eind          : 2021-04-26 09:35
 +
  Getroffen    : VPNsec gebruikers
 +
 +
Door een kapotte PDU is een switch uitgegaan en is de VPN server onbereikbaar (en nog meer dingen, waar gebruikers geen last van hebben).
 +
[/nl]
 +
[en]
 +
  Begin        : 2021-04-24
 +
  End          : 2021-04-26 09:35
 +
  Affected      : VPNsec users
 +
 +
A broken PDU has offlined a switch, which has caused the VPN server to be unreachable (and several other things, which don't affect users).
 +
[/en]
 +
 +
=== Central E-mail/Calendar disruption (exchange) ===
 +
<itemTags>medewerkers,docenten,students</itemTags>
 +
  Begin        : 2021-04-14    09:30
 +
  Eind          : 2021-04-14    13:30
 +
  Getroffen    : All users of Exchange (e-mail and calendar)
 +
 +
Due to an emergency maintenance, the central microsoft exchange server is unavailable for 4 hours. This may also affect systems that are dependent on exchange.
 +
E-mail and calendar functionality is expected to be restored when the maintenance is done around 13:30 Today.
 +
 +
=== [Ceph probleem][Ceph problem] ===
 +
<itemTags>medewerkers</itemTags>
 +
[nl]
 +
  Begin        : 2021-03-24 19:00
 +
  Eind          : 2021-03-24 21:00
 +
  Getroffen    : gebruikers van Ceph filesystemen
 +
 +
Bij een routine upgrade proces bleek dat er een bug in de laatste versie zit waardoor de ceph manager onbereikbaar werd. Het upgrade proces is afgebroken en met hulp van de ceph-users mailinglijst is alles weer bereikbaar door een work-around.[/nl]
 +
[en]
 +
  Begin        : 2021-03-24 19:00
 +
  End          : 2021-03-24 21:00
 +
  Affected      : users with ceph based filesystems
 +
 +
During a routine upgrade of ceph, a bug in the latest version manifested itself and made the ceph manager unreachable. After aborting the upgrade and with help from the ceph-users mailinglist, everything became available again using a workaround.[/en]
 +
 +
=== [Lilo7 herstart][Lilo7 restart] ===
 +
<itemTags>medewerkers,studenten</itemTags>
 +
[nl]
 +
  Begin        : 2021-03-17 21:00
 +
  Eind          : 2021-03-17 21:15
 +
  Getroffen    : gebruikers van lilo
 +
 +
Om het netwerk van lilo7 aan te passen, is het helaas noodzakelijk om deze loginserver te herstarten. Wie gedurende deze onderhoudstijd een stabiele verbinding wil hebben met een loginserver, kan beter lilo6 of de binnenkort uitgefaseerde lilo5 gebruiken. Zie evt. [https://wiki.cncz.science.ru.nl/index.php?title=Hardware_servers&setlang=nl#Linux_.5Bloginservers.5D.5Blogin_servers.5D de pagina over de C&CZ loginservers].
 +
[/nl]
 +
[en]
 +
  Begin        : 2021-03-17 21:00
 +
  End          : 2021-03-17 21:15
 +
  Affected      : users of lilo
 +
 +
To change the network of lilo7, we need to reboot this loginserver. If you want a stable connection to a loginserver during this downtime, please use lilo6 or the soon to be taken down lilo5. For more info see [https://wiki.cncz.science.ru.nl/index.php?title=Hardware_servers&setlang=en#Linux_.5Bloginservers.5D.5Blogin_servers.5D the page on C&CZ loginservers].
 +
[/en]
 +
 +
=== [Server met diverse services (virtuele servers, waaronder Roundcube en websites) stuk][Host of several virtual servers broken: Roundcube, websites and others] ===
 +
<itemTags>medewerkers,studenten</itemTags>
 +
[nl]
 +
  Begin        : 2021-03-05 07:45
 +
  Eind          : 2021-03-05 09:40
 +
  Getroffen    : gebruikers van de virtuele servers: Roundcube, websites met een database op deze server, ...
 +
 +
Gisteravond gaf de SSD-opstartschijf van deze VM-host al de eerste signalen van problemen, vanochtend stopten daardoor de virtuele servers die op deze VM-host draaien. Door de VM's te verhuizen naar een andere VM-host is het probleem opgelost. Er wordt nagedacht hoe we dit probleem in de toekomst het beste kunnen voorkomen of de impact ervan kunnen beperken.
 +
[/nl]
 +
[en]
 +
  Begin        : 2021-03-05 07:45
 +
  End          : 2021-03-05 09:40
 +
  Affected      : users of the virtual servers: Roundcube, websites with databases on this server, ...
 +
 +
Yesterday the SSD bootdisk of this VM host reported the first problems. This morning this had the effect of stopping all VMs running on this host. By moving the VMs to a different VM host, the problem has been solved. We will investigate how to best prevent this problem in the future or lessen its impact.
 +
[/en]
 +
 +
=== [Lilo6 stuk][Lilo6 down] ===
 +
<itemTags>medewerkers,studenten</itemTags>
 +
[nl]
 +
  Begin        : 2021-02-25 17:30
 +
  Eind          : 2021-03-04 16:45
 +
  Getroffen    : gebruikers van lilo
 +
 +
Sinds donderdagmiddag is lilo6 door hardware problemen offline. Omdat dit de default linux login server was (lilo verwees naar lilo6) is dit voor veel gebruikers van lilo opvallend. De impact is beperkt, omdat er nog twee lilo's zijn, namelijk lilo5 en lilo7.
 +
Lilo7 is vervroegd de nieuwe lilo geworden, dus kun je een melding verwachten dat ssh een waarschuwing geeft over DNS SPOOFING, lilo7 heeft<br/> ECDSA <tt>SHA256:si3g2elo5m6TShx3PjX0+vF50pZ8NK/iXz/ESB+ZeP0</tt>
 +
[/nl]
 +
[en]
 +
  Begin        : 2021-02-25 17:30
 +
  End          : 2021-03-04 16:45
 +
  Affected      : users of lilo
 +
 +
As of Thursday afternoon, the lilo6 is down due to hardware issues. Because lilo6 was the default linux login server (lilo referred to lilo6), this affected many users of lilo. The impact is limited, because we have alternative lilo's, namely lilo5 and lilo7. As of March 1st lilo now refers to lilo7, ssh will warn about DNS SPOOFING, which is due to the difference host keys for lilo7 <br/> <tt>ECDSA SHA256:si3g2elo5m6TShx3PjX0+vF50pZ8NK/iXz/ESB+ZeP0</tt>
 +
[/en]
 +
 +
=== [Groot RU netwerkonderhoud zaterdag 27 februari 08:00-20:00][Major RU network maintenance Saturday Feb. 27 08:00-20:00] ===
 +
<itemTags>medewerkers,studenten</itemTags>
 +
[nl]
 +
  Begin        : 2021-02-27 08:00
 +
  Eind          : 2021-02-27 20:00
 +
  Getroffen    : gebruikers van het RU-netwerk of -diensten
 +
 +
ISC netwerkbeheer [https://www.ru.nl/systeem-meldingen/ kondigde aan] dat a.s. zaterdag 27 februari gepland groot onderhoud aan het RU-netwerk uitgevoerd zal worden, waardoor alle RU-diensten diverse keren maximaal een uur lang niet bereikbaar zullen zijn. Dit gaat om alle RU-diensten, inclusief die van FNWI/C&CZ: e-mail, VPN, wifi, BASS, OSIRIS, Brightspace, Syllabus+, Corsa, etc.
 +
[/nl]
 +
[en]
 +
  Begin        : 2021-02-27 08:00
 +
  End          : 2021-02-27 20:00
 +
  Affected      : users of the RU network or services
 +
 +
The ISC [https://www.ru.nl/systeem-meldingen/ announced] that Saturday February 27 08:00-20:00 major RU network maintenance work will be carried out. This will mean that all RU services will be unavailable several times for at most an hour. This concerns all RU services including those of FNWI/C&CZ: e-mail, VPN, wifi, BASS, OSIRIS, Brightspace, Syllabus+, Corsa, etc.
 +
[/en]
  
 
=== [DNS-problemen vanaf buiten met ru.nl][DNS problems from outside with ru.nl] ===
 
=== [DNS-problemen vanaf buiten met ru.nl][DNS problems from outside with ru.nl] ===
 
<itemTags>medewerkers,studenten,websites</itemTags>
 
<itemTags>medewerkers,studenten,websites</itemTags>
 
[nl]
 
[nl]
   Begin        : 2021-02-21
+
   Begin        : 2021-02-21 07:10
   Eind          :  
+
   Eind          : 2021-02-23 14:30
 
   Getroffen    : iedereen die iets in ru.nl wil benaderen van buiten de campus
 
   Getroffen    : iedereen die iets in ru.nl wil benaderen van buiten de campus
  
De externe DNS-servers van ru.nl werken niet goed, waardoor ook o.a. science.ru.nl soms niet gevonden kan worden: DNS-namen onder ru.nl resolven dan niet naar een IP-adres. We hebben enkele TTL's (time-to-lives) vergroot om te proberen de overlast iets minder te maken. Deze kleine TTL's waren bedoeld om in het geval van problemen met deze servers de service snel te kunnen verhuizen, maar dragen nu bij aan de overlast. Wie VPN gebruikt heeft na het starten van de VPN hier geen probleem meer mee, omdat dan de interne DNS-servers gebruikt worden. We hopen dat men de DNS-problemen met ru.nl snel kan verhelpen.
+
De centrale DNS-servers van ru.nl voor externe requests werkten niet goed doordat ze te veel bevraagd werden, waardoor ook o.a. science.ru.nl soms niet gevonden kon worden: DNS-namen onder ru.nl resolven dan niet naar een IP-adres. We hebben enkele TTL's (time-to-lives) vergroot om te proberen de overlast iets minder te maken. Deze kleine TTL's waren bedoeld om in het geval van problemen met deze servers de service snel te kunnen verhuizen, maar dragen nu bij aan de overlast. Wie VPN gebruikt heeft na het starten van de VPN hier geen probleem meer mee, omdat dan de interne DNS-servers gebruikt worden. Door aanpassingen aan de RU DNS-servers zijn hopelijk sinds 2021-02-23 14:30 de DNS-problemen minder of verdwenen.
 
[/nl]
 
[/nl]
 
[en]
 
[en]
   Begin        : 2021-02-21
+
   Begin        : 2021-02-21 07:10
   End          :
+
   End          : 2021-02-23 14:30 (?)
 
   Affected      : everyone trying to access something in ru.nl from off-campus.
 
   Affected      : everyone trying to access something in ru.nl from off-campus.
  
The external DNS servers for ru.nl have problems, which result in science.ru.nl a.o. not being found. DNS names within ru.nl then will resolve to an IP address. We enlarged some TTLs (Time-To+lives) to try to lessen the problem. These small TTLs were meant to be able to move a service to a new server in case of problems, but now they just make the problem bigger. After starting VPN you won't notice this problem, because the internal DNS servers that you use then are not affected. We hope the the administrators of ru.nl can fix this soon.
+
The central DNS servers for ru.nl for external requests had problems, because they received too many requests, which resulted in science.ru.nl and others not being found. DNS names within ru.nl then will not resolve to an IP address. We enlarged some TTLs (Time-To+lives) to try to lessen the problem. These small TTLs were meant to be able to move a service to a new server in case of problems, but now they just make the problem bigger. After starting VPN you won't notice this problem, because the internal DNS servers that you use then are not affected. Changes to the RU DNS servers hopefully lessened or removed the problems as of 2021-02-23 14:30.
 
[/en]
 
[/en]
  

Revision as of 11:53, 12 October 2021



Standard RU IT maintenance windows

The ISC announces the IT maintenance windows for the current academic year in time.

Report a problem

Use this form to report less urgent problems. For urgent problems, call 20000 (helpdesk).

Current Service Interruptions and Planned Maintenance

Windows 7 computers disabled in B-FAC domain

 Begin         : 2021-03-24
 End           : after upgrade to other OS
 Affected      : users of Windows 7 in the B-FAC domain

Because of security issues the last remaining Windows 7 machines wil be disabled, effective 24-03-2021, as member of the Active Directory Domain B-FAC. Please upgrade these computers to a more up-to-date OS. See also previous announcements on Windows 10 and the end of Windows 7.

EduroamCAT not working with Science accounts

 Begin         : 2019-02-28 00:00
 End           : ?
 Affected      : EduroamCAT users with Science accounts

EduroamCAT is the Eduroam Configuration Assistant Tool for many different devices. However, this hasn't (yet) been set up for the use of Science accounts (). C&CZ is looking for a solution. In the meantime Eduroam connections have to be configured manually (please consult www.ru.nl/wireless) or using the U/S/E number.

Recently Resolved Service Interruptions and Maintainance

To be quickly informed about service interruptions one can subscribe to the CPK mailinglist.

Server room network switch powerless

 Begin         : 2021-10-12 11:50
 End           : 2021-10-12 12:05
 Affected      : Users of one of the many servers behind this switch

Two modules of an important switch in the main C&CZ server room lost power during the preparation of planned maintenance. This disconnected ca. 75% of the servers in the room from the network. Moving the modules to new PDU's kimited the downtime to ca. 15 minutes.

License server problem

 Begin         : 2021-10-11 04:40
 End           : 2021-10-11 08:26
 Affected      : Users of one of the licenses of this server

An error in the management software prevented all license processes from starting correctly at the reboot of the license server. After fixing this error, all licenses were available again.

Fileserver 'flock' overloaded

 Begin         : 2021-09-17 14:30
 End           : 2021-09-17 15:30
 Affected      : Users of one of the 

Course software that had been tested caused an overload of the fileserver when it was used by 100 students. The performance of the fileserver was impaired for all users of network shares of this server.

VPN onbereikbaar

 Begin         : 2021-04-24
 End           : 2021-04-26 09:35
 Affected      : VPNsec users

A broken PDU has offlined a switch, which has caused the VPN server to be unreachable (and several other things, which don't affect users).

Central E-mail/Calendar disruption (exchange)

 Begin         : 2021-04-14    09:30
 Eind          : 2021-04-14    13:30
 Getroffen     : All users of Exchange (e-mail and calendar)

Due to an emergency maintenance, the central microsoft exchange server is unavailable for 4 hours. This may also affect systems that are dependent on exchange. E-mail and calendar functionality is expected to be restored when the maintenance is done around 13:30 Today.

Ceph problem

 Begin         : 2021-03-24 19:00
 End           : 2021-03-24 21:00
 Affected      : users with ceph based filesystems

During a routine upgrade of ceph, a bug in the latest version manifested itself and made the ceph manager unreachable. After aborting the upgrade and with help from the ceph-users mailinglist, everything became available again using a workaround.

Lilo7 restart

 Begin         : 2021-03-17 21:00
 End           : 2021-03-17 21:15
 Affected      : users of lilo

To change the network of lilo7, we need to reboot this loginserver. If you want a stable connection to a loginserver during this downtime, please use lilo6 or the soon to be taken down lilo5. For more info see the page on C&CZ loginservers.

Host of several virtual servers broken: Roundcube, websites and others

 Begin         : 2021-03-05 07:45
 End           : 2021-03-05 09:40
 Affected      : users of the virtual servers: Roundcube, websites with databases on this server, ...

Yesterday the SSD bootdisk of this VM host reported the first problems. This morning this had the effect of stopping all VMs running on this host. By moving the VMs to a different VM host, the problem has been solved. We will investigate how to best prevent this problem in the future or lessen its impact.

Lilo6 down

 Begin         : 2021-02-25 17:30
 End           : 2021-03-04 16:45
 Affected      : users of lilo

As of Thursday afternoon, the lilo6 is down due to hardware issues. Because lilo6 was the default linux login server (lilo referred to lilo6), this affected many users of lilo. The impact is limited, because we have alternative lilo's, namely lilo5 and lilo7. As of March 1st lilo now refers to lilo7, ssh will warn about DNS SPOOFING, which is due to the difference host keys for lilo7
ECDSA SHA256:si3g2elo5m6TShx3PjX0+vF50pZ8NK/iXz/ESB+ZeP0

Major RU network maintenance Saturday Feb. 27 08:00-20:00

 Begin         : 2021-02-27 08:00
 End           : 2021-02-27 20:00
 Affected      : users of the RU network or services

The ISC announced that Saturday February 27 08:00-20:00 major RU network maintenance work will be carried out. This will mean that all RU services will be unavailable several times for at most an hour. This concerns all RU services including those of FNWI/C&CZ: e-mail, VPN, wifi, BASS, OSIRIS, Brightspace, Syllabus+, Corsa, etc.

DNS problems from outside with ru.nl

 Begin         : 2021-02-21 07:10
 End           : 2021-02-23 14:30 (?)
 Affected      : everyone trying to access something in ru.nl from off-campus.

The central DNS servers for ru.nl for external requests had problems, because they received too many requests, which resulted in science.ru.nl and others not being found. DNS names within ru.nl then will not resolve to an IP address. We enlarged some TTLs (Time-To+lives) to try to lessen the problem. These small TTLs were meant to be able to move a service to a new server in case of problems, but now they just make the problem bigger. After starting VPN you won't notice this problem, because the internal DNS servers that you use then are not affected. Changes to the RU DNS servers hopefully lessened or removed the problems as of 2021-02-23 14:30.

DNS broken for subdomains of ru.nl

 Begin         : 2021-02-11 ~11:15
 End           : 2021-02-11 ~13:00
 Affected      : everyone trying to resolve *.science.ru.nl *.astro.ru.nl etc.

DNS-servers for ru.nl did not serve information about subdomains such as science.ru.nl. Thus no DNS-name will resolve to an IP address at FNWI. A workaround is to use as DNS servers: 131.174.224.4 en 8.8.8.8. If you try to connect to a service for the first time after ca 11:15, you'll get an error like: "No such domain" or "Cannot resolve". Restarting RU DNS servers at 12:45 may have fixed the problem. Without a real explanation, the problem went away after a few hours.

Gitlab upgrade

  Begin         : 2021-02-07  04:00
  End           : 2021-02-07  12:50
  Affected      : GitLab and Mattermost users

Services will not be available for a while because of a GitLab and Mattermost upgrade.

Science VPNsec disruption

 Begin         : 2021-02-03 13:00
 End           : 2021-02-03 14:02 (for Apple macOS/iOS last fix on February 10)
 Affected      : Users of Science VPN

The expiration date of the certificate of our VPNsec service was apparently not yet checked regularly. This made it possible for the certificate to expire. We put a new certificate into place within an hour. Of course we will check this certificate regularly from now on. For Apple/Mac we needed to construct a new mobileconfig, this took some time, because in the meantime RU had moved to a different Certificate Authority. For Apple macOS this was ready at the end of Feb. 4, with a new installation procedure. For Apple iOS (iPhone/iPad) the old profile has to be deleted and a new mobileconfig has to be installed.

DIY temporarily not usable

  Begin         : 2021-01-25 07:15
  End           : 2021-01-25 07:45
  Affected      : Users wanting to manage their science account

Due to a management operation (planned around this time), the DIY website was unusable. Since the time was very early, it's expected nobody was inconvenienced by this temporary unavailability.

Science smtp service temporarily not usable

  Begin         : 2021-01-22 10:00
  End           : 2021-01-22 10:30
  Affected      : Science mail users wanting to send mail

A configuration change unwantedly made the smtp service unusable. When we noticed this, it was repaired immediately.

Very long mail aliases temporarily not usable

  Begin         : 2021-01-21 15:52
  End           : 2021-01-22 09:55
  Affected      : Science mail aliases of more than 1024 characters

A configuration change had as unwanted effect the disappearance of all very long mail aliases. When this was reported next morning, it was repaired immediately.

Switch crash; gitlab+mattermost, licenses and DHZ

  Begin         : 2021-01-07 ~14:30
  End           : 2021-01-07 ~15:00
  Affected      : GitLab and Mattermost users, Licenses, DHZ (diy)

Due to a simple management command the switch (as-ak008-04) crasht and had to be reset manually. The switch sits between the network and servers for gitlab+mattermost, licenses and the database for DHZ(diy).

Gitlab upgrade

  Begin         : 2020-11-27  04:00
  End           : 2020-11-27 ~08:00
  Affected      : GitLab and Mattermost users (including PEP)

Services will not be available for a while because of a GitLab and Mattermost upgrade.

Eduroam problem on campus

 Begin         : 2020-07-10 evenng
 End           : 2020-07-10 evening
 Affected      : Eduroam users on campus

The ISC announced: For security reasons, the certificate of the wifi server will be replaced in the evening of Friday, July 10. This has consequences for connecting your mobile device to Eduroam when you’re on campus:

• If you get the message that you have to accept the new certificate to use eduroam, choose 'yes'. You can then use eduroam again;

• If you don't get this message and can't connect to Eduroam, choose the wireless network 'eduroam-config'. Accept the terms and conditions. Follow the instructions to reinstall Eduroam.

More information can also be found at www.ru.nl/ict-uk/eduroam (you will need an internet connection for this).

If you have any questions, please contact the ICT Helpdesk (024 - 36 22222).

RU mail erroneously in Spam folder

 Begin         : 2020-03-25 17:52
 End           : 2020-07-07 13:13
 Affected      : FNWI employees with Science mail

March 25, a rule "2020 Radboud Universiteit" was added to the Science spamfilter. Recently, this matched RU-central mailings. Therefore RU-wide mailings from e.g. the RU Board and Radboud Recharge have erroneously been delivered in the Spam folder of Science emplyees. The Science spamfilter tries to fight spam and phishing, this is partly manual work in which errors can't be excluded. C&CZ apologizes for the inconvenience this has caused.

Webserver 'havik' offline

 Begin         : 2020-06-18 15:45
 End           : 2020-06-18 16:25
 Affected      : Users of various websites.

Several parts have been replaced. We assume the problem, that occurred twice, is now resolved. For dual-boot pcs, the boot menu was served by an alternative method during the repair.

Science radius disruption

 Begin : 2020-06-17 11:11
 End   : 2020-06-17 11:56

Affected : Users of Science VPN and Eduroam based on science account

The certificate of the LDAP servers has been replaced this morning, this has also changed the certificate chain. The radius server uses LDAP as authentication backend and in the radius configuration the certificate chain had to be changed too. This was initially overlooked. Radius is the authentication mechanism used by all VPN servers and Eduroam

Webserver 'havik' offline

 Begin         : 2020-06-17 03:38
 End           : 2020-06-17 08:52
 Affected      : Users of dual boot PC's (the dual-boot menu is served by a website) and various websites.

The server went down in the same way as the previous time (3rd of June 2020). The cause is most likely a system board problem. This part will be replaced Tomorrow by a support engineer.

Webserver 'havik' offline

 Begin         : 2020-06-03 06:30
 End           : 2020-06-03 10:12
 Affected      : Users of dual boot PC's (the dual-boot menu is served

by a website) and various websites.

The server couldn't be reached after the scheduled weekly reboot, not even on its management interface. Because also C&CZ employees work from home and the interruption didn't get enough urgency fast enough, the interruption lasted too long, apologies for that. The support partner has been contacted and the server has been updated, but the origin of the problem is still unclear. We will also look at making these services more redundant or more easily movable to a different server.

CN00 Slurm master ubuntu 16.04 down

 Begin         : 2020-05-18 09:50
 End           : 2020-05-19 12:15
 Affected      : slurm on ubuntu 16.04 (cn07)

Due to a failed BIOS upgrade, the hardware of the database server appears to be bricked. We transfered the disks to another machine (cn00) and all database services are now up again, at the cost of not having cn00 running. When the hardware is working well again, we will swap it all back and restore the original situation.

Sperwer Database server storing

 Begin         : 2020-05-18 06:30
 End           : 2020-05-18-10:00
 Affected      : various websites and slurm

Due to a failed BIOS upgrade, the hardware of the database server appears to be bricked. We transfered the disks to another machine (cn00) and all database services are now up again, at the cost of not having cn00 running. When the hardware is working well again, we will swap it all back and restore the original situation.

Update May 19th, 12:15 : hardware fixed, situation back to the original state.

Science VPN disruption

 Begin         : 2020-05-06 05:00
 End           : 2020-05-06 08:00
 Affected      : Users of Science VPN

Unexplained crashes starting around 5am on the host system. System offline at around 6am. After a hard reset around 08:00, all seems to be all right again.

Science datacenter network problem

 Begin         : 2020-04-30 12:08
 End           : 2020-04-30 21:44
 Affected      : Users of Ceph storage and a few new compute clusternodes

A broken transceiver caused flapping af a 100 Gb/s connection between two C&CZ datacenters. Hours later the flapping increased, which took down the complete redundant new connection between the two server rooms. When this was noticed, a workarpound was found quickly by shutting down the interface with the broken transceiver. With this the connection was restored. De broken transceiver has been replaced thanks to a swift action from our supplier. Now we have these spare parts ready to use. We asked the supplier whether a configuration change will make the connection more redundant, that just one broken transceiver will not take down the connection.

Jitsi.science.ru.nl not working properly

 Begin         : 2020-04-19 15:00
 End           : 2020-04-20 11:40
 Affected      : Users of jitsi.science.ru.nl

Due to performance tuning having gone wrong the jitsi.science.ru.nl conference rooms cannot be joined by more than one person at the moment. Solved by reinstalling server.

Mailserver certificate problem

 Begin         : 2020-04-13 14:00
 End           : 2020-04-13 14:35
 Affected      : Users Science mail

The new certificate of the Science mailserver hadn't yet been placed in the right place. The expiration of the old certificate caused a problem for Science mail users, that was resolved by replacing the old certificate.

Problems with a virtual host

 Begin         : 2020-02-18 05:30
 End           : 2020-02-18 09:08
 Affected      : Users of mx3, smtp3, crestron, gitlab (PEP), goudsmit, msql01 and labservanttestvm.

The virtual machine host 'oscar' could not boot. Again, a broken LVM snapshot caused the problem.

Archived service interruptions can be found in the service interruptions archive.