Recente Storingen

From Cncz
Jump to navigation Jump to search

Standard RU IT maintenance windows

The ISC announces the IT maintenance windows for the current academic year in time.

Report a problem

Use this form to report less urgent problems. For urgent problems, call 20000 (helpdesk).

Current Service Interruptions and Planned Maintenance

Recently Resolved Service Interruptions and Maintainance

To be quickly informed about service interruptions one can subscribe to the CPK mailinglist.

Mail problems because of spam run after phishing

 Begin         : 2022-09-29 19:27
 End           : 2022-09-30
 Affected      : some mail couldn't be delivered

Some time ago, some Science users fell for a phishing mail, therefore their account could be misused by spammers to send mail through our mailservers. Therefore our mailservers ended up in blacklists yesterday evening. Some mails could therefore not be delivered to other servers, that mail was returned to the sender. This morning we removed the IP addresses of our servers from several blacklists, we think that the problems are resolved now.

Friday September 23 18:30-22:30 RU firewall maintenance

 Begin         : 2022-09-23 18:30
 End           : 2022-09-23 22:30
 Affected      : connection campus/internet (18:30-20:30) and central RU-servers (19:30-22:30)

ILS network management announced: Friday September 23 18.30 – 22.30 network maintenance is carried out on the DMZ and SRV firewalls. Connectivity interruptions of 20 to 40 seconds are expected. In case of problems, interruptions of 5 to 10 minutes might occur.

Eduroam: Tuesday September 27 23:30 1 hour down for maintenance

 Begin         : 2022-09-27 23:30
 End           : 2022-09-28 00:30
 Affected      : Eduroam wireless network in FNWI buildings a.o.

ILS network management announced: Tuesday September 27 23:30 maintenance is carried out on the Eduroam wireless network in FNWI buildings a.o. In most cases connectivity will be interrupted for at least 30 and at most 60 minutes.

Network switches failed after a power failure on campus

 Begin:     2022-09-22 12:20
 End:       2022-09-22 18:30
 Affected:  Users of the network and phones

A few campus-wide dips in the power supply made network and telephony unusable for about 15 minutes, except in server rooms where UPSs guarantee the power supply. Because 7 modules of desktop switches in Huygens/HFML/Felix had become defective, several hundred network outlets will only have worked again after the relevant switch module had been replaced.

Network failure datacenter

 Begin         : 2022-09-04 16:00
 End           : 2022-09-05 11:00
 Affected      : many links of FNWI servers

Still unclear what the cause is, several links are strongly degraded. Update: A defective switch has been replaced.

Saturday May 14 adjacent buildings (Mercator, Proeftuin, Logistiek) 5 minutes without network

 Begin         : 2022-05-14 09:00
 End           : 2022-05-14 10:00
 Affected      : all network outlets in Mercator, Proeftuin and Logistiek Centrum will be down for max. 5 minutes

RU/ILS network management will switch to new hardware. This will lead to a network interruption of at most 5 minutes.

Coma, coma01 and coma46 network problem

 Begin         : 2022-05-03 13:47
 End           : 2022-05-03 14:55
 Affected      : Users of coma, coma01 en coma46

This afternoon three coma nodes lost their network because of an incorrect network configuration. They must have shown intermittent network problems earlier. It took us some time to find out what caused this network problem, but when found, it was easy to fix. DNS(SEC) service down

 Begin         : 2022-04-28 12:02
 End           : 2022-05-13 13:00
 Affected      : Users wanting to communicate with of from

During the regular change of DNSSEC keys that secure DNS traffic, an incorrect key was introduced in the external DNS of for This made disappear from the internet. This error was partly corrected 2022-05-02 at ca. 14:00 hours, but the automatic process used an not accepted encryption. It took ILS until 2022-05-22 to correct that by hand after we eventually noticed the error. Because the DNS answer that doesn't exist may be cached for 24 hours, the problem was not completely over until 2022-05-13 13:00. The main problems for users were that mail from an address bounced.

SUSE Linux 15.3 Eduroam doesn't work with U- or s-number, but does with Science account

 Begin         : 2022-02-14
 End           : ?
 Affected      : Eduroam users with SUSE Linux 15.3

February 14, ILS switched off antique versions of TLS (1.0 and 1.1) for the Eduroam authentication on ILS LDAP servers. From then on, SUSE Linux 15.3 clients can't authenticate with U- or s-number. They only have TLS1.2 and the ILS servers offer TLS1.3 first, after that an error occurs. By using the Science-account to authenticate, these users succeed in connecting to Eduroam.

Network switch of Astro Coma cluster down

 Begin         : 2022-02-22 13:10
 End           : 2022-02-22 15:47
 Affected      : Users of the Coma compute cluster

The network switch of the Coma cluster seems to be broken, all attached nodes are separated from the rest of the network. We'll replace the switch a.s.a.p. and (let) analyze the problem after that.

Interrupted link to new datacenter switches

 Begin         : 2021-12-15 12:45
 End           : 2021-12-15 13:42
 Affected      : all 25 gigabit connected machines (shares, websites, clusternodes)

Due to human error, the connection between the new datacenter switches and the central router was interrupted.

vmhost07 poweroff

 Begin         : 2021-12-02 13:10
 End           : 2021-12-02 13:20
 Affected      : Users of one of services mentioned

Vmhost07 was accidentally shut down. Cause: human error.


Ceph storage expansion caused performance issues

 Begin         : 2021-11-16
 End           : 2021-11-17
 Affected      : users of Ceph filesystems and websites on webvm01

As a result of the expansion of the Ceph storage cluster, the cluster had performance and availability issues. The problems were resolved this morning.

Server room network switch powerless

 Begin         : 2021-10-12 11:50
 End           : 2021-10-12 12:05
 Affected      : Users of one of the many servers behind this switch

Two modules of an important switch in the main C&CZ server room lost power during the preparation of planned maintenance. This disconnected ca. 75% of the servers in the room from the network. Moving the modules to new PDU's kimited the downtime to ca. 15 minutes.

License server problem

 Begin         : 2021-10-11 04:40
 End           : 2021-10-11 08:26
 Affected      : Users of one of the licenses of this server

An error in the management software prevented all license processes from starting correctly at the reboot of the license server. After fixing this error, all licenses were available again.

Fileserver 'flock' overloaded

 Begin         : 2021-09-17 14:30
 End           : 2021-09-17 15:30
 Affected      : Users of one of the 

Course software that had been tested caused an overload of the fileserver when it was used by 100 students. The performance of the fileserver was impaired for all users of network shares of this server.

VPN onbereikbaar

 Begin         : 2021-04-24
 End           : 2021-04-26 09:35
 Affected      : VPNsec users

A broken PDU has offlined a switch, which has caused the VPN server to be unreachable (and several other things, which don't affect users).

Central E-mail/Calendar disruption (exchange)

 Begin         : 2021-04-14    09:30
 Eind          : 2021-04-14    13:30
 Getroffen     : All users of Exchange (e-mail and calendar)

Due to an emergency maintenance, the central microsoft exchange server is unavailable for 4 hours. This may also affect systems that are dependent on exchange. E-mail and calendar functionality is expected to be restored when the maintenance is done around 13:30 Today.

Ceph problem

 Begin         : 2021-03-24 19:00
 End           : 2021-03-24 21:00
 Affected      : users with ceph based filesystems

During a routine upgrade of ceph, a bug in the latest version manifested itself and made the ceph manager unreachable. After aborting the upgrade and with help from the ceph-users mailinglist, everything became available again using a workaround.

Windows 7 computers disabled in B-FAC domain

 Begin         : 2021-03-24
 End           : after upgrade to other OS
 Affected      : users of Windows 7 in the B-FAC domain

Because of security issues the last remaining Windows 7 machines wil be disabled, effective 24-03-2021, as member of the Active Directory Domain B-FAC. Please upgrade these computers to a more up-to-date OS. See also previous announcements on Windows 10 and the end of Windows 7.

Lilo7 restart

 Begin         : 2021-03-17 21:00
 End           : 2021-03-17 21:15
 Affected      : users of lilo

To change the network of lilo7, we need to reboot this loginserver. If you want a stable connection to a loginserver during this downtime, please use lilo6 or the soon to be taken down lilo5. For more info see the page on C&CZ loginservers.

Host of several virtual servers broken: Roundcube, websites and others

 Begin         : 2021-03-05 07:45
 End           : 2021-03-05 09:40
 Affected      : users of the virtual servers: Roundcube, websites with databases on this server, ...

Yesterday the SSD bootdisk of this VM host reported the first problems. This morning this had the effect of stopping all VMs running on this host. By moving the VMs to a different VM host, the problem has been solved. We will investigate how to best prevent this problem in the future or lessen its impact.

Lilo6 down

 Begin         : 2021-02-25 17:30
 End           : 2021-03-04 16:45
 Affected      : users of lilo

As of Thursday afternoon, the lilo6 is down due to hardware issues. Because lilo6 was the default linux login server (lilo referred to lilo6), this affected many users of lilo. The impact is limited, because we have alternative lilo's, namely lilo5 and lilo7. As of March 1st lilo now refers to lilo7, ssh will warn about DNS SPOOFING, which is due to the difference host keys for lilo7
ECDSA SHA256:si3g2elo5m6TShx3PjX0+vF50pZ8NK/iXz/ESB+ZeP0

Major RU network maintenance Saturday Feb. 27 08:00-20:00

 Begin         : 2021-02-27 08:00
 End           : 2021-02-27 20:00
 Affected      : users of the RU network or services

The ISC announced that Saturday February 27 08:00-20:00 major RU network maintenance work will be carried out. This will mean that all RU services will be unavailable several times for at most an hour. This concerns all RU services including those of FNWI/C&CZ: e-mail, VPN, wifi, BASS, OSIRIS, Brightspace, Syllabus+, Corsa, etc.

DNS problems from outside with

 Begin         : 2021-02-21 07:10
 End           : 2021-02-23 14:30 (?)
 Affected      : everyone trying to access something in from off-campus.

The central DNS servers for for external requests had problems, because they received too many requests, which resulted in and others not being found. DNS names within then will not resolve to an IP address. We enlarged some TTLs (Time-To+lives) to try to lessen the problem. These small TTLs were meant to be able to move a service to a new server in case of problems, but now they just make the problem bigger. After starting VPN you won't notice this problem, because the internal DNS servers that you use then are not affected. Changes to the RU DNS servers hopefully lessened or removed the problems as of 2021-02-23 14:30.

DNS broken for subdomains of

 Begin         : 2021-02-11 ~11:15
 End           : 2021-02-11 ~13:00
 Affected      : everyone trying to resolve * * etc.

DNS-servers for did not serve information about subdomains such as Thus no DNS-name will resolve to an IP address at FNWI. A workaround is to use as DNS servers: en If you try to connect to a service for the first time after ca 11:15, you'll get an error like: "No such domain" or "Cannot resolve". Restarting RU DNS servers at 12:45 may have fixed the problem. Without a real explanation, the problem went away after a few hours.

Gitlab upgrade

  Begin         : 2021-02-07  04:00
  End           : 2021-02-07  12:50
  Affected      : GitLab and Mattermost users

Services will not be available for a while because of a GitLab and Mattermost upgrade.

Science VPNsec disruption

 Begin         : 2021-02-03 13:00
 End           : 2021-02-03 14:02 (for Apple macOS/iOS last fix on February 10)
 Affected      : Users of Science VPN

The expiration date of the certificate of our VPNsec service was apparently not yet checked regularly. This made it possible for the certificate to expire. We put a new certificate into place within an hour. Of course we will check this certificate regularly from now on. For Apple/Mac we needed to construct a new mobileconfig, this took some time, because in the meantime RU had moved to a different Certificate Authority. For Apple macOS this was ready at the end of Feb. 4, with a new installation procedure. For Apple iOS (iPhone/iPad) the old profile has to be deleted and a new mobileconfig has to be installed.

DIY temporarily not usable

  Begin         : 2021-01-25 07:15
  End           : 2021-01-25 07:45
  Affected      : Users wanting to manage their science account

Due to a management operation (planned around this time), the DIY website was unusable. Since the time was very early, it's expected nobody was inconvenienced by this temporary unavailability.

Science smtp service temporarily not usable

  Begin         : 2021-01-22 10:00
  End           : 2021-01-22 10:30
  Affected      : Science mail users wanting to send mail

A configuration change unwantedly made the smtp service unusable. When we noticed this, it was repaired immediately.

Very long mail aliases temporarily not usable

  Begin         : 2021-01-21 15:52
  End           : 2021-01-22 09:55
  Affected      : Science mail aliases of more than 1024 characters

A configuration change had as unwanted effect the disappearance of all very long mail aliases. When this was reported next morning, it was repaired immediately.

Switch crash; gitlab+mattermost, licenses and DHZ

  Begin         : 2021-01-07 ~14:30
  End           : 2021-01-07 ~15:00
  Affected      : GitLab and Mattermost users, Licenses, DHZ (diy)

Due to a simple management command the switch (as-ak008-04) crasht and had to be reset manually. The switch sits between the network and servers for gitlab+mattermost, licenses and the database for DHZ(diy).

Gitlab upgrade

  Begin         : 2020-11-27  04:00
  End           : 2020-11-27 ~08:00
  Affected      : GitLab and Mattermost users (including PEP)

Services will not be available for a while because of a GitLab and Mattermost upgrade.

Eduroam problem on campus

 Begin         : 2020-07-10 evenng
 End           : 2020-07-10 evening
 Affected      : Eduroam users on campus

The ISC announced: For security reasons, the certificate of the wifi server will be replaced in the evening of Friday, July 10. This has consequences for connecting your mobile device to Eduroam when you’re on campus:

• If you get the message that you have to accept the new certificate to use eduroam, choose 'yes'. You can then use eduroam again;

• If you don't get this message and can't connect to Eduroam, choose the wireless network 'eduroam-config'. Accept the terms and conditions. Follow the instructions to reinstall Eduroam.

More information can also be found at (you will need an internet connection for this).

If you have any questions, please contact the ICT Helpdesk (024 - 36 22222).

RU mail erroneously in Spam folder

 Begin         : 2020-03-25 17:52
 End           : 2020-07-07 13:13
 Affected      : FNWI employees with Science mail

March 25, a rule "2020 Radboud Universiteit" was added to the Science spamfilter. Recently, this matched RU-central mailings. Therefore RU-wide mailings from e.g. the RU Board and Radboud Recharge have erroneously been delivered in the Spam folder of Science emplyees. The Science spamfilter tries to fight spam and phishing, this is partly manual work in which errors can't be excluded. C&CZ apologizes for the inconvenience this has caused.

Webserver 'havik' offline

 Begin         : 2020-06-18 15:45
 End           : 2020-06-18 16:25
 Affected      : Users of various websites.

Several parts have been replaced. We assume the problem, that occurred twice, is now resolved. For dual-boot pcs, the boot menu was served by an alternative method during the repair.

Science radius disruption

 Begin : 2020-06-17 11:11
 End   : 2020-06-17 11:56

Affected : Users of Science VPN and Eduroam based on science account

The certificate of the LDAP servers has been replaced this morning, this has also changed the certificate chain. The radius server uses LDAP as authentication backend and in the radius configuration the certificate chain had to be changed too. This was initially overlooked. Radius is the authentication mechanism used by all VPN servers and Eduroam

Webserver 'havik' offline

 Begin         : 2020-06-17 03:38
 End           : 2020-06-17 08:52
 Affected      : Users of dual boot PC's (the dual-boot menu is served by a website) and various websites.

The server went down in the same way as the previous time (3rd of June 2020). The cause is most likely a system board problem. This part will be replaced Tomorrow by a support engineer.

Webserver 'havik' offline

 Begin         : 2020-06-03 06:30
 End           : 2020-06-03 10:12
 Affected      : Users of dual boot PC's (the dual-boot menu is served

by a website) and various websites.

The server couldn't be reached after the scheduled weekly reboot, not even on its management interface. Because also C&CZ employees work from home and the interruption didn't get enough urgency fast enough, the interruption lasted too long, apologies for that. The support partner has been contacted and the server has been updated, but the origin of the problem is still unclear. We will also look at making these services more redundant or more easily movable to a different server.

CN00 Slurm master ubuntu 16.04 down

 Begin         : 2020-05-18 09:50
 End           : 2020-05-19 12:15
 Affected      : slurm on ubuntu 16.04 (cn07)

Due to a failed BIOS upgrade, the hardware of the database server appears to be bricked. We transfered the disks to another machine (cn00) and all database services are now up again, at the cost of not having cn00 running. When the hardware is working well again, we will swap it all back and restore the original situation.

Sperwer Database server storing

 Begin         : 2020-05-18 06:30
 End           : 2020-05-18-10:00
 Affected      : various websites and slurm

Due to a failed BIOS upgrade, the hardware of the database server appears to be bricked. We transfered the disks to another machine (cn00) and all database services are now up again, at the cost of not having cn00 running. When the hardware is working well again, we will swap it all back and restore the original situation.

Update May 19th, 12:15 : hardware fixed, situation back to the original state.

Science VPN disruption

 Begin         : 2020-05-06 05:00
 End           : 2020-05-06 08:00
 Affected      : Users of Science VPN

Unexplained crashes starting around 5am on the host system. System offline at around 6am. After a hard reset around 08:00, all seems to be all right again.

Science datacenter network problem

 Begin         : 2020-04-30 12:08
 End           : 2020-04-30 21:44
 Affected      : Users of Ceph storage and a few new compute clusternodes

A broken transceiver caused flapping af a 100 Gb/s connection between two C&CZ datacenters. Hours later the flapping increased, which took down the complete redundant new connection between the two server rooms. When this was noticed, a workarpound was found quickly by shutting down the interface with the broken transceiver. With this the connection was restored. De broken transceiver has been replaced thanks to a swift action from our supplier. Now we have these spare parts ready to use. We asked the supplier whether a configuration change will make the connection more redundant, that just one broken transceiver will not take down the connection. not working properly

 Begin         : 2020-04-19 15:00
 End           : 2020-04-20 11:40
 Affected      : Users of

Due to performance tuning having gone wrong the conference rooms cannot be joined by more than one person at the moment. Solved by reinstalling server.

Mailserver certificate problem

 Begin         : 2020-04-13 14:00
 End           : 2020-04-13 14:35
 Affected      : Users Science mail

The new certificate of the Science mailserver hadn't yet been placed in the right place. The expiration of the old certificate caused a problem for Science mail users, that was resolved by replacing the old certificate.

Problems with a virtual host

 Begin         : 2020-02-18 05:30
 End           : 2020-02-18 09:08
 Affected      : Users of mx3, smtp3, crestron, gitlab (PEP), goudsmit, msql01 and labservanttestvm.

The virtual machine host 'oscar' could not boot. Again, a broken LVM snapshot caused the problem.

Archived service interruptions can be found in the service interruptions archive.