Kategori: Driftinfo

Vid 23:45 17/2 gick ett linjekort på en switch sönder. Switchen stängdes av. Inga tjänster påverkades då switchen är speglad.
04:30 18/2 påbörjades utbyte av switch, vi passade på att uppgradera till ny infrastruktur i noden där switch byttes ut.

Detta krävde kortade nedtid för vissa kunder för nätverksförändringar. Några få kunder fick avbrott som var längre men dock inom SLA-villkor.
Vid 06:30 var allt arbete slutfört och noden har uppgraderats till ny generation av switchar som medför kraftigt ökad kapacitet.

From roughly 18.20 a customer of ours was target of a DDOS.
We started mitigating the attack immediately and around 18.50 most reachability was restored, although with service degradation.

We’re working on mitigating remaining traffic in this DDOS.
Impact: Packetloss / network unreachable

When: Sunday 12.00 – 18.00 on 2021-06-20
Why: Datacenter facility provider maintenance
Affects: Hammarby Datacenter, Interconnect between Hammarby DC and other datacenters
as well as other services on 46.253.192.0/20 hosting range

Details:
Our datacenter provider GlobalConnect are performing structural integrity improvements in the facility. This requires us to move equipment in the zone we’re located in. We’ll be doing this with ”construction electricity” to avoid powering down the equipment.

We need to move some uplink cables, which may cause momentary interruptions.
Our intention is that the disruptions will only last minutes at a time, but we’re still announcing a service window. Please plan your activities and services according this this service window.

We had an issue with a core switch for our bladechassis that did not failover as it should following a fault.
Switch was still reporting functional and secondary switches did not take over as they should. This caused delay in troubleshooting process.

Effected datacenters: VBDC , interconnection between VBDC and HY
When: 12.55 CET – 13.41 CET
Component: dr-se-vb-sw02

We’ll perform software updates during night time to prevent any software causes if this issue.
If it still happens again, we’ll replace the switch as faulty.

Adminor is scheduling a router maintenance effecting customers on IPranges: 46.253.192.0/20 and 185.111.240.0/22 .

Primary Service window is 2021-03-22 Monday 00.00 – 04.00 (two week notice!).

We are aiming for an outage less than 30min (15min or less usually), but keeping the window longer incase we hit an unforeseen snag or hardware issues that needs to be resolved.

Emergency service window (if we need to schedule sooner) 2021-03-15 Monday 00.00 – 04.00 CET.

We will try to keep this maintenance brief but are letting you know as our valued customer.
 
Best regards
Adminor AB

CloudFlare are having issues with their DNS services.
This may impact sites and services globally.

We’re starting to see some of their services returning to normal but we still caution that there may be more impacts.

More information at https://www.cloudflarestatus.com/

One of our network upstreams has announced network maintenance.
Please note that routing changes may cause momentary traffic dip during this time frame.

Our own traffic should automatically failover to secondary ISPs.


Date and time for maintenance window
Start date and time: 2020-05-05 03:00 UTC
End date and time: 2020-05-05 04:00 UTC

At 04.30 we performed a reboot of one of our core routers to ensure the issue is not on our end.

We will reboot some other equipment to troubleshoot issues.
We apologize for the disruption.

Update: One core switch was also rebooted manually at 05.50 – 05.52 . (short dip)

We have made adjustments to network traffic to find a stopgap solution to flapping network connection.

One of our upstream transits suffered an outage at 19.05. This caused network re-routing to happen and may have been felt as momentary network dip.
Traffic has been re-routed to failovers. We’re monitoring the networks.

When: 19:05 – 19:10 (upstream has failed over)
Impact: 5min re-routing, traffic dip
What: 46.253.192.0/20

Update: Some more network dips occured during the evening. The network upstream suspected of causing high CPU usage in our cores have been isolated. We’re continuing to monitor the situation.

When: 20.30, 23.30 , duration (roughly 2-3min per incident)

At 22.00 we experienced some routing issues. We are currently investigating the cause of this.

Cause: One of our Upstream Transit lost BGP connectivity / flapped
Region: Stockholm
Impact: 5 – 10min routing table rebuilds

Update (22.30): One of our transit operators having issues. Recovered after a few minutes.