| Date and Time
|
Notice
|
|
2009-11-20 20:17:37
|
Latency on 21CN network (59087) - UPDATE
This is an update the the post we made last weekend about ping spikes and increased latency across our 21CN (ADSL2+) platform.
We are continuing to investigate reports from customers who are noticing increased latency at certain times of the day. This is most noticeable for those that use their connection for latency-sensitive activities like gaming and VoIP applications.
At the advise of our hardware vendor we carried out some changes earlier in the week that we hoped would resolve the problems - http://usertools.plus.net/status/archive/1258554008.htm
Unfortunately these changes were unsuccessful and had to be reversed. We have provided our vendor with further diagnostic information and are continuing to work alongside them to arrive at a solution to the problem.
We would like to say sorry to those affected and will provide another update next week.
Kind Regards,
Bob Pullen
Customer Support
|
|
2009-11-20 19:13:11
|
Broadband Network Capacity (285660) - UPDATE
NB: This announcement was posted earlier this evening however one of the links contained in the update was wrong. This is the revised version containing the correct link.
This is an update to Wednesday's post about the ongoing speed and performance problems some of our customers are experiencing when the network is busy.
Over recent days our network engineers have been re-cabling parts of the network at one of our London data centres. This has made it easier to balance customer traffic across the platform, and has allowed us to perform some maintenance this afternoon on the problematic gateway pcl-ag02 - http://usertools.plus.net/status/archive/1258728887.htm
We have now finished this maintenance but no customers were disconnected. This means that we will no longer need to perform any network balancing ahead of the weekend.
As things are relatively well balanced already, we have taken the decision to return to a 'Plan A' operation - http://www.plus.net/support/broadband/quality_broadband/load.shtml
This should result in a noticeable improvement in speeds during the evening. We will remain operating under 'Plan A' over the weekend and will review the situation again on Monday when we will provide another update.
There is still the potential for your connection to become slower than normal when things get busy, however we have further maintenance work and fixes that are due to be rolled out over the coming weeks to address these issues.
Included in this work are plans to:
- Work with our vendors to develop and install a software patch to improve the stability of our edge routers (gateways).
- Upgrade the line cards on the edge routers to more modern versions.
- Change the 'virtual routing' of more customers to better balance traffic across our core network.
- Increase the capacity of the edge routers
- Upgrade our load balancers to reduce the potential for occasional DNS lookup failures
- Light more central capacity
On a related note we published a summary of the recent problems to our website earlier today that you can see here - http://www.plus.net/support/network_update.shtml
We would like to apologise again for the problems over recent weeks, and will provide another update on Monday.
Kind Regards,
Bob Pullen
Customer Support
|
|
2009-11-20 16:56:57
|
Broadband Network Capacity (285660) - UPDATE
This is an update to Wednesday's post about the ongoing speed and performance problems some of our customers are experiencing when the network is busy.
Over recent days our network engineers have been recabling parts of the network at one of our London data centres. This has made it easier to balance customer traffic across the platform, and has allowed us to perform some maintenance this afternoon on the problematic gateway pcl-ag02 - http://usertools.plus.net/status/archive/1258728887.htm
We have now finished this maintenance but no customers were disconnected. This means that we will no longer need to perform any network balancing ahead of the weekend.
As things are relatively well balanced already, we have taken the decision to return to a 'Plan A' operation - http://www.plus.net/support/broadband/quality_broadband/load.shtml
This should result in a noticeable improvement in speeds during the evening. We will remain operating under 'Plan A' over the weekend and will review the situation again on Monday when we will provide another update.
There is still the potential for your connection to become slower than normal when things get busy, however we have further maintenance work and fixes that are due to be rolled out over the coming weeks to address these issues.
Included in this work are plans to:
- Work with our vendors to develop and install a software patch to improve the stability of our edge routers (gateways).
- Upgrade the line cards on the edge routers to more modern versions.
- Change the 'virtual routing' of more customers to better balance traffic across our core network.
- Increase the capacity of the edge routers
- Upgrade our load balancers to reduce the potential for occasional DNS lookup failures
- Light more central capacity
On a related note we published a summary of the recent problems to our website earlier today that you can see here - http://www.plus.net/support/broadband/quality_broadband/load.shtml
We would like to apologise again for the problems over recent weeks, and will provide another update on Monday.
Kind Regards,
Bob Pullen
Customer Support
|
|
2009-11-20 14:54:47
|
Broadband Network Balancing - Friday 20th November 3.00pm-5.00pm
When's this work happening?
This afternoon, 20th November.
What does it affect?
Broadband Connectivity.
How long will it take?
Up to two hours.
What does the work involve?
We need to disconnect all broadband customers connected to the gateway known as pcl-ag02 because of a problem that's stopping any more customers from connecting to it. Once this is done we will be disconnecting a number of customers from other gateways to help bring the network into balance ahead of the weekend.
Is it likely to cause problems?
Some broadband customers will be briefly disconnected from the Internet at some point during the maintenance window. This may happen more than once in some cases. Anybody disconnected should be able to reconnect almost immediately.
Is there anything else I need to know?
Any customers having problems connecting once the maintenance is complete are advised to switch off their equipment for 30 minutes before attempting to reconnect.
Kind Regards,
Bob Pullen
Customer Support
|
|
2009-11-20 14:30:40
|
Adverse Weather Conditions (SER018/09) - NEW
Due to severe weather in parts of the UK, we have received notice that Openreach are declaring an MBORC (Matters Beyond Our Reasonable Control) across some areas of the UK.
This declaration is effective from Tuesday 17th November and may apply to faults on both broadband and talk services caused by the severe weather in the following areas:
Cumbria:
Cleator Moor
Harrington
Workington
If you are in one of the affected areas, please bear in mind that your fault resolution may be delayed due to this notice.
We will provide an update as we receive more information on this.
Kind Regards,
Matt Taylor
Customer Support
|
|
2009-11-20 09:41:55
|
Planned DNS Server Maintenance - Friday 20th November 10.00am-11.00am
When's this work happening?
This morning, 20th November.
What does it affect?
One of our old authoritative DNS servers.
How long will it take?
About fifteen minutes.
What does the work involve?
We will be relocating one of our DNS servers to another rack in one of our London data centres to allow for expansion of our broadband platform.
Is it likely to cause problems?
There will be a blip in name resolution for any services that still use this server. Some customers will encounter a 'temporary fault' error when trying to log into our website.
Is there anything else I need to know?
No.
Kind Regards,
Bob Pullen
Customer Support
|
|
2009-11-19 10:17:56
|
Fault Checker Errors (59331) - RESOLVED
This is an update to yesterday's post regarding problems when using our Broadband Faults Checker.
We completed some work overnight to move some of the servers which host our internal systems and also introduced some new servers into the cluster. Following this work the Broadband Faults Checker is again working correctly.
Sorry if this caused you any inconvenience.
Kind Regards,
Chris Parr
Customer Support
|
|
2009-11-18 17:26:35
|
Fault Checker Errors (59331) - NEW
We're currently experiencing problems with our Broadband Faults Checker. When attempting to raise a fault through this system customers may be seeing error messages during the process. This is also affecting our Customer Support Centre when raising new faults. This is not affecting the investigation of already open faults.
If you experience a fault before we resolve this issue, please either call our support centre or raise a standard support ticket. Where possible we would ask if you can refrain from raising the fault until tomorrow when the automated faults checker will be available again.
Sorry for any inconvenience this may be causing.
Kind Regards,
Chris Parr
Customer Support
|
|
2009-11-18 16:45:45
|
Planned Server Maintenance - Thursday 19th November 11.00am-5.00pm
When's this work happening?
Tomorrow, 19th November.
What does it affect?
The server that processes hardware orders.
How long will it take?
About three hours.
What does the work involve?
We will be moving the server that we use for hardware processing to a new data centre in Sheffield.
Is it likely to cause problems?
No it shouldn't do. It will delay the processing of hardware orders for a few hours but this won't have any adverse affect on customers.
Is there anything else I need to know?
No.
Kind Regards,
Bob Pullen
Customer Support
|
|
2009-11-18 15:48:03
|
Broadband Network Capacity (285660) - UPDATE
This is an update to last week's post about the ongoing speed and performance problems some of our customers continue to suffer, especially during the evenings when the network is at its busiest.
Last Friday the network was performing notably better following some changes our engineers had made earlier in the week. Unfortunately there has been a degradation in performance since then, particularly due to some unforeseen problems we encountered over the weekend and towards the start of this week.
We mentioned in the last announcement that we expected the network to be very busy last Sunday and Monday evening - http://usertools.plus.net/status/archive/1258133233.htm
Due to this, and because of some of the problems that we're yet to fix, we switched into a 'Plan B' network configuration - http://www.plus.net/support/broadband/quality_broadband/load.shtml
The network was busy on Sunday but was actually a lot quieter than we anticipated. Despite this we had a problem late Saturday afternoon that caused all customers connected to pcl-ag03 to be disconnected from the Internet. Whilst these customers were able to reconnect almost immediately, anybody connecting back on to the same gateway would not have been able to browse web pages due to data transfer problems. We didn't become fully aware of this problem until Sunday afternoon at which point we took the decision to perform the following work which would have resulted in more disconnections - http://usertools.plus.net/status/archive/1258297848.htm
Unfortunately, despite our efforts, this only fixed things for some customers and by early Monday morning we were still receiving reports of problems. At this point we stopped any more customers from connecting to the broken gateway and carried out a software update - this fully resolved the problem but resulted in further disconnections for anybody connected to pcl-ag03 - http://usertools.plus.net/status/archive/1258384040.htm
pcl-ag03 is now fully working and is being brought back into balance with the rest of the network.
On Monday afternoon a slightly different problem was identified with another gateway on the network (pcl-ag02). No customers were disconnected this time round and there weren't any data transfer problems, however the gateway stopped accepting any new sessions causing it to drop out of balance as customers began to disconnect or 'churn' from it. We've put a temporary fix in place this afternoon and are continuing to work with our hardware vendor to identify the underlying cause.
Because of the above problems and the imbalance of the network, we remained in 'Plan B' operation last night and will continue to do so until further notice. This *will* result in degraded performance but should ensure that the quality of web browsing is not significantly affected. Some streaming services may be impacted by this dependent on the account type you are subscribed to.
We are aware that there are customers still reporting problems where web pages and other Internet resources fail to connect but are immediately accessible seconds later after trying again. We're still of the opinion that this is due to congestion on some of our core network links, and we have engineers en route to London to recable and reconfigure parts of the network. We will post further details about these changes nearer the time.
Thanks once more for your patience and sorry about the continued inconvenience, we'll provide another update before the end of the week.
Kind Regards,
Bob Pullen
Customer Support
|
|
2009-11-18 15:46:31
|
Adverse Weather Conditions (289234) - NEW
Due to severe weather in parts of the UK, we have received notice that Openreach are declaring an MBORC (Matters Beyond Our Reasonable Control) across some areas of the UK.
This declaration is effective from Tuesday 17th November and may apply to faults caused by the severe weather in the following areas:
Wales and West
SALISBURY
BOURNEMOUTH
TAUNTON
EXETER
PLYMOUTH
TRURO
NORTH WALES
WEST WALES
South East
CANTERBURY
TUNBRIDGE WELLS
BRIGHTON
PORTSMOUTH
ISLE OF WIGHT
CHELMSFORD
If you are in one of the affected areas, please bear in mind that your fault resolution may be delayed due to this notice.
We will provide an update as we receive more information on this.
Kind Regards,
Chris Parr
Customer Support
|
|
2009-11-18 15:45:18
|
Planned Internal System Maintenance - Thursday 19th November Midnight-2.00am
When's this work happening?
Tomorrow morning, 19th November.
What does it affect?
Our website.
How long will it take?
Up to two hours.
What does the work involve?
We will be moving some of the servers that host our internal systems and CRM tools to a new data centre in Sheffield.
Is it likely to cause problems?
No it shouldn't do,although it is worth noting that this system is the one our Customer Support Centre use to access customers' account details.
Is there anything else I need to know?
This is an update to the earlier post on the same maintenance work as we have delayed part of the work. This means that there will be *no* holding pages in place on our website and you will be able to access your account as normal.
Kind Regards,
Chris Parr
Customer Support
|
|
2009-11-18 14:20:08
|
Planned Router Maintenance (21CN/ADSL2+) - Wednesday 18th November 2.30pm-5.30pm
When's this work happening?
This afternoon, 18th November.
What does it affect?
Traffic shaping on our 'Up to 20Mbps' 21CN ADSL2+ network.
How long will it take?
A couple of hours.
What does the work involve?
One of our hardware vendors has proposed that we make some configuration changes to some of our edge routers to try and fix a recently identified problem causing intermittent latency spikes on our ADSL2+ network.
Is it likely to cause problems?
It shouldn't cause any noticeable problems although there might be a five minute period during which 21CN customers will only be able to connect to the gateway known as pcl-ag04 (you can normally connect to thn-ag2 as well). There is a slim possibility that the change could negatively affect some customers' traffic management profiles. If this does happen we'll announce further details during the time.
Is there anything else I need to know?
We're only applying these changes to thn-ag2 for the time being. We will then monitor the platform overnight and if considered successful, will apply the same change to pcl-ag04.
Kind Regards,
Bob Pullen
Customer Support
|
|
2009-11-18 13:33:39
|
Planned Internal System Maintenance - Thursday 19th November 0.01am-6.00am
When's this work happening?
Tomorrow morning, 19th November.
What does it affect?
Our website and internal systems
How long will it take?
Up to two hours.
What does the work involve?
We're making some alterations to some internal systems. During this we need to apply holding pages across our websites and the system used to access customers' account details.
Is it likely to cause problems?
Yes, between 3.30am and 5.30am our website will not be accessible as there will be holding pages in place. Our Customer Support Centre will also be unable to access customers' account details during this time frame.
Is there anything else I need to know?
No.
Kind Regards,
Chris Parr
Customer Support
|
|
2009-11-16 21:56:43
|
Data Transfer and Web Browsing problems (59297) - UPDATE
This is an update to the posts we made earlier today about problems with data transfer on the gateway known as pcl-ag03.
Our engineers have re-routed a third of the traffic destined for pcl-ag03 to a different gateway (pcl-ag04). Earlier this afternoon the software on the problematic gateway was also downgraded to a previous version (the same as that running on some of the other gateways on the network). Since carrying out this work, we have been moitoring traffic flows and now believe the problem to be resolved.
Please accept our apologies for any disruption to your service this problem may have caused.
Kind Regards,
Bob Pullen
Customer Support
|
|
2009-11-16 17:14:54
|
Latency on 21CN network (59087) - NEW
Our network engineers are currently investigating intermittent ping spikes and increased latency across the 21CN platform. Alongside our own investigations we are liaising with one of our suppliers.
We are currently testing a possible solution to this issue and will provide a further update later on this week.
Kind Regards,
Matt Taylor
Customer Support
|
|
2009-11-16 15:07:20
|
Emergency Router Maintenance - Monday 16th November 3.15pm-5.00pm
When's this work happening?
This afternoon.
What does it affect?
Broadband Internet connectivity.
How long will it take?
Up to two hours.
What does the work involve?
At the advice of our vendor we are going to be downgrading the software installed on the gateway known as pcl-ag03. This is an attempt to resolve the recent problems with data transfer and Internet connectivity on this gateway.
Is it likely to cause problems?
Yes, whilst we are no longer accepting new connections on this gateway, there are still a number of customers connected to it. These customers will be disconnected from the Internet but should be able to reconnect almost immediately.
Is there anything else I need to know?
Any customers having problems connecting once the maintenance is complete are advised to switch off their equipment for 30 minutes before attempting to reconnect.
Kind Regards,
Bob Pullen
Customer Support
|
|
2009-11-16 15:06:24
|
Data Transfer and Web Browsing problems (59297) - UPDATE
This is an update to the post we made earlier today about problems with data transfer on the gateway known as pcl-ag03.
Some customers connected to this gateway may be unable to browse web pages or access any other Internet resources.
We made a configuration change earlier this afternoon so that no new sessions are being established on the problematic gateway. This prevents the problem from affecting any more customers.
At the advise of our vendor we will shortly be undertaking some engineering works. Details about this work will be posted in a separate announcement.
We will provide another update once the maintenance is complete.
Sorry about the continued inconvenience.
Kind Regards,
Bob Pullen
Customer Support
|
|
2009-11-16 13:02:32
|
Planned Internal Systems Maintenance - Tuesday 17th November 9.00am-5.00pm
When's this work happening?
Tomorrow, Tuesday 17th November.
What does it affect?
One of our internal systems.
How long will it take?
All day.
What does the work involve?
We will be moving some of the servers that host our internal systems and CRM tools to a new data centre in Sheffield.
Is it likely to cause problems?
No it shouldn't do,although it is worth noting that this system is the one our Customer Support Centre use to access customers' account details.
Is there anything else I need to know?
No.
Kind Regards,
Bob Pullen
Customer Support
|
|
2009-11-16 12:16:03
|
Planned Portal Server Maintenance - Wednesday 18th November 9.00am-5.00pm
When's this work happening?
This Wednesday, 18th November.
What does it affect?
The servers that host our website.
How long will it take?
All day.
What does the work involve?
We will be moving some of our portal servers to a new data centre in Sheffield.
Is it likely to cause problems?
No it shouldn't do. We've multiple portal servers and the platform is capable of operating normally with a few servers removed from service.
Is there anything else I need to know?
No.
Kind Regards,
Bob Pullen
Customer Support
|
