| Date and Time |
Notice |
|
20/11/2009 @ 19:13
|
Broadband Network Capacity (285660) - UPDATE
NB: This announcement was posted earlier this evening however one of the links contained in the update was wrong. This is the revised version containing the correct link.
This is an update to Wednesday's post about the ongoing speed and performance problems some of our customers are experiencing when the network is busy.
Over recent days our network engineers have been re-cabling parts of the network at one of our London data centres. This has made it easier to balance customer traffic across the platform, and has allowed us to perform some maintenance this afternoon on the problematic gateway pcl-ag02 - http://usertools.plus.net/status/archive/1258728887.htm
We have now finished this maintenance but no customers were disconnected. This means that we will no longer need to perform any network balancing ahead of the weekend.
As things are relatively well balanced already, we have taken the decision to return to a 'Plan A' operation - http://www.plus.net/support/broadband/quality_broadband/load.shtml
This should result in a noticeable improvement in speeds during the evening. We will remain operating under 'Plan A' over the weekend and will review the situation again on Monday when we will provide another update.
There is still the potential for your connection to become slower than normal when things get busy, however we have further maintenance work and fixes that are due to be rolled out over the coming weeks to address these issues.
Included in this work are plans to:
- Work with our vendors to develop and install a software patch to improve the stability of our edge routers (gateways).
- Upgrade the line cards on the edge routers to more modern versions.
- Change the 'virtual routing' of more customers to better balance traffic across our core network.
- Increase the capacity of the edge routers
- Upgrade our load balancers to reduce the potential for occasional DNS lookup failures
- Light more central capacity
On a related note we published a summary of the recent problems to our website earlier today that you can see here - http://www.plus.net/support/network_update.shtml
We would like to apologise again for the problems over recent weeks, and will provide another update on Monday.
Kind Regards,
Bob Pullen
Customer Support
|
|
20/11/2009 @ 16:56
|
Broadband Network Capacity (285660) - UPDATE
This is an update to Wednesday's post about the ongoing speed and performance problems some of our customers are experiencing when the network is busy.
Over recent days our network engineers have been recabling parts of the network at one of our London data centres. This has made it easier to balance customer traffic across the platform, and has allowed us to perform some maintenance this afternoon on the problematic gateway pcl-ag02 - http://usertools.plus.net/status/archive/1258728887.htm
We have now finished this maintenance but no customers were disconnected. This means that we will no longer need to perform any network balancing ahead of the weekend.
As things are relatively well balanced already, we have taken the decision to return to a 'Plan A' operation - http://www.plus.net/support/broadband/quality_broadband/load.shtml
This should result in a noticeable improvement in speeds during the evening. We will remain operating under 'Plan A' over the weekend and will review the situation again on Monday when we will provide another update.
There is still the potential for your connection to become slower than normal when things get busy, however we have further maintenance work and fixes that are due to be rolled out over the coming weeks to address these issues.
Included in this work are plans to:
- Work with our vendors to develop and install a software patch to improve the stability of our edge routers (gateways).
- Upgrade the line cards on the edge routers to more modern versions.
- Change the 'virtual routing' of more customers to better balance traffic across our core network.
- Increase the capacity of the edge routers
- Upgrade our load balancers to reduce the potential for occasional DNS lookup failures
- Light more central capacity
On a related note we published a summary of the recent problems to our website earlier today that you can see here - http://www.plus.net/support/broadband/quality_broadband/load.shtml
We would like to apologise again for the problems over recent weeks, and will provide another update on Monday.
Kind Regards,
Bob Pullen
Customer Support
|
|
18/11/2009 @ 15:48
|
Broadband Network Capacity (285660) - UPDATE
This is an update to last week's post about the ongoing speed and performance problems some of our customers continue to suffer, especially during the evenings when the network is at its busiest.
Last Friday the network was performing notably better following some changes our engineers had made earlier in the week. Unfortunately there has been a degradation in performance since then, particularly due to some unforeseen problems we encountered over the weekend and towards the start of this week.
We mentioned in the last announcement that we expected the network to be very busy last Sunday and Monday evening - http://usertools.plus.net/status/archive/1258133233.htm
Due to this, and because of some of the problems that we're yet to fix, we switched into a 'Plan B' network configuration - http://www.plus.net/support/broadband/quality_broadband/load.shtml
The network was busy on Sunday but was actually a lot quieter than we anticipated. Despite this we had a problem late Saturday afternoon that caused all customers connected to pcl-ag03 to be disconnected from the Internet. Whilst these customers were able to reconnect almost immediately, anybody connecting back on to the same gateway would not have been able to browse web pages due to data transfer problems. We didn't become fully aware of this problem until Sunday afternoon at which point we took the decision to perform the following work which would have resulted in more disconnections - http://usertools.plus.net/status/archive/1258297848.htm
Unfortunately, despite our efforts, this only fixed things for some customers and by early Monday morning we were still receiving reports of problems. At this point we stopped any more customers from connecting to the broken gateway and carried out a software update - this fully resolved the problem but resulted in further disconnections for anybody connected to pcl-ag03 - http://usertools.plus.net/status/archive/1258384040.htm
pcl-ag03 is now fully working and is being brought back into balance with the rest of the network.
On Monday afternoon a slightly different problem was identified with another gateway on the network (pcl-ag02). No customers were disconnected this time round and there weren't any data transfer problems, however the gateway stopped accepting any new sessions causing it to drop out of balance as customers began to disconnect or 'churn' from it. We've put a temporary fix in place this afternoon and are continuing to work with our hardware vendor to identify the underlying cause.
Because of the above problems and the imbalance of the network, we remained in 'Plan B' operation last night and will continue to do so until further notice. This *will* result in degraded performance but should ensure that the quality of web browsing is not significantly affected. Some streaming services may be impacted by this dependent on the account type you are subscribed to.
We are aware that there are customers still reporting problems where web pages and other Internet resources fail to connect but are immediately accessible seconds later after trying again. We're still of the opinion that this is due to congestion on some of our core network links, and we have engineers en route to London to recable and reconfigure parts of the network. We will post further details about these changes nearer the time.
Thanks once more for your patience and sorry about the continued inconvenience, we'll provide another update before the end of the week.
Kind Regards,
Bob Pullen
Customer Support
|
|
13/11/2009 @ 17:27
|
Broadband Network Capacity (285660) - UPDATE
This is an update to the post we made a couple of days ago about the performance problems and speed issues that some customers have been experiencing during peak times when the network is busy.
We have seen an improvement in the performance of the network since carrying out some engineering work earlier in the week. Whilst feedback from customers seems more positive, we are aware that some of you are still experiencing problems with speed and network performance during the evenings.
There are still a number of unresolved problems that our engineers are working to fix and until this happens there will be the potential for things to slow down during peak hours, especially after 6.00pm.
We would like to make customers aware that there's a high possibility that the network will be *very* busy in the evening this coming Sunday and Monday. The BBC are streaming both Doctor Who and Top Gear in high definition and the ITV are also broadcasting I'm a Celebrity Get Me Out Of Here and The X Factor. Whilst this might sound trivial, the last series of Top Gear that was broadcast in Standard Definition saw iPlayer traffic on our network jump from a base of around 700Mbps to 1.3Gbps. We're fairly confident that iPlayer traffic will exceed 2Gbps this weekend and possibly reach an all time high of 3Gbps+
Because of the above we will be switching back to 'Plan B' operation on Sunday and Monday evening - http://www.plus.net/support/broadband/quality_broadband/load.shtml
This *will* result in degraded performance but should ensure that the quality of web browsing is not significantly affected. Some streaming services may be impacted by this dependent on the account type you are subscribed to.
We would like to extend our utmost thanks and appreciation to customers for their patience during these difficulties. We know it's not been pleasant for some of you however we'd like to assure you that we've been doing everything we can to address the issues.
Sorry once more for the continued inconvenience, we'll provide another update early next week.
Kind Regards,
Bob Pullen
Customer Support
|
|
11/11/2009 @ 18:47
|
Broadband Network Capacity (285660) - UPDATE
This is an update to last week's announcement about the performance problems and speed issues that some customers have been experiencing during peak times when the network is busy.
In last week's update we mentioned a problem our engineers had identified with the amount of traffic being pushed between some of our core network interfaces. We think that this is one of the main contributing factors to the sluggish browsing, gaming and intermittent DNS problems that have been reported by some customers in our discussion forums.
On Monday and earlier today we carried out some engineering work aimed at spreading customer traffic more evenly across the various interfaces on our core network. This was achieved by using some of the redundant traffic management switches on our network. Now the work is complete customers should see an improvement in the responsiveness of their web browsing, and less occasions where pages intermittently fail to load or DNS lookups appear to fail.
We also lit another 155Mbps segment of central bandwidth yesterday that should go part way towards improving download speeds and the performance of lesser priority traffic such as Peer to Peer and Binary Usenet. In addition to this we increased the bandwidth on our 21CN host links which will improve performance for customers who are currently on an 'Up to 20Mbps' product or those that originally took part in our ADSL2+ trials.
We’re continuing the migration of small numbers of ‘Up to 8Mbps’ customers to the 21CN. We’re emailing each of these customers in advance to let them know about it, although we’re confident it will result in an improved experience.
Tomorrow morning our engineers will be looking to refine some of our traffic management signatures which should help improve performance on some of the more recent games that customers have been having difficulties with
We would like to thanks customers for their continued patience and offer our assurances that we're doing everything we can to restore service to the expected levels.
Whilst Service Status remains at amber some customers may continue to experience -
* Slower speedtest results, especially when comparing the results across different gateways.
* Slower download speeds from download sites/servers like Akamai & Limelight Networks - http://www.plus.net/support/broadband/quality_broadband/download_servers.shtml
* Lower speeds than those advertised on our website for lower priority Internet activities like Peer to Peer and binary Usenet - http://www.plus.net/support/broadband/quality_broadband/speed.shtml
* Higher than normal latency when gaming.
We would like to say sorry to those who continue to be affected by these problems and will provide another update before the end of the week.
Kind Regards,
Bob Pullen
Customer Support
|
|
