VoIP.co.uk Network Status

Inbound PSTN issues

theo — Fri, 4 Jul 2008 12:48:53 +0100

We've been seeing intermittent call failures on inbound calls from the PSTN to our network. We have tracked this down to a faulty line card, and have shut-down the affected ports. Calls are now flowing normally again.

Outgoing PSTN calls (update)

theo — Thu, 26 Jun 2008 12:41:21 +0100

The issue that caused us to take the PSTN gateway we took out of service this morning has been tracked down to a misconfiguration on a dial-peer on the gateway having problems. It has been now been put back into service and we are closely monitoring calls.

No calls will have been affected since it was taken out of service, however this morning a number of outbound calls (from 9:33 until 9:38) hitting that gateway from the B platform will have received a 'network busy' response. Re-dialling may have caused the call to go through, as it was down to random gateway selection if this error was received.

The failover testing we're been trialing on the beta platform has gone extremely well, and the results have been so far perfect in eliminating PSTN gateway issues we've started to see as we add more and more PSTN interconnects. We will schedule some maintenance work over the next few days to put the same functionality on the B platform, which will stop future unforeseen problems like this affected any calls at all.

Outgoing PSTN calls

theo — Thu, 26 Jun 2008 09:38:56 +0100

we are investigating a PSTN gateway that was causing problems this morning rejecting outgoing calls.

The gateway has been removed from service and is being investigated.

[MW-0806-03] controlled PSTN failover tests

theo — Sat, 21 Jun 2008 14:52:20 +0100

MAINTENANCE WINDOW NOTIFICATION

At risk Window Commences: 2008-06-26 03:00:00 BST (UTC +0100)

At risk Window Concludes: 2008-06-26 04:00:00 BST (UTC +0100)

Reason for notification: Controlled testing of PSTN gateway failure failover

During the above maintenance window, we will be performing simulated hardware, software, and network failures to ensure that PSTN calls fail over successfully and are not adversely affected by platform component failure.

This work will involve disabling parts of the platform in a controlled manner, and ensuring traffic flows correctly through alternate gateways. We will then re-enable the disabled parts, while ensuring throughout that traffic is not affected.

It is hoped there will be no downtime at all, however the very nature of the work means that there is an increased risk of outgoing call failures.

The network will be closely monitored by engineers during and over the hours following this maintenance. As always, if you have any problems outside of this window please contact us through your normal support channel.

An all clear notification reporting results of maintenance will be issued on conclusion. Should you be suffering any loss of service after this time, or outside this maintenance window, please get in touch as soon as possible.

Contingency plan:

Should the above testing fail to be successful as planned then the network will be restored to its prior state for analysis of redundancy failure.

[MW-0806-01]: Altered to 2008-06-26 0100 - 0500

theo — Sat, 21 Jun 2008 02:15:20 +0100

We've needed to change this maintenance window by a few days in order to ensure platform stability.

This work will now take place between 0100 and 0500 on Thursday the 26th of this month.

billing issues 2008-06-19

theo — Fri, 20 Jun 2008 17:06:05 +0100

We've recently noticed a side effect of yesterdays PSTN gateway problems meant that anyone who was on a call on the B cluster (proxy.voip.co.uk) at a number of points in the morning may not have seen their call get cut of from a billing perspective. This resulted in those specific calls being either over or under charged.

The problem has been tracked down to a signalling node that isn't clearing billing information down correctly under a certain failure scenario, of which yesterday was the first time such a scenario happened. A fix has been put into the stack already, and this will be rolled out over the weekend onto the live platform.

We will of course refund anyone who was adversely affected by this over the weekend. The process will be automatic as we have safeguards in place to ensure we can retrospectively correlate billing information in situations like this and re-generate the call records, however should you notice it because it's affected your ability to make calls, please contact support and we'll credit your account temporarily until the call is re-rated at the correct amount.

An update will be posted here as soon as there is further information.

PSTN gateway problem (Resolved)

theo — Thu, 19 Jun 2008 12:31:48 +0100

At 1202 one of the PSTN gateways which primarily services incoming calls was seeing 100% CPU utilisation and was failing to accept any new incoming calls. For some reason (which we are investigating), incoming calls did not fail over to an alternate gateway. The affected gateway was taken out of service at 1210, and normal inbound service resumed at 1211.

We apologise for this issue, and are working on (a) ensuring if this problem happens again calls automatically fail over to another gateway rather than simply get rejected with "Network Error", and (b) investigating why this happened in the first place.

Forum upgrades

ian — Wed, 18 Jun 2008 12:41:39 +0100

Forum upgrades are now complete and the forum is back online.

Forum upgrades

ian — Wed, 18 Jun 2008 12:20:35 +0100

Our forum is currently down for routine maintenance, it will be back shortly.

[MW-0806-02] UPS Maintenance 23-24 June, 1 July

theo — Wed, 18 Jun 2008 07:05:09 +0100

We have been notified by one of our data centre facility operators that there will be three maintenance windows during which parts of the site-wide UPS systems will be replaced and maintained. These will take place as follows:

MW-0806-02A: 23/06/08 .. 0800 to 1500

MW-0806-02B: 24/06/08 .. 0800 to 1500

MW-0806-02C: 01/07/08 .. 0800 to 1500

This will reduce the redundancy of the UPS systems during the maintenance thus presenting a slightly increased risk of problems, although we do not expect customers to be affected.