Petra, Debt and Wales in the Red

Wales in the Red

This weeks BBC Week In Week Out focused on debt in Wales. The whole 30 minute programme is worth a watch but Torfaen CAB is featured for a few seconds at 18 minutes. Torfaen was cited by the programme as the third most indebted part of Wales behind Merthyr and Denbighshire

BBC Week In Week Out - Wales in the Red

Changes to the way we do Debt Advice

We are just a few months into the new Money Advice Service Debt Advice contract and some changes are beginning to be rolled out across the CAB network. These changes are part of a complete overhaul to the system of MAS funded debt advice across the UK.

The Common Initial Assessment (CIA) is an online diagnostic triage tool which improves the client experience within the Citizens Advice service.

It is designed to enhance and standardise both the quality of information gathered from the client at an early stage in the debt advice process as well as promote consistency of advice across the bureaux network.

By working in this way, the CIA aims to improve the outcomes of the debt advice given across the service.

The CIA also enhances the ability of bureaux to manage demands for Specialist Debt Assistance. By asking a set of standard questions, which gathers essential information from the client, it allows the bureau to provide a faster and better quality route to the most appropriate next step.

At the end of the assessment there will be a summary page with all the required information that the assessor has recorded. This page should be copied and pasted onto Petra GAI assessment.

The tool has been developed for use by someone with limited debt and financial experience.

It is very important that all advisers who Gateway a client with a debt issue use this tool. Links will be on the desktops of all advice room PC's at Cwmbran by Monday morning and there will be a short briefing at each morning meeting this week to show staff and volunteers how to use the CIA including the importance of changing the funder from CORE to MASDAP in the Gateway.

We aim to have the system fully implemented by the end of next week.

There will be more changes detailing how different specialist services will be added to the referral routes of debt clients over the next few months.

The Cablink pages explaining the CIA including a link to the CIA tool itself are available here

Disruption to PETRA

Most of the Citizens Advice network has experienced problems this week with Petra being unavailable for significant periods. The short story is that this was caused by several thousand bureau computers all trying to download a particularly large Microsoft security update simultaneously. The good news is there are some quick and some long term solutions in place to make sure it does not happen again.

The lengthy email pasted below explains the full picture and I have included it here for those who are interested but it is far from compulsory reading. 

We will be making the changes to the way our computers handle updates (as outlined in the email below) this week on our Pontypool computers (we don't need to do this at Cwmbran because we have our own separate fire-walled internet connection so the updates do not use the Citizens Advice network)

What we need to do about Petra

Dear Chairs, Chief Officers and Petra users,

I would like to explain what is happening with Petra, the immediate steps we can take together to fix the current problem, and give you the full context about what we're doing.

What we need to do now

The problem we are experiencing with Petra right now strongly appears to be a result of our network being unable to cope with a large Microsoft patch released earlier this week. Essentially, every computer in the service is requesting to download this at the same time, creating an enormous queue of outbound requests that is is blocking access to Petra, and making it appear to run slowly.

We are doing everything we can to fix this. But we are strongly advised that the fastest way to get things back on track would be for everyone who downloads Microsoft updates through our network to make sure they are only doing this during the night.

Please therefore make sure that every computer in your bureaux is set to only download and install Microsoft Updates during the night. If we all do this, we are assured that performance will improve almost immediately.

The way to do this depends on your windows version, but for most versions it will be:

• Click on the start menu, and open the Control Panel.
• Open “System and Security”.
• Click on the “Automatic Updates” tab
• Select "Check for Updates but let me choose when to download and install them" - this will show a notification in the bottom right corner of the screen as a reminder
• Press “ok”

Having to do this in this way is clearly far from ideal, and I know the last few days have been particularly frustrating. I’ve been in Leicester today - in between phone calls about our network and Petra - where advisers have been clear with me about the impact it’s having on their work, and how much its performance and reliability matters, day in, day out in bureaux. It is deeply frustrating when it goes down, or appears to within bureaux, with the unacceptable knock on impact for clients.

Lots of you have emailed me today to say how difficult this is making things, and wanting a fuller explanation of what is happening. So I wanted to let you know what we’ve been doing to make Petra work. 

(Just so you know, Petra is the thing that I wake up at night thinking about.) 

This is an unusually long email because I want to be as open as possible; please do forward this to people within your bureaux.

The underlying issues

There are three fundamental causes of problem - the Petra code itself, the fact that Petra requires a certain kind of network to run, and hosting mistakes by CGI - and it can unfortunately be very hard to identify which is at fault when we experience problems.

The code problems are what makes Petra run slowly at peak times, particularly between 11 and 1, or when people run reports. The simple fact is Petra was poorly coded by our supplier Logica (since taken over by CGI), and the extent of the problems came to light too late to change the fundamentals. Essentially, the code is inelegant and makes the application run slowly. So we’ve always been running to catch up.

Because it matters so much, we’ve thrown everything but the kitchen sink at it: Petra now runs on two of the fastest computers that money can buy, with 80 cores each. Its data sits on incredibly expensive fusion disks. We’ve worked through thousands of lines of code to tweak performance. But it’s still too slow, too unreliable, and deeply frustrating for too many of us.

What we’re doing to make this better includes:

• Giving you a much better money advice tool, outside of Petra’s code. The prototype is being tested with hundreds of money advisers now, and it’s clear that it’s a huge improvement. Taking that out of Petra’s code should simplify the code considerably, and make it run faster on the servers.
• Trying to separate out the reporting data from the real time data you need to see when advising and writing up, freeing up more performance. (It seems that people running management reports during peak times is one of the reasons it is performing poorly at the moment, so if you can possibly avoid this, please do so!) This again should simplify and streamline how the application works, improving speed.
• Painstakingly going through hundreds more lines of code to explore even more ways we clean it up. Each bug we fix improves things a bit, but this approach will never make a huge difference.

The network problems are more complicated, but I’ll do my best to explain what’s going on. The root issue is that Petra requires a secure private network (because it’s not secure over the internet, in the way that logging in to online banking or email is). This means that we need to provide you with both internet and connections to Petra through a secure, private MPLS network, which is hosted by Azzuri. Each bureaux connects to this MPLS network, and then the MPLS network connects to CGI’s data centers in one direction and the internet in another.

What this means is that each bureaux office uses the same local link into the MPLS for both internet and Petra, and that locally your internet traffic competes with Petra traffic over your local connection. So if you use services that take a lot of bandwidth, such a video streaming or downloading software, this makes Petra run slowly locally.

Once a month, unfortunately, the network experiences a tidal wave of activity. This is caused by the way Microsoft releases security and other ‘patches’ to its software in big, regular bundles: when it does this, every computer in the service tries to simultaneously download the patch, going through the bureaux’s local connection to the MPLS network, and from this to the internet. This does two things: first, it overwhelms the MPLS’ connection to the internet, which creates a huge queue of requests going back and forward. (Basically, every computer keeps sending requests for a bit more of the patch, receiving a reply saying ‘you’re at this point in the queue’, and then immediately requests a bit more of the patch). This in turn overwhelms local connections to the MPLS, which compete with Petra data. So the rush of demand effectively blocks both the MPLS network as a whole, and your local network connection until all your computers in your local network have downloaded the patch. Restricting access to Microsoft servers would make this worse, not better, as it would increase the amount of ‘request’ traffic.

Some bureaux have tried to mitigate this problem by installing separate broadband connections to the internet. Where this is done absolutely securely, with proper virus protections, managed firewalls, secure routing and secure Wi-Fi it means that patches download locally through an alternative connection, and Petra performance in that bureaux is more ‘protected’ from the Microsoft patching problem, only being affected when the whole network is overloaded, as it is now. However, where the right security controls aren’t in place, it has introduced viruses into the MPLS network, which can create a large amount of data traffic within the network, and slow down local connections to Petra. (If you have a separate broadband link your local TSA will be more than happy to come and check it is all configured ok.)

This week, we’ve had both the Microsoft patch problem and a virus problem, which have compounded each other to make Petra performance slow in bureaux.

What we’re doing to make this better includes:

• Doubling the speed of the MPLS connection to the internet, giving it the fastest connection we can technically obtain. This will be in place by the next Microsoft patch day. While this will help a bit, it still won’t be fast enough to allow every computer on the network to download patches simultaneously, so will only make a bit of difference. (It’s worth saying that Chromebooks don’t update in this ‘bulky’ way, which is one reason to go for Google Apps).
• Trying to make Petra secure over the open internet. This would remove the need for us to provide an MPLS network. All bureaux could have their own locally arranged broadband connections, at the speed they need locally. This would fundamentally and permanently solve the network problems, but isn’t straightforward to do and needs us to move Petra away from CGI’s hosting first. I can’t tell you yet whether it will be possible to achieve, or when we’ll know - but hope to be able to in the next few months.
• Splitting internet traffic from Petra traffic over your local connections. This would essentially give Petra traffic priority over any other internet traffic. While this would protect Petra, it might have the unwelcome consequence of making Adviceguide and Advisernet run much more slowly unless we can simultaneously increase the speed of all local connections. We’re piloting this approach at the moment to make sure it can be delivered safely - unfortunately it’s not straightforward.

The good news is that this problem is fundamentally fixable. The bad news is that it’s technically very hard to do and brings considerable risk.

The way that large corporates deal with this situation is unfortunately unable to us, as it would require us having administration rights over every computer connecting to our network, and then staggering updates in a sequenced way.

The one thing that will make a significant difference is for all computers in every bureaux to be set to only download (and install) updates during the night. But we can’t set this from our end - it needs to be done in every bureaux, on every machine.

While we’re working on the more systemic solutions, it’s unfortunately fairly safe to bet that for a few days after Microsoft releases a patch (normally the second and sometimes the fourth Tuesday of every month), Petra’s performance will appear slow in bureaux. The size of the issue depends on the size of the patch. When this happens, it’s worth leaving computers on overnight for at least a few days, as they’ll download at least some of the Microsoft patch then.

The hosting mistakes by CGI come in all shapes and sizes, which is why they’re so hard to prevent, and why they make me utterly, utterly furious. A couple of months ago the air conditioning in the data centre broke, the servers got too hot and shut down, for example. We’re also currently investigating whether CGI have correctly configured how Petra uses the cores in the extremely powerful computers we have purchased to power it.

These kinds of problems are the focus of in depth calls, pretty much every day, between our teams, Microsoft and CGI. People work incredibly hard to fix them. But they’re beyond unacceptable.

That’s why we are working hard on moving Petra away from CGI completely, focusing on:

• Testing whether Petra can be moved to Amazon Web Services, which would enable us to put an effectively unlimited amount of server power behind the application, able to add more power at peak times. This would be an ideal solution, but because of the way Petra has been coded, it may be impossible to do.
• Moving Petra to a specialist hosting provider that provides similar services.

Clearly, moving anything like Petra involves risk. So please rest assured: we’ll test everything to within an inch of its life, and will minimise and mitigate the risks of any changes as robustly as it’s possible to do.

The longer term solution

I’d like to thank you for your ongoing patience - and I hope this update helps to explain why Petra is frustrating for everyone involved, and the effort and approaches we’re taking to fix it.

I’d also to thank the teams within Citizens Advice, who are working so hard to make things as good as they can be, particularly the service desk.

I know it’s hard to look ahead when today is so frustrating. But the only long term solution to these problems is to replace Petra with something simpler, faster and intuitive.

I’m confident we’ve made huge progress on this over the past few months. Hundreds of clients, advisers and others across the service have been involved in testing new ideas and prototypes. And lots more have been following our work as it unfolds in the open on the project blog here.

Please do pass this email on to those who would find it useful.

Thanks,
Mike

-- 

Mike Dixon
Assistant CE
Citizens Advice
Internal: 231101  |  External: 03000 231101  | Mobile: 07861 250 530   | 

______________________________________________________________________

Please consider the environment - do you really need to print this email?

The Citizens Advice service helps people resolve their legal, money and other problems by providing information and advice, and by influencing policymakers.
Notice: this email originates from Citizens Advice, an operating name of The National Association of Citizens Advice Bureaux (Charity Registration Number 279057, VAT number 726 0202 76, Company limited by guarantee, Registered Number 1436945 England, Registered office 3rd Floor North, 200 Aldersgate, London EC1A 4HD). It contains proprietary information, some or all of which may be legally privileged. It is for the intended recipient only. If an addressing or transmission error has misdirected this email, please notify the author by replying to this email. If you are not the intended recipient you must not use, disclose, distribute, copy, print or reply to this email (other than for the reason stated above).

This footnote confirms that this email message has been swept by Symantec.Cloud for Content Security threats, including computer viruses
______________________________________________________________________

______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
For more information please visit http://www.symanteccloud.com
______________________________________________________________________