Dear Chairs, Chief Officers and Petra users,
I would like to explain what is happening with Petra, the immediate steps we can take together to fix the current problem,
and give you the full context about what we're doing.
What we need to do now
The problem we are experiencing with Petra right now strongly appears to be a result of our network being unable
to cope with a large Microsoft patch released earlier this week. Essentially, every computer in the service is requesting to download this at the same time, creating an enormous queue of outbound requests that is is blocking access to Petra, and making it
appear to run slowly.
We are doing everything we can to fix this. But we are strongly advised that the fastest way to get things back
on track would be for everyone who downloads Microsoft updates through our network to make sure they are only doing this during the night.
Please therefore make sure that every computer in your bureaux is set to only download and
install Microsoft Updates during the night. If we all do this, we are assured that performance will improve almost immediately.
The way to do this depends on your windows version, but for most versions it will be:
Click on the start menu, and open the Control Panel.
Open “System and Security”.
Click on the “Automatic Updates” tab
Select "Check for Updates but let me choose when to download and install them" - this will show
a notification in the bottom right corner of the screen as a reminder
Press “ok”
Having to do this in this way is clearly far from ideal, and I know the last few days have been particularly
frustrating. I’ve been in Leicester today - in between phone calls about our network and Petra - where advisers have been clear with me about the impact it’s having on their work, and how much its performance and reliability matters, day in, day out in bureaux.
It is deeply frustrating when it goes down, or appears to within bureaux, with the unacceptable knock on impact for clients.
Lots of you have emailed me today to say how difficult this is making things, and wanting a fuller explanation
of what is happening. So I wanted to let you know what we’ve been doing to make Petra work.
(Just so you know, Petra is the thing that I wake up at night thinking about.)
This is an unusually long email because I want to be as open as possible; please do forward this to people within
your bureaux.
The underlying issues
There are three fundamental causes of problem - the Petra code itself, the fact that Petra requires a certain
kind of network to run, and hosting mistakes by CGI - and it can unfortunately be very hard to identify which is at fault when we experience problems.
The code problems
are what makes Petra run slowly at peak times, particularly between 11 and 1, or when people run reports.
The simple fact is Petra was poorly coded by our supplier Logica (since taken over by CGI), and the extent of the problems came to light too late to change the fundamentals. Essentially, the code is inelegant and makes the application run slowly. So we’ve
always been running to catch up.
Because it matters so much, we’ve thrown everything but the kitchen sink at it: Petra now runs on two of the
fastest computers that money can buy, with 80 cores each. Its data sits on incredibly expensive fusion disks. We’ve worked through thousands of lines of code to tweak performance. But it’s still too slow, too unreliable, and deeply frustrating for too many
of us.
What we’re doing to make this better includes:
Giving you a much better money advice tool, outside of Petra’s code. The prototype is being tested
with hundreds of money advisers now, and it’s clear that it’s a huge improvement. Taking that out of Petra’s code should simplify the code considerably, and make it run faster on the servers.
Trying to separate out the reporting data from the real time data you need to see when advising
and writing up, freeing up more performance. (It seems that people running management reports during peak times is one of the reasons it is performing poorly at the moment, so if you can possibly avoid this, please do so!) This again should simplify and streamline
how the application works, improving speed.
Painstakingly going through hundreds more lines of code to explore even more ways we clean it
up. Each bug we fix improves things a bit, but this approach will never make a huge difference.
The network problems
are more complicated, but I’ll do my best to explain what’s going on. The root issue is that Petra requires
a secure private network (because it’s not secure over the internet, in the way that logging in to online banking or email is). This means that we need to provide you with both internet and connections to Petra through a secure, private MPLS network, which
is hosted by Azzuri. Each bureaux connects to this MPLS network, and then the MPLS network connects to CGI’s data centers in one direction and the internet in another.
What this means is that each bureaux office uses the same local link into the MPLS for both internet and Petra,
and that locally your internet traffic competes with Petra traffic over your local connection. So if you use services that take a lot of bandwidth, such a video streaming or downloading software, this makes Petra run slowly locally.
Once a month, unfortunately, the network experiences a tidal wave of activity. This is caused by the way Microsoft
releases security and other ‘patches’ to its software in big, regular bundles: when it does this, every computer in the service tries to simultaneously download the patch, going through the bureaux’s local connection to the MPLS network, and from this to the
internet. This does two things: first, it overwhelms the MPLS’ connection to the internet, which creates a huge queue of requests going back and forward. (Basically, every computer keeps sending requests for a bit more of the patch, receiving a reply saying
‘you’re at this point in the queue’, and then immediately requests a bit more of the patch). This in turn overwhelms local connections to the MPLS, which compete with Petra data. So the rush of demand effectively blocks both the MPLS network as a whole, and
your local network connection until all your computers in your local network have downloaded the patch. Restricting access to Microsoft servers would make this worse, not better, as it would increase the amount of ‘request’ traffic.
Some bureaux have tried to mitigate this problem by installing separate broadband connections to the internet.
Where this is done absolutely securely, with proper virus protections, managed firewalls, secure routing and secure Wi-Fi it means that patches download locally through an alternative connection, and Petra performance in that bureaux is more ‘protected’ from
the Microsoft patching problem, only being affected when the whole network is overloaded, as it is now. However, where the right security controls aren’t in place, it has introduced viruses into the MPLS network, which can create a large amount of data traffic
within the network, and slow down local connections to Petra. (If you have a separate broadband link your local TSA will be more than happy to come and check it is all configured ok.)
This week, we’ve had both the Microsoft patch problem and a virus problem, which have compounded each other
to make Petra performance slow in bureaux.
What we’re doing to make this better includes:
Doubling the speed of the MPLS connection to the internet, giving it the fastest connection we
can technically obtain. This will be in place by the next Microsoft patch day. While this will help a bit, it still won’t be fast enough to allow every computer on the network to download patches simultaneously, so will only make a bit of difference. (It’s
worth saying that Chromebooks don’t update in this ‘bulky’ way, which is one reason to go for Google Apps).
Trying to make Petra secure over the open internet. This would remove the need for us to provide
an MPLS network. All bureaux could have their own locally arranged broadband connections, at the speed they need locally. This would fundamentally and permanently solve the network problems, but isn’t straightforward to do and needs us to move Petra away from
CGI’s hosting first. I can’t tell you yet whether it will be possible to achieve, or when we’ll know - but hope to be able to in the next few months.
Splitting internet traffic from Petra traffic over your local connections. This would essentially
give Petra traffic priority over any other internet traffic. While this would protect Petra, it might have the unwelcome consequence of making Adviceguide and Advisernet run much more slowly unless we can simultaneously increase the speed of all local connections.
We’re piloting this approach at the moment to make sure it can be delivered safely - unfortunately it’s not straightforward.
The good news is that this problem is fundamentally fixable. The bad news is that it’s technically very hard
to do and brings considerable risk.
The way that large corporates deal with this situation is unfortunately unable to us, as it would require us
having administration rights over every computer connecting to our network, and then staggering updates in a sequenced way.
The one thing that will make a significant difference is for all computers in every bureaux
to be set to only download (and install) updates during the night. But we can’t set this from our end - it needs to be done in every bureaux, on every machine.
While we’re working on the more systemic solutions, it’s unfortunately fairly safe to bet that for a few days
after Microsoft releases a patch (normally the second and sometimes the fourth Tuesday of every month), Petra’s performance will appear slow in bureaux. The size of the issue depends on the size of the patch. When this happens, it’s worth leaving computers
on overnight for at least a few days, as they’ll download at least some of the Microsoft patch then.
The hosting mistakes by CGI
come in all shapes and sizes, which is why they’re so hard to prevent, and why they make me utterly, utterly furious. A couple of months ago the air conditioning in the data centre broke, the servers got too hot and shut down, for example. We’re also currently
investigating whether CGI have correctly configured how Petra uses the cores in the extremely powerful computers we have purchased to power it.
These kinds of problems are the focus of in depth calls, pretty much every day, between our teams, Microsoft
and CGI. People work incredibly hard to fix them. But they’re beyond unacceptable.
That’s why we are working hard on moving Petra away from CGI completely, focusing on:
Testing whether Petra can be moved to Amazon Web Services, which would enable us to put an effectively
unlimited amount of server power behind the application, able to add more power at peak times. This would be an ideal solution, but because of the way Petra has been coded, it may be impossible to do.
Moving Petra to a specialist hosting provider that provides similar services.
Clearly, moving anything like Petra involves risk. So please rest assured: we’ll test everything to within an
inch of its life, and will minimise and mitigate the risks of any changes as robustly as it’s possible to do.
The longer term solution
I’d like to thank you for your ongoing patience - and I hope this update helps to explain why Petra is frustrating
for everyone involved, and the effort and approaches we’re taking to fix it.
I’d also to thank the teams within Citizens Advice, who are working so hard to make things as good as they can
be, particularly the service desk.
I know it’s hard to look ahead when today is so frustrating. But the only long term solution to these problems
is to replace Petra with something simpler, faster and intuitive.
I’m confident we’ve made huge progress on this over the past few months. Hundreds of clients, advisers and others
across the service have been involved in testing new ideas and prototypes. And lots more have been following our work as it unfolds in the open on the project blog
here.
Please do pass this email on to those who would find it useful.
Thanks,
Mike
--
Mike Dixon
Assistant CE
Citizens Advice
Internal: 231101 | External: 03000 231101 | Mobile: 07861 250 530 |
