Analysis

Government databases

Philip Howard By: Philip Howard, Research Director - Data Management, Bloor Research
Published: 25th March 2009
Copyright Bloor Research © 2009
 Logo for Bloor Research

There has been, and continues to be, much discussion about the deployment of (UK) government databases. There seem to be three issues: first, can you make them work; second, can you secure them; and third, should they exist at all and, if they should, what limits should be imposed on whose data (and how much of it) they should hold? The last of these is essentially an ethical question and depends on how much you trust the government and its agencies. This is not the forum for such a discussion and I shall not address this third question but I will discuss the first two points.

In terms of whether you can make large databases work the answer is surely that you can, given appropriate design, product selection and implementation skills. However, there are issues to be considered. The first is that there appears to be a view that the correct approach is to construct single monolithic databases. I do not believe that this is always, or even often, the best possible approach, particularly when it is a question of consolidating multiple, existing databases. For example, there was much talk after the Soham murders about constructing a single, national police database because there was no way of linking the different police forces' databases. But of course you can do exactly that (and you could have done at the time) using federation from someone like Composite Software or IBM.

The second point is that government procurement seems to be perpetually behind the curve: it seems unaware of the latest technology that is available to it or, even, yesterday's options. Instead, as the Rowntree Reform Trust's recent report puts it "one noticeable effect is that the UK public sector always appears to get sold whatever technology or methodology is just going out of fashion in the public sector". Now, I can't believe that this is the fault of the vendors because they typically want to sell the latest, brightest technology because it's cool. So, the conclusion must be that it is the system integrators and outsourcers who are advising the government who are behind the curve. And this is either because they simply haven't done the appropriate research or because they figure they can make more money out of older technology, or both. Probably both.

The second question is with respect to security. This isn't just about encryption, which is easy enough to handle with proper procedures but also about data quality. For example, the ContactPoint child database has been delayed because there are duplicate records appearing in the database that might help people to identify children at risk. The problem is that children at risk are supposed to be de-identified. And they are. But when the database is updated from external sources such as the child benefit database then duplicate, not de-identified records can enter the system with links created to children on the at-risk register. So, we have a data quality problem. The obvious answer is that the incoming data should be filtered, cleansed, de-duplicated and transformed with appropriate processes before the ContactPoint database is updated.

However, data quality problems will not go away. There is no such thing as perfect data quality. And even if you had perfect data today you wouldn't have it tomorrow because quality deteriorates. To take a simple example, it is estimated that name and address data deteriorates at 1.4% per month (that's people moving house, dying, getting married and changing their name, buying a new mobile phone, and so on). So any government database is going to have to take account of data quality issues. In fact, I am surprised that the civil liberties lobby haven't made more of this point.

While this has been a brief summary, I have to say that, albeit that there have been some successes, I am tired of the expensive failures of the UK government when it comes to be implementing large databases and IT in general. Frankly, this is no more than incompetence: it is perfectly possible to design and implement these systems in a successful manner, given the right tools, knowledge, people and methodologies. The blame lies with the outsourced contractors who clearly do not have (or do not care about) these, and the government for trusting them.

Reader Comments

Sorry, we are no longer accepting comments on this item. We suggest trying to contact the author directly.

26th March 2009: 'Fabrice Pragnaud, LogLogic' said:

Hi Philip,

The government database article is a good read and make many important points. With the database market valued at more than £20bn and the amount of sensitive information stored growing rapidly, it’s little wonder that databases are a huge target for security attacks today. Containing customer credit card information, financial data and intellectual property as bait, some extremely large and sophisticated predators are willing and able to attack and crack open the database for illegal and malicious purposes.

Bearing this in mind, it makes for interesting reading that the UK government is giving 390,000 people access to a £224m “ContactPoint” database containing details of all 11 million children living in the UK! The public sector is a very juicy target for security attacks because typically they spend less time and money than they should on securing databases. The type of data they hold is an attractive motivator – especially for identity theft so it will be interesting to monitor the considerations they give to security. On the flip side, commercial organisations tend to value their data better but don’t always invest in the correct areas and leave database security open to attack. Needless to say, both sectors could do with taking a closer look at what’s going on, who has access to what and how better to manage security.

It’s an issue I’m sure they are both already aware of – with information theft, loss and attack being the Number 1 concern for executives right now. However, despite being cited as a top priority, there were a significant number of public security breaches last year – let alone the amount that went undetected. There’s a huge black market out there for personal data - bank account details sell for 5-10% of the account value and credit card data can sell for up to £30 per account. Take this and multiply by the 4.5 million records breached at online recruitment site Monster.co.uk in January, and whoever did this would have a pretty nice payday!

Insider threat, in addition to external hackers, is also a growing trend and cannot be ignored. Unfortunately in most organisations, privileged database access is granted excessively and managed poorly. Developers and external consultants often have access to sensitive information without much restriction and the database itself is difficult to lock down. On the plus side, database-stored data is in fact subject to most compliance requirements (including PCI DSS, SOX, HIPAA, EU Data Protection Act and others) like privileged user monitoring, audit trails and reporting and keeping patches up to date. So, theoretically most organisations should be on track to cover the database as part of their overall governance and compliance strategy but most often aren’t. For example, several industry research polls have shown that 90 per cent of database vulnerabilities go unpatched – an astounding and very bad practice.

The database is basically a one stop shop for valuable information. What organisations need to do is to consider threats and attacks from both internal and external threats and protect all data copies, locations and platforms. It should be actionable in real-time to detect, alert and prevent. Here are a few tips on how threats can be minimised:

• Reduce exposure - Treat personal identifiable information as sensitive information and put proper security measures around them. Also rigorously apply security principles such as segregation of duties and least privileges – just enough for people to do their jobs, nothing more. Finally, consider 3rd party access carefully and encrypt data.

• Don’t overload the database – make sure that your security solution has little or no impact on the database’s performance.

• Harden the infrastructure - Apply vendor security patches as quickly as possible (use virtual patching if not possible to apply in a timely fashion) and use only strong passwords. Also remove all default usernames and passwords. If a breach occurs - detect and respond.

• Monitor access to databases in real-time – check the logs and audit, audit and audit. Be proactive not reactive and monitor insiders and privileged users – not just the perimeter. Detect all access to sensitive data and identify malicious or suspicious activity as it happens – before it’s too late.

• Develop response capabilities - Automate breach prevention capabilities and prepare a rapid response plan in case of a breach. Isolate and mitigate incidents.

The need to preserve the confidentiality and integrity of data and monitor privileged user activity is driving CIOs and auditors to re-consider their strategy for database security and impose stringent controls across database systems. It’s critical they implement a workable, secure solution and that they not only act upon it, but that they maintain processes and stay up-to-date with patches and controls. Compliance demands it and customers/consumers expect it.

Reply to Fabrice Pragnaud, LogLogic?

The messages above were all contributed by IT-Director.com readers. Whilst we take care to remove any posts deemed inappropriate, we can take no responsibility for these comments. If you would like a comment removed please contact our editorial team.