Third parties such as outsourced service providers and SaaS vendors are a fact of life in the IT world. It’s the nature of a hyper-connected world where hundreds (if not thousands) of applications are required to run even a modestly sized organization. There is no alternative but to trust a third party with access to confidential data. And no matter how strongly you word your outsourcing and SaaS contracts, it is your organization that must answer to its customers when something goes wrong.
Given the continuous stream of breach announcements involving failures with third-party security, CISOs should be apprehensive. The safest assumption to make is that all third-party data sharing is insecure until proven otherwise. This entails an active third-party assessment program, and the old tenet applies: If someone hasn’t checked it, it hasn’t been done.
Assessing your own risk is hard enough, how can you do it for an external organization? Not to mention you have a limited amount of time because most organizations have at least a dozen major third-party vendors.
If you remember your information security basics, proper risk assessments start with a complete asset analysis, followed by a threat analysis, controls assessment, vulnerability analysis, and impact calculations. That is a lot to do. Because of this, most organizations send their third parties questionnaires to fill out and expect them to honestly self-assess their security posture. The problem with self-assessments is that people often interpret things differently than you’d like. One organization’s idea of a security policy might be a thirty-page comprehensive document, signed and reviewed by all employees. Another’s might be a one-page, high-level guideline taped up to the server room door. Add to this discrepancy the danger that the assessed third party may tend to report what they think you want to hear. So, self-assessments always should be taken with a grain of salt.
Even if an assessment is done with demands for evidence, such as documentation, diagrams, and scan reports, data becomes irrelevant quickly in fast-moving organizations. Documentation can be incomplete, obsolete, or simply ignored. A lot of network diagrams are almost immediately out of date after publishing, especially manually created ones. Scan outputs are not reality, but rather the tool’s perception of reality. Where does this leave us?
With apologies to Winston Churchill1: Audit is the worst form of measuring a security program—except for all those other forms that have been tried from time to time.
To be more consistent in our approach, we need to do away with this ambiguity and upgrade from an assessment to an audit. The difference between the two is that an audit is assessment to an accepted standard. The audit standard defines the breadth, depth, and frequency of measurement so you have some reliably consistent results. Most audits mandate specific inspection and verification methods by trained professionals so you will have a more accurate picture.
There are some downsides of audits, the biggest one being that they are not cheap. Many organizations require audit reports from their third parties who, in turn, pass this cost back in added (and sometimes hidden) service fees. The second downside of audits is subtler: they tell a biased story based on their certification standard. Most audits have a logic of their own, independent from reality, based on their focus. For example, PCI DSS audits focus solely on payment card environments but do not look at protecting anything beyond payment cards. SSAE 16 SOC 1 audits are about the integrity of financial transactions but do not cover confidentiality or availability. The coverage of an ISO 27001 audit is auditee-defined and so varied that it is the primary shared statement in certification.
A universal bias of audit is based on the auditor’s work itself: if the auditor did not see something and report it, you will not know about it. This is an obvious fact that is usually covered in the fine print with the audit report. But people sometimes forget this when they blindly trust an audit report as a perfect representation of reality. It is not, and it will never be, but most of the time an audit can produce reasonable evidence of the security controls in place at an organization.
When you receive an audit report from a third party, the first thing you know before even opening it up is that there are some resources being dedicated to security. Like I said, audits aren’t cheap, so you know that the third party has at least organized and dedicated some work towards a security program that’s good enough to make it through an audit. It’s a useful threshold to consider when evaluating third-party security.
When you open the report, it’s important to scrutinize the audit scope and timeliness (audits always represent the past state, not the current). The audit scope defines what was in or out of the assessment. If the scope only covers the mail server in the wiring closet during the first quarter of last year, it’s probably not a relevant measure of your secure e-commerce web hosting facility. Remember the different types of audit, as well. Don’t expect a SSAE 16 SOC 1 audit to give you details on HIPAA data handling procedures.
Beyond reading the scope and timeliness, the next thing to review is the controls tested and the results. The auditor and chosen auditor standard is supposed to have already ensured that the control set under review is appropriate and adequate to cover the risks to the scoped systems. If the auditor judges the controls sufficiently effective (and that does not mean all controls must work perfectly all the time), then he or she will certify the audit as either unqualified or passed. If there are too many control failures or design flaws, they will be noted in the review. Be aware about how much of the reporting and analysis leans on scope and relevance. Once again, make sure your audit type matches your concerns with the third party. One of the most common mistakes in reviewing third-party audit reports is mismatching audit type with risk need.
At this point in time, there is no comprehensive, reliable, or scalable way to measure third-party security beyond audit. An audit tells a biased, limited, and perishable story, but it is one of the best tools we have. In the end, there is no such thing as a proved state of secure. We can only send out pings in the dark and listen for echoes to make our guesses.
