The Problem
Over the last few years more and more R&D teams have started to abandon the traditional method of using 3rd party vendors, which required them to install and maintain their own instances of 3rd party’s solutions. Instead, teams started to adopt the new approach in which “Everything is a SaaS”. For example, nowdays its common for modern developer teams to use a SaaS solutions for things like SSO using solutions like Okta, or SaaS solutions for application logging using DataDog, Coralogix or Logz, CI/CD solutions like CircleCI, CodeFresh. The list goes on and on. Sure most of these vendors offer an “On-Prem” version of their product, but most companies will choose to go with managed SaaS solution instead of dealing with the overhead of maintaining more servers and increasing the complexities of their cloud environments. This makes perfect sense from an engineering and operations based perspective, but there is a very large downside to all of this which is security.
This move towards “Everything is SaaS” makes the impact of major security breaches in any of those SaaS vendors much greater, since core parts of the development Eco-system is now public on the internet, and types of vulnerabilities on such platforms that allow attackers to gain read or write access to customer data will often lead to large scale supply chain attacks against the vendors customers.
Here is an example – Imagine a software company named Alice inc. who use a SaaS solution for all their logs, lets call that logging platform BobLogs.io, now what happens if BobLogs.io has a web based security vulnerability which allows one tenant of BobLogs.io to read the log data of other customers of BobLogs.io? Most likely this will allow an attacker who exploits this vulnerability to obtain sensitive data on BobLogs.io’s customers including Alice inc, and since most companies write sensitive data to their logs (which is a bad practice in of it self) like cookies, JWT, secrets and sometimes even passwords, the attacker in this example will gain access to valuable information about Alice which will likely result in a major breach into Alice’s systems.
Now what happens to Alice’s customers? Their data may be exposed as well and if they are also service providers, then we are looking at a 2nd order and 3rd order supply chain attacks, since the vendor of their vendor was breached impacting all victims down the supply chain.
We’ve seen these types of attacks in the last few years become more and more popular, a few examples are the CodeCov breach which allowed attackers to steal CI/CD data of CircleCI’s customers, or the SolarWinds attack which resulted in malware being spread to many of SolarWind’s customers.
Now don’t get me wrong, I’m not saying this “Everything is SaaS” model is wrong, I’m of the mindset that product comes first, and security shouldn’t stand in the way of a company’s ability to execute great products fast – I just think that this increasing risk should be part of every company’s decision process before adopting more SaaS vendors into their development Eco-System. So Here are few things you can to in order to mitigate this risk.
So how should you approach this problem? You should do it in to phases which I’ll describe below. The first part is determining the risk and the second part is reducing the risk.
Phase 1 -Risk Analysis
- Perform Vendor Security Assessment
Design and execute a formal process in which every new on-boarded Vendor undergoes a security assessment to determine their security posture. Set minimum criteria for passing depending on your risk appetite. Vendor risk assessment will vary widely from company to company but make sure you check that the vendor is performing at least annual independent penetration testing assessments and are requiring that all new code changes will be reviewed by someone other than the person writing the code (PR Reviews in short). Ask for a copy the latest penetration test report and the status of the findings from that report. You want make sure that the vendor fixes vulnerabilities in timely manner. This will help you asses if the vendor is taking its security seriously or not.
Another very common practice is to check and ask for the vendor security compliance certifications such as SOC 2 type II or ISO 27001. Most vendors will present this certificate on their website, but if you really want to picky you can ask them what deviations they have in their audit.
An additional step that should be taken here is creating a security questionnaire and sending it to your vendor. Most vendors won’t have a problem answering such questionnaires given that they are relevant concise and to the point. The security questionnaire should focus around security best practices and controls the company has in place. After receiving the answers, go over them and see if the vendor has any major security gaps.
2. Determine the level of risk from on boarding the vendor
After you’ve completed the vendor risk assessment you should have some picture of how seriously the vendor is taking its security. Now you should analyze what risk you are taking by using the vendor’s services. Will the vendor hold sensitive information? if so what kind? Will the vendor have access to you systems? What is the worst case scenario if a system of the vendors gets compromised and how will it affect your systems?
The goal here is to perform a premortem analysis of how will your company be affect in the event of severe security incident with this vendor. Determine if the business risk from the vendor is Low, Medium or High.
3. Make sure the level of risk matches the value from the vendor
Once you have some sense of the level of risk you are taking, see if it correlates to the value you are getting and the importance of the service. It doesn’t make sense to take huge risk for “nice to have” service, but its much more acceptable if this service is a game changer for your business and will solve major problems you may be having.
Phase 2 – Risk Reduction
Once you have some idea of the level of risk you are taking, you shouldn’t just accept it as it as, you should proceed to understand if there is anything you can do about. The following tips will help you reduce the risk from SaaS providers you already have or about to onboard.
1. Opt-in for IP restriction whenever possible
Many vendors these days support some IP restriction functionality which allows access to specific customer accounts only if they are coming from specific IP addresses. If the vendor supports this option and it works with your solution, then you should definitely adopt this security control. IP restriction will act as second layer of security and ensure that even in the event that credentials to the services were somehow compromised, it’s still only possible to access the services from your VPN IPs or office IPs.
2. Remove secrets from your source code
Whether you want to admit it or not your source code is stored right now in more place you’d like to admit. Its stored in you SVN, in your CI/CD solution, it’s saved in every developer’s local computer, IDE plugins have access to source code, in cloud backups, etc. One best practices you should adopt is removing all tokens, API keys, passwords and secrets from your source code, since in all likelihood your source code will leak one day and you don’t want to be in a situation in which all R&D are pulling all nighters in order to rotate keys and replacing passwords, I’ve seen this happen and trust me when I say it’s no fun.
3. Retrieve the Vendors security logs regularly
If you had ever found your self in the situation of have to ask you SaaS vendors for you security logs, you will most likely discovered the following two truths:
- Most vendors won’t store your logs for more than 30 days.
- It might take at least a few days for the vendor to send you the logs
The problem here is in the event of a major supply chain attack, you’ll likely need your access logs from the vendor ASAP and you’ll probably need to look at more that 30 days back. Since this is likely not possible, you’ll need to setup beforehand a solution which downloads your security logs regularly, either using some SIEM solution or just an automated job the downloads a log file (Sometimes this is possible via the vendor’s API). If the vendor doesn’t support either, make sure to take note of that and increase the risk score from that vendor.
4. Enable Tenant Level Encryption whenever possible
Tenant Level Encryption is a process in which the data of every tenant (customer) of the service is encrypted with its own unique key. If done right this method will reduce the impact of most major security incident which involve cross-account data access, since even if an attacker gains access to your data, it will be encrypted with a unique key, which hopefully that attacker doesn’t have access to. This is not a fool-proof protection, because some vendors do a poor job of protecting their decryption keys, but then again, nothing is 100% fool-proof.
If you vendor supports this option make sure you opt in.
5. Map your existing Vendors
You can’t protect what you can’t see (or at least much harder to do so), make sure you have an up-to-date list of all your SaaS vendors and the level of access each vendor has to your system. In the event if a security breach in one of those vendors, you’ll want to understand as quickly as possible how this affects you, what sensitive data just got compromised and what actions you need to take immediately in order mitigation this risk.
6. Prepare an incident response plan
Given a long enough time line, all companies will suffer a security breach at some level, this is just the reality of the world we live in today. Make sure you have an incident response plan. Here are few high level tips:
- The plan should be actionable and concise. It should not be a legal document. In time of crisis you need act fast, no one will read a long boring legal document when everything is up in flames.
- Run periodical incident response drills and improve on them.
- Consider hiring an incident response service.
- Consider purchasing a Cyber security insurance policy.
7. You don’t always have to go for the fully public SaaS solution
Some companies, especially from highly security aware sectors such as the banking industry, are pushing backing against the trend of “Everything is SaaS”, that lead some vendors the develop hybrid or fully on-prem versions of their solution. Like I mentioned at the beginning of the post, this may have its disadvantages, but in terms of security this is likely a better option. So if the security risk of a certain vendor doesn’t meet your risk appetite make sure to ask the vendor if they have a hybrid or an on-prem solution.