Cloud based apps are extremely vulnerable - here's what to do

And the Two Design Patterns That All Developers Should Know

15 per cent of business cloud users have been hacked.

That is according to a recent Netskope report (article here).

Recent debates about whether cloud storage are secure have focusd on the infrastructure of the cloud: that is, if your data is in the cloud, can other cloud users see it? Can the cloud provider see it?

But there has been little attention to the event more important issue of whether cloud apps themselves are secure. This is so important because if your data is in the cloud, then it is not behind your company’s firewall – it is accessible over the Internet. All one needs is the password. So if you think things were bad before, just wait until hackers shift their focus to the cloud.

Executives think that IT staff know how to

write secure software applications

– but most don’t.

Companies spend huge amounts of money trying to make their infrastructure secure, but they invest essentially nothing in making sure that their application code itself is secure. As I wrote in my book High-Assurance Design,

•    The average programmer is woefully untrained in basic principles related to reliability and security.

•    The tools available to programmers are woefully inadequate to expect that the average programmer can produce reliable and secure applications.

•    Organizations that procure applications are woefully unaware of this state of affairs, and take far too much for granted with regard to security and reliability.

The last bullet is the most important one: Executives of companies think that IT staff know how to write secure software applications – that to do otherwise would be unethical, and their staff are definitely not unethical. But this attitude is the heart of the problem, because the fact is, most software developers – and even most senior software architects – know very little about how to write secure software. Security just isn’t that interesting to most programmers: no one rewards you for writing secure code – not like you get rewarded for writing more features. And no one is asking for it, because there is an assumption – an incorrect one – that programmers create secure code in the course of their work, just as plumbers create well sealed pipes in the course of plumbing. True for plumbers, in general, but not true for programmers.

Recently I was on a DevOps team in which the client was very concerned about security. The client ran its own scans of our servers in the cloud, and found many issues that needed to be fixed. All of these issues were infrastructure related: primarily OS hardening. None had to do with the design of the application. The general feeling of the team was that the security of the application itself would not be questioned, so we did not have to worry about it. At one point, one of our databases in our cloud test environment was hacked. The database was shut down and a forensic analysis was supposedly performed (we were never told what they found). There was no impact on the team’s work – it was business as usual.

If we don’t fix this dysfunction in our industry,

then the Internet Of Things (IOT) will be a disaster.

This state of affairs is unsustainable. If we don’t fix this deep rooted dysfunction in our industry, then the Internet Of Things (IOT) will be a disaster: Imagine having every device you own connected to the Internet – to a cloud service of some kind – and all of these devices and accounts hackable. And imagine the continuous software updates to keep pace with newly discovered security vulnerabilities. This is not a future that I want – do you? Not only is George Jetson’s car a pain to maintain – with constant software updates – but it might come crashing down. People will be afraid to drive their car or use these IOT devices.

The only way to fix this is for organizations to demand that developers learn how to write secure software. You cannot scan for application level security: doing so is not effective. Having a “security officer” oversee things is not effective either – not unless that person intends to inspect every line of code written by every programmer – and that is not feasible in an Agile setting, where the code changes every day. The only way to produce secure software in an Agile environment is to for the programmers to know how to do it.

It is not that there are not lots of resources available for this, my own textbook included. There are tons of books, there are online resources – notably OWASP – and there are even certifications – and these certifications are the real deal: these are not fluff courses.

People like magic bullets. Unfortunately, there is no magic bullet for security: knowledge is the only path. But if I were asked what two things software developers should know to make their code more secure, I would have to say that they should know about these two design patterns: (1) Compartmentalization, and (2) Privilege Separation.

Your systems will be hacked. The only question is,

What will the hackers get away with?

Your systems will be hacked. There is no question about that. The only question is, What will the hackers get away with? Will they be discovered right away through intrusion detection monitoring and shut down? And if not, will they be able to retrieve an entire database of information – all of your customers’ personal data? That is, will one compromised account enable them to pull down an entire complete set of information?

Compartmentalization is an old concept: In the context of computers, it was first formalized by the Bell LaPadula model for security. It became the basis for security in early military computer systems, and it formalizes the essential concept used by the military and intelligence communities for protecting sensitive information. It is based on the concept that a person requesting access to information should have (A) sufficient trust level – i.e., they have been vetted with a defined level of thoroughness – and (B) a need to know: that is, they have a legitimate reason for accessing the information. No one – not even the most senior and trusted person – can automatically have access to everything: they must have a need to know. Thus, if someone needs information, you don’t open the whole filing cabinet: you open only those files that they have an immediate need for. To open others, you have to request permission for those.

Military computing systems are onerous to use because of the layers of security, but in a civilian setting for business applications there are ways to adopt the basic model but make parts of the process automatic. For example, restrict the amount of information that an individual can access in one request: don’t allow someone to download an entire database – regardless what level of access they have. And if they start issuing a-lot of requests – more than you would expect based on their job function – then trigger an alarm. Note that to implement this type of policy, you have to design the application accordingly: this type of security is not something that you can bolt on, because it requires designing the user’s application in such a way that they only access what they need for each transaction and are not given access to everything “in that file cabinet”.

The other key concept that programmers need to know is “privilege separation”. No one should be able to access a large set – e.g., a table – of sensitive data directly: instead, they should have to access a software service that does it for them. For example, if a user needs to examine a table to find out which rows of the table meet a set of criteria, the user should not be able to access or peruse the table directly: the user should only be able to initiate the filter action and receive the single result. The filter action is a software service that performs the required action under the privileged account of the server – which the user does not have access to. The user performs his or her work using an account that is only able to initiate the software service. If the user’s account is obtained through a phishing attack, that account cannot be used to obtain the raw data in the database: retrieving the entire table would require a huge number of calls to the service and intrusion monitoring should be watching for abnormal use such as that. This does not prevent hacking, but it greatly limits what can be lost when a hack occurs.

These measures are not sufficient, but they are a start, and they provide a foundation for how to think about application level security, from which programmers can learn more. The key is to start with an access model based on the kinds of actions that users need to perform and the subsets of data that they need direct access to for each transaction – access is not simply based on their overall level of trust or general need to access an entire class of data.

Organizations are completely to blame for the current state of affairs – and organizations can fix it.

Organizations are completely to blame for the current state of affairs: If organizations demand that programmers know how to write secure code, then programmers will respond. People are merely focusing on what their bosses are telling them is important.

So if you are an executive in an IT organization, it is up to you. The industry will not fix things: You need to make security a priority. You need to tell your teams that you expect them to learn how to write secure code. You need to create incentives for programmers and software architects to become knowledgeable and even certified in secure coding. You need to create a culture that values security. Security is up to you.

Real Agile Testing, In Large Organizations – Part 4

(Continued from Part 3)

Is everyone a tester?

This is one of the greatest debates about Agile testing: who does it? One camp claims that everyone on an Agile team is a tester: there should be no “tester” role. The Scrum camp is perhaps most adamant about this. Another camp claims that there are testers, and that the separate role is very important.

Again, the right answer depends. Even Jeff Sutherland – the inventor of Scrum – has complimented the performance of projects that had separate test teams. This one stands out. So if it is ok with Dr. Sutherland, it should be ok for Scrum adherents. The question is, when does it make sense, and when does it not make sense?

For Morticia’s website (see Part 1), Thing did most of the testing but we all chipped in, and that was fine. But for the EFT Management Portal, the testing is so complex that it would sure make sense to have a test lead, and several people to focus on pulling together the performance testing, the security testing strategy, the testing of the multiple back end partner interfaces, the testing of the legal compliance rules, and so on. These things are like projects in their own right and so they need leads. But don’t forget about learning: some people might want to learn about new types of testing and test automation, even if they have not done it before, so allow team members to change roles with appropriate supervision (e.g., through pairing).

Saying “You should never have a test lead” is not very Agile,

and saying “You should always have a test lead”

is not very Agile either.

If unsure, use common sense: don’t use doctrine. Agile is first and foremost about applying judgment: that is why the Agile Manifesto is written the way it is, with phrases like “While there is value in the items on the right, we value the items on the left more.” In other words, it is not prescriptive. They wanted us to keep our thinking flexible: e.g., saying “You should never have a test lead” is not very Agile, and saying “You should always have a test lead” is not very Agile either.

Who needs to understand these things

One of the most important aspects of Agile in general is the conversations among the team – the exchange of ideas, the helping each other, and the talking through of issues. This is critically important as it relates to testing and understanding when we have met the needs of the end users. Ensuring that the team understands – not just reads – the testing strategy, is critical, so that the testing strategies are ever present in the minds of the developers. Developers need to think about testability as they design and code, and they need to be thinking about failure modes and how those are going to be tested, because developers often identify situations that that testers have overlooked. Communicating testing concerns across team roles is extremely important.

In Part 1 of this article we pointed out that an Agile test strategy is developed and maintained by the team, in collaboration with external stakeholders. The team leader(s) (e.g., Scrum Master, coach, project manager, tech lead, etc. – however the team(s) is/are constituted) need(s) to understand what an Agile test strategy is for, so that it is accounted for during iteration planning. If the organization has support functions such as Security, Testing, Architecture, etc., those support functions need to understand how an Agile test strategy is different from a traditional test plan: the collaborative nature of Agile testing, the need to test continually, the need to automate as much as possible, and the need to evolve the testing strategies as the team learns more about the application and its requirements. The support function managers need to know all this so that they can make sure that they provide staff to collaborate with the team in the initial development of the testing strategies and throughout software development.

The support functions will need to come to terms with the fact that their role changes significantly with respect to waterfall development: waterfall teams obtain services from support functions, services such as Testing, Architecture, etc., and those services operate largely independently. In an Agile setting, the support functions need to operate in a collaborative way, working side by side with the team. In fact, much of their work should shift from “doing” to “teaching” – i.e., the support functions need to coach the team in how to perform the things that the support function used to do – to the extent that that is practical. Thus, support functions become coaching centers and resource centers. (In the article How To Rapidly Infuse Technical Practices Into Your Agile Teams we talk about how to transition waterfall oriented support functions to Agile support functions.)

Agile coaches need to work with the various support functions to help them to think through these changes. Agile will impact the types of people who work in the various support functions – they need to be more people-oriented, with an interest in helping others instead of doing the work themselves. The suppot function staff will also have to learn about Agile practices and automation tools. Agile will impact how the support functions are measured by senior management: they will need to be measured on how effectively they help teams to become self sufficient in technical practices, and the support functions also need to be measured in terms of whether they stay current in the rapidly evolving landscape of Agile tools. Given these changes, Agile will therefore impact funding for these functions. It will shift the balance of power in the organization, and that is why the CIO needs to be the driver for these discussions. In a successful Agile transformation, the support functions are not eliminated: they are transformed and reorganized. Knowledge must increase – not decrease – and to make continual learning a sustainable practice, it really helps to have organizational functions that focus on helping practitioners – the teams – to continue to learn new things in an endless cycle of learning, doing, and improving.

Creating a learning organization

In a discussion thread in the LinkedIn group Agile and Lean Software Development, Claes Jonsson, a Continuous Deployment architect at TPG Objektfabriken in Sweden, asked this pertinent question:

How is [assurance achieved] in an organization that is committed to delivering the right thing, with extremely high quality, minimal waste and with the shortest possible time to market using Lean Startup principles and Continuous Release practices? And do note that this does NOT mean unstructured, or disorganized, it instead relies on high organizational alignment, and extreme discipline.

The only way for an organization to preserve assurance – that is, manage risk – while becoming more Agile is to elevate people's knowledge. E.g., consider security: the process of having "gates" for security review is antithetical to Agile because gates impose risk management at discrete points instead of integrating it into the development process itself. But if you teach developers how to write secure code, then you don't need the gates anymore! The same thing applies to other areas of assurance.

But wait: I am using a little bit of hyperbole here: gates are a form of oversight, and it is not really that you don’t need any kind of oversight – you do – but it takes a different form. In Part 2 of this article we talked about how the concept of test coverage, and the role that a quality assurance (QA) function might play in an Agile setting. We explained that an Agile form of QA is still independent, but that it works alongside a development team – not as a gated phase. Again, Morticia’s website probably doesn’t need such a setup, but the EFT management portal probably does: you need to make a judgment about how much independent quality oversight is necessary to properly manage all of the risks in your project or program. Agile is about transparently and collaboratively making those kinds of judgments – not blindly following a plan or procedure.

Many large organizations utilize gated software development processes due to historical reasons. These “gates” typically include a review phase for things such as security and regulatory compliance adherence. What we find when working with these organizations – and this is different from small companies and startups – is that the gates are used in lieu of conversations about how the software will meet compliance requirements: i.e., the transparent and collaborative discussions about what process to use for the project do not take place. By shifting to a culture of learning through conversations, gated processes can eventually be reduced to a few minimal stages or eliminated entirely.

There is an artificial comfort in gated processes.

There is an artificial comfort in gated processes: one feels secure because the gates are in place, but the comfort is naïve because the gates do not address the underlying reason the gates were created in the first place: that those who are building systems either do not know how to implement the compliance and risk management requirements, or they are not testing sufficiently for these things. Learning organizations move past this dilemma by ensuring that there is a much broader understanding of the requirements and how to test for them.

Agile transformation is really about systematically creating a learning organization. You have to identify the things that people need to know, and make sure that there are people who know those things embedded in the development process, by creating a system for people to learn and share that knowledge (here is one approach). Ideally, everyone knows everything, but that is not practical, so there is a balance that needs to be achieved between specialists and generalists. But all need to be involved in real time or near real time.

The chart at the end of this article lists some of the things that each part of the IT organization will need to learn. As you can see, it is a-lot, and that is why learning – not process re-engineering – is the “long pole in the tent” for Agile transformation. Learning is one of the first steps on the long road of changing to an Agile culture.

Conclusions

Morticia’s website and the EFT Management Portal are two extremes – as business systems go. Most business applications are somewhere inbetween. That is why there is no single answer to how one should plan and execute testing under Agile.

Trust the team – that works fine for Morticia’s website. Top down planning – quite a bit of that is needed for the EFT Management Portal, although we try to approach it in an Agile way by putting the team in the driver’s seat, by keeping the documentation light, by allowing things to evolve, to a degree, by doing testing repeatedly and with as much automation as possible, and by implementing a learning strategy built around coaching to help teams to learn what they need to know to address all of the things that need to be tested.

In the case of the EFT Management Portal we also found that external parties demand – have a right to – oversight, and their risk management team will want to talk to our development leaders – hence we need to have development team leaders if only for the purpose of interfacing to these risk management folks – and the risk management folks will want to review our testing strategies and the ways that we measure test coverage. They will also be watching closely when our first release is deployed for demonstration: by that time, we should have already tested the application at scale in our cloud based test environment and so we should know what the outcome will be – there should be no surprises – but the first release is still a major visible milestone, even if it is not a production release, and people’s credibility rides on it to a large degree.

The fact that credibility rides on a first release is cultural – but it is also human, and to a large extent independent of culture – and so even though Agile encourages tolerance for failure, that has its limits: the idea is to fail early to learn so that you do not fail when it counts and everyone is watching – including the risk management people who are tasked with finding out if you know what you are doing. Testing is Agile’s primary tool for ensuring early failure (feedback), so it is crucial to do the early planning needed to make sure that testing is thorough.

One sign that your organization is embracing early contained failure as a strategy for ensuring long term success is when teams start using demonstrations as a way of “proving” that compliance requirements are met. Teams often look forward to being able to do this. This then continues to strengthen trust within the organization. The more an organization implicitly trusts its teams, the less process rigor is needed – but this will only occur if continual learning is implemented as a strategic way of ensuring that the teams have and continue to have the knowledge that they need.

Authors (alphabetically):
Scott Barnes
Cliff Berg

As PDF.