Every Security team should write down their Security Standards and Patterns
I have the privilege of working with a startup like company that has gone through a hypergrowth phase over the last 5 years. When I was hired, I was the 2nd Application Security Engineer in a 9 person Security team . We’re now a ~70 person Security org made of 7 different teams that are responsible for various aspects of securing our member’s data and the company.
In this post I plan on sharing a few things I’ve learned and observed over the last 5 years that I wish someone had told me when we were a small Security team. Most of these observations were more prominent in 2020 because of COVID-19 and the entire workforce moving to a work from home situation. The company I work for has always preferred in-person collaboration(which I really enjoy) and rarely allowed remote work. But when the pandemic hit that model really affected us as most of the tools and processes that were built was not built for employees working remotely.
While there are many aspects I’d like to share in this post I’m going to write about the need for documenting security standards, patterns and concepts. I realize that this might not be the most “fun” or “shiny” thing to write about, but I do strongly feel this is important. I also realize that a lot of this might sound like “duh” but it’s surprising how very few security teams do it right. Having your security requirements, standards and patterns written down helps in ways that’s not often spoken about or is given credit for. While it’s true that we should be building enough self service tools, libraries and automation in place that reduces the number of decisions every engineer needs to make, that does take time and does not substitute for having your standards and patterns written. I also think it’s a mindset change for the most part — When we were a 3 person Application Security team, I very much had the mindset that I rather spend my time finding and fixing vulnerabilities or building libraries than document standards or patterns, which in retrospect was setting myself and the team for failure.
Let me define what I call a Standard, Pattern and a Concept:
Standard — A list of security/engineering requirements on specific topic.
Pattern — A “how to” guide on implementing a solution based on security standards.
Concept — A high level document that explains a system/process without necessarily having to read an entire Technical Design Document (TDD).
Why should a 3 person team spend time writing these?
- Do you find yourself repeating? — When you find yourself repeating the exact same patterns and requirements to more than one team, it’s a sign that you got to document those requirements or patterns so that it’s available for other teams solving for the same issue. It’s also a sign that some of these are things one should be getting into the framework — for example teams shouldn’t necessarily have to worry about how to securely parse a file (pdf, xml etc) or how to transform data to ensure there’s no sensitive data being ETL’d to the data warehouse etc. Documenting these in a central repository instead of providing them ad-hoc on tickets/documents act as a stopgap solution which helps you get to working on writing those libraries or work on other interesting/demanding problems.
- Have one minute hallway conversations been replaced with meetings? — Something that I realized when we all started working remotely is that most of the days, my calendar was filled with 15 or 30 minute by folks that wanted to know what are security requirements for doing X? I kept asking myself why isn’t this already documented and started to document them. In retrospect I realize I’ve been always answering such questions to my colleagues over hallway conversations or at the kitchen but I didn’t associate it as a lack of documentation issue then — but now, those hallway conversations have become meetings and something that should have taken 2 minutes to answer now take up a lot more of my time.
- Most of the knowledge on accepted patterns is only known by a small list of individuals— Given that the team was small and didn’t prioritize on documenting this knowledge, when your company is in hypergrowth phase, there are individuals that are bottlenecks across the entire organization as they are required to be involved in designs. As the company grows and has offices in multiple locations, enforcing standards and educating people becomes challenging. Different engineering teams often will come up with different solutions for the same set of problems. In order to not reinvent the wheel, document the approved patterns.
- Makes you realize you’re being reasonable with your engineers — Working with engineers on these standards, patterns and concepts makes you become more reasonable and not an alarmist. Working with engineering on this also ensures that the requirements and standards which are being set are things that they have agreed to. If a pattern or a standard needs to be updated they take ownership of proposing a change as they have been involved in getting these documented in the first place.
- You actually figure out what are the “musts” — When documenting standards, you figure out what are things that actually matter. I’ve seen that this makes things very transparent to product and engineering teams. Requirements are no longer a surprise to them — this ensures they design services and features with these in mind.
- Consistency in your reviews — It ensures that any security engineer performing a review has a runbook or a checklist to follow thereby keeping reviews consistent. Having your standards and patterns documented also helps ensure designs are consistent and no one is reinventing the wheel — Engineering teams are incentivized to use a security approved pattern as it reduces the number of reviews and lets them move quickly.
- Helps onboard new hires — Often companies that are going through a hypergrowth phase hire in huge numbers without realizing that they lack the necessary documentation and processes to onboard these individuals. How quickly a new hire is able to contribute back directly relates to how well they are on-boarded. Having these standards, patterns and concepts documented help in onboarding a new hire.
- You realize what functions of engineering aren’t really given the same level of scrutiny — When I was documenting standards and patterns for the company I’m currently in, I realized that we’ve spent most of the time securing the cloud, platform services and the product/application. As a Security team, we’ve spent way less time on how data engineering, native clients and IT operate. It wasn’t that we didn’t have contact with those teams or that we didn’t review their design and code, it’s just that we haven’t reviewed those teams as much as we’ve spent time reviewing platform or product engineering teams for us to have common standards and patterns documented.
Learnings and observations:
- Standards don’t often need an explanation — Keep them clear, concise but comprehensive. If someone wants to get more details about why a requirement was set or challenges a standard they should — but you should setup a different medium for that. Details about how can one dispute content must be made available on the portal that hosts this content.
- Keep your content relevant to your company — Do not duplicate content from OWASP cheat sheets or Kubernetes/Cloud provider best practices whitepapers. You can definitely link them in your Appendix or resources section but if your content is filled with generic best practices and things not relevant to your company’s posture your documentation is no longer relevant to your engineers.
- Writing security standards is the Security Engineering team’s responsibility. Not compliance’s — I definitely have worked for companies where writing down security standards and policies were a Governance and Compliance team’s job. In order for you to be successful with this endeavor one needs to know the reality of how engineers in that company work and what controls exist. In most companies, the security engineering team is best equipped to do this!
- Add a contributing guide. Make this really simple — Let’s face it, you’re not going to be anywhere successful without opening up the platform to the rest of engineering to contribute. If you make it simple enough, people will contribute. Choose a platform that supports simple markdown files in a directory like structure which can be edited via the SCM’s UI and doesn’t necessarily need a code editor to write documentation. Have recommendations and guidelines on the standards language, how to add architecture diagrams and tables for patterns/concepts etc.
- Get your engineering peers to review and contribute — Add your trusted engineering counterparts or security champions as reviewers/approvers on this repo. This ensures all the right people are being involved and aligned on updates being introduced or updated.
- Whatever platform you choose to host your documentation, make sure it has a great search plugin— it’s going to be the most used feature. Engineers coming to the platform are coming to your documentation platform with a reason — it’s either to figure out the requirements or see if there are patterns for a problem they’re looking to solve for. The way they’re going to get to it is by using the search feature.
- Having tutorials helps — Engineers most often will do the right thing and stick to the standards if they know how to do it. Definitely have secure defaults for any configuration in your framework but for situations where that’s not possible or hasn’t been built out yet, have templates or walkthroughs/tutorials on how to do it the right way. Again, keep it simple and to the point.
- Organize content based on topics — How you organize your content is definitely one of the most important factors on getting your engineers to use this portal. Instead of organizing your content as Standards, Patterns and Concepts as three different sections and having topics under them, in all reality, engineers do not necessarily know if there are patterns for a topic/problem they’re dealing with — instead break the sections as topics and have standards, patterns and concepts as sub-sections. For example a topic can be “Secrets Management” and you can have sub-sections that describe standards and patterns on how to handle secrets. This is how GCP does it; this model works.
- Policy as code — There’s very little value in preaching “Thou shalt not do X” in a document somewhere when in reality one can write down policies for all structured configurations like Kubernetes configurations, Terraform code, Serverless configurations or any other structured data as CI job. I’ve written down most of my standards as code in rego and I use OPA for validating changes against my standards. Letting engineers know about a misconfiguration right when the change is being introduced drastically helps reduce a misconfiguration and keeps them closer to the standard. I’ve written more about it in a different blog post here.
- Recognition and Gamification — To gain the initial momentum of getting engineers using this portal, having a leaderboard of who contributed the most helps motivate engineers to contribute more often. Sending swag/coupons or recognizing these contributors/moderators publicly in a wide audience also really helps motivate engineers to contribute more. Also having metadata about who authored a post and when was it last modified helps a consumer know how updated/relevant the content is. I’ll be honest that this won’t last long unless you keep iterating with different ideas (which can be a full-time job) to keep your engineers engaged with adding new content. Don’t be disheartened if your efforts on gamification fails. It’s not an easy problem.
- If it’s not part of your job description the content won’t get updated — If keeping this content updated is not part of security engineering team’s responsibility the content will get outdated quickly and this portal won’t be relevant anymore. While we do need engineers to help contribute, ensure that this is part of security engineers daily responsibilities. Something else that has worked for me is adding a contribution metric as a criteria for an engineer’s promotion (if they’ve contributed). I do realize that this might not work out in all organizations.
So was this helpful to my organization? Definitely. Here are some statistics on how this improved our engagement and reduced friction with Engineering - Prior to having these documented, an engineering team would spend an average of two 30 minute sessions with Security on getting a design reviewed; one for consult (where they got the requirements and asked for if there were patterns of other teams solving similar issues) and one for design review (where a TDD was reviewed). By documenting the requirements and patterns, in a matter of 3 months, we’ve had ~70% of the teams who now skip the consult session for discussing the requirements or patterns. While we do still have some teams that come and ask us for requirements and suggestions to some very specific designs, the amount of total time spent on reviewing a design has come down drastically. Here’s another interesting statistic on the consistency of these reviews— because not all the security engineers had the same context, we’ve had cases in the past where we’ve had to change the requirements of a design after it was signed off. On an average we’ve had about 6 designs per quarter that needed re-review. Post launch of this portal, we’ve only had 2 cases of that happen.
I’m going to conclude by emphasizing that you don’t necessarily have to be a big company to get to documenting these down. If you’re a security team that’s trying to reduce the friction with engineering or reduce the number of reviews one needs to go through or if you’re trying to automate the boring aspects of the job (or all the above) and you don’t currently have your standards and patterns documented (or haven’t updated them in a while), do it.