Mike Rahmati: My Serverless journey with Cloud Conformity
Mike Rahmati is the Head of the Advisory Board at Dashbird. He is the Co-Founder and CTO of Cloud Conformity (acquired by Trend Micro) – a Cloud Security Posture Management Solution – one of the largest and earliest adopters of serverless.
Mike is also an active AWS Community Hero. In this article, he shares his journey and experience with serverless.
Cloud Conformity was founded in 2014 as a result of our own experience of issues migrating to the cloud. It was clear early on that as a startup, resources like DevOps would be limited, which is where utilizing serverless made simple sense – we could offload most activities to AWS allowing us to focus on the business itself.
The serverless ecosystem was very young back then, and a few mistakes were made and learned from along the way. Most notably that we essentially had one fat Lambda for a while, which later needed redesigning and re-architecting! Today, we have over 1,000 Lambda functions within our architecture, use AWS Step Functions to stitch it all together, AWS API Gateway to expose the backend to customers and use many of the other commonly used AWS Serverless services like DynamoDB, S3, ElasticSearch and KMS.
It’s been quite the journey for us but we are now one of the largest and earliest adopters of serverless.
Cloud Conformity and serverless
From my own experience and meeting with many others in the community, I fully believe that serverless is the new normal and it’s the future for most organizations. When competing in business, speed is king whether that’s for new product development or better user engagement, however, of course, this doesn’t mean serverless is at the core of the business itself.
At Cloud Conformity, serverless wasn’t our main business but we heavily relied on it and therefore needed a tool that could help so we could continue to focus on business logic.
As a Cloud Security Posture Management (CSPM) Solution, Cloud Conformity helps customers: to detect misconfigurations in your cloud architecture providing the rationale behind good and bad practices, to correct the misconfigurations, and prevent them or others from happening again. To produce production-quality code in serverless, considerations for reliability, cost and security must be front of mind and I’d always recommend getting these in place as soon as possible. The fundamentals, such as using AWS X-Ray and the rule of least privilege with IAM, are things people often overlook until it’s too late.
The serverless challenge
We knew we needed to have speed on our side because of market conditions and serverless enabled this with quick builds going from prototype to market in remarkable speed, however, as I’ve learned myself and seen from others, serverless architecture scales much faster than you expect.
The tipping point when serverless starts to work at scale is much more about how the services are running, as opposed to the number of components. For example, a bank in Australia uses serverless as part of their Amazon Alexa service, giving customers the opportunity to request details of their bank account. The service uses a Lambda function that could be invoked thousands of times a day, prompting the need for scalability and the need to know if it ever failed.
I like to see serverless like Lego; it’s important to have a solid foundation with correct configurations from the start. Having this strong base means that when, inevitably, many other separate, small, and potentially dense components are brought together, there is cohesion and it becomes easier to manage. With so many moving parts it’s quite a different architectural model to the traditional, as over a period of just a few days, you could expand to tens of different services for ten different components. This is how easily your serverless footprint could grow.
For smaller footprints or those only used for prototyping, using native AWS services may be enough, however, once you’re at a handful of serverless components with other services like AWS S3 or AWS CloudTrail linked to them, your infrastructure will grow exponentially and observability tools become increasingly important.
As an AWS Community Hero and part of the Serverless Leadership team, I see this issue raised time again with many people overlooking the importance and business value in instilling good practice and underestimating how quickly serverless architecture scales due to its very nature.
One of the challenges of running and growing an unknown, bleeding edge platform is that there will always be gaps in there, which you don’t know about and even when you do find out, the problem might not ever have been solved before, meaning it comes down to needing a tool, like Dashbird, to identify the issue early on.
We knew we needed a monitoring and observability tool for our growing serverless infrastructure, but didn’t want to add strain to our own resources to build this.
Dashbird came out the winner in our trials because of its simplicity and agentless nature. It wasn’t intrusive and didn’t need any extra code changes to work. Instead, it ran side-by-side with our existing cloud infrastructure footprint, with cross-account access using a dedicated IAM role assigned that could easily be removed at any time. The straightforward dashboard and automated alerts meant that we were always aware of any failing statuses or issues, which allowed for much quicker fixes and fewer affected customers.
As Cloud Conformity developed and the demand for cloud grew, the importance of ‘Shift Left’ became clearer. Preventing failures before deployment and essentially finding these errors earlier in the CI/CD pipeline is where the cloud is already moving, and I believe serverless will too. In addition, there is still a gap in full end-to-end monitoring within the space.
Keep an eye on these two elements as I’m predicting more will happen and soon.