Working on Real Projects: AWS Beyond the Certifications (1/10)
It requires some skill to mug up the names of AWS services. Yet another to clear certifications. However, building well-architected enterprise-scale applications requires a different metal. I have seen many newcomers spend themselves acquiring the first two and find themselves in a mess when they face the real world.
It is not rocket science, but it requires a different mindset, some training, and a lot of learning. A lot of good training material is available on the internet. However, it is buried beneath the SEO-optimized content that talks of nothing beyond the certifications.
This series of blogs is based on my experience and interaction with such projects. I have gathered a list of common points that can help you avoid problems and simplify the process. Such a list can not be complete. Please append your experiences in the comments.
The series will cover the following topics:
- Creating the AWS Account.
- Protecting the AWS Account.
- Setting up the development environment.
- Infrastructure as Code — you need it.
- Servers, Containers, or Lambda?
- Everything fails all the time.
- Event-driven architecture.
- Monitoring — now or never.
- Reduce your bills.
- Data — cost, complexity, and opportunity.
Creating the Account
Let us start with day zero. You have a brilliant idea, and you want to implement that. Being wise, you naturally choose to build it on AWS. You gather a team of bright young minds who cracked the AWS certification and are eager to work on the new start-up. Now, you need an AWS account so that you can dive in. So you tell that motivated fresher to go ahead and create that account quickly.
Wait! The mistakes begin at this point.
How do you create that AWS account? Which plan do you choose? Which country do you specify? How many AWS accounts should you create? How do you manage access to those accounts? How do you manage passwords? What about the compliances?
Now is the time to think and plan for it. These points are often ignored in the hurry and excitement. “We can check that later.” And that later never arrives. If you procrastinate now, you will soon end up in a mess that is impossible to resolve.
Which Country?
AWS has a similar infrastructure across the globe. Any account can use services from any region. However, the legal constraints of each country enforce some differences in the billing structure. Each country has its own taxation rules, and your bills can vary depending on where you register the account. Some of the services are blocked in some countries.
Once your account is registered in one country, it is difficult (if not impossible) to migrate it to another. This can cause problems, especially if you hope to sell your work someday.
Sometimes, we do not have a choice, so we have to live with this problem. However, if you have a choice (with an international team), do some research and make a smart choice about the country where you register the AWS account.
Root User
Great! So you decided on the country and went ahead to create the account. Which Email ID do you use to create the account? Obviously, you don’t want all the spam in your inbox, so you try to delegate the task to someone else. However, make sure you think well before choosing this email id. You should be able to monitor that inbox. Anyone having access to that mailbox can hijack your AWS account.
Anyone accessing that mailbox can reset your password and break into your AWS account. In an unfortunate scenario, if your account is compromised, AWS can detect it. They have algorithms that can identify unusual activity and alert the user by emailing the registered id. You should not miss such emails.
I know a friend who used an unmonitored mailbox to keep his inbox clean. Unfortunately, a hacker found his way into the account and created a fleet of EC2 instances. AWS generated several alarm emails. However, they were ignored; it was too late when he saw the problem.
I like to use group emails for this purpose. That reduces the chance of missing important emails, and even if one person turns into a traitor or tries to misuse the access to reset a password, the others will be able to jump in and protect the account.
Support Plan
You choose the appropriate email id and then complete the account setup. Being a bootstrapped startup, you are short of funds, so you choose the free, basic plan and finally create the account successfully.
The basic plan is good enough to start your work. But once you are ready, don’t forget to upgrade it, at least for a month. The cost is low, and the benefits are pretty high. Of course, you don’t have to stay on the higher plan for all your life.
However, it makes sense to consult the experts for their advice to make sure you are on the right path. There are a lot of things that you are doing for the first time, and the experts have seen every day for the last few years. They will point out some problems you have yet to think of. I have always had a great experience with them.
I was surprised when they told me how to reduce my bills and avoid vendor lock-ins!
Don’t forget to switch back to the basic plan once you are done.
Credentials
Now is the time for the next big mistake! This is a horrible one. But, unfortunately, a lot of educated people do this. Once the account is ready, they shoot out an email to the team — providing the root user credentials.
Yes, you were born lucky. Grace has always saved you from the greatest disasters. However, that does not mean you invite more of them. Never share the root credentials with anyone. You should set up a huge, ugly, and impossible password for the root user. Top it with MFA and save the credentials in the safest possible place. Use the root id only if you must.
Additionally, protect your root account with appropriate alarms.
Even if you are the only user, do not use the root account for your regular tasks. When you use an account frequently, it is more likely to fall into the wrong hands. And if you share the root credentials with the whole team, it will undoubtedly land in the wrong hands.
In an unfortunate case, if an IAM user or SSO user is compromised, you can log in with the root user and delete the IAM/SSO user. However, if the root user is compromised, the only thing you can do is pay the bill!
I had witnessed the chaos when an account was compromised. AWS support was highly cooperative and they helped us in every possible way. That was because we had followed all the prescribed best practices. If you miss out on those, only God can help you.
How many Accounts?
Foremost, understand that any meaningful enterprise needs more than one account. If you are a loaner, just learning AWS, or working on a POC, don’t complicate your life with more accounts. (More accounts imply more risk of getting hacked.) However, you better get organized now if you hope to do anything beyond the POC. It is now or never.
Never, ever have development, test, and production deployments in the same AWS account. That is the key to disaster. Sooner or later, you are bound to face the fire when someone deletes a production resource “by mistake,” thinking it was a test.
Yes, you can have precise access control with granular policies. With extreme caution, you can create the right user groups, resource tags, and policies based on tags. You can ensure that all the account resources are appropriately classified into the correct environments. You can have a smart resource naming and tagging strategy to make sure there are no clashes. With all the effort, you can ensure that all the users can do just what they are supposed to do and nothing more than that.
With rigorous discipline, we can have a secure system with dev-test-prod in the same AWS account. However, I feel it is better to save that rigor for something more useful. Be practical, and don’t test your fate. Just follow the AWS guidelines and stay safe.
Even within the production environment, a multi-account approach has several advantages.
AWS Organization
This is often ignored in most tutorials. It has little weightage in the associate certification exams. So most architects know little beyond the name. If that is you, I suggest you invest a day and get your hands dirty. You will thank yourself a year down the line.
Okay, so how many accounts should we create? You need two accounts on day zero if you have just started your development. The management account and a development account.
Immediately after you have created the first account, and secured the root user, go ahead and Create an organization.
That will take a few seconds to process. Meanwhile, go back to your mailbox and verify the email address for the organization root.
Now, the organization has only one account — the management account. This is the account you created some time back.
Like the root user, the management account should be handled with absolute care. Never use it for anything other than managing the organization. Set up the correct alarms so that we can track any activity there. Check out this tutorial to see how.
AWS is generous — especially with startups. Throughout the development process, you will gather a lot of billing credits. Make sure you apply them to the management account.
Development Account
Once the organization is initiated, we add the development account to it. Navigate to the Organization accounts page and “Add an AWS Account.”
Select “Create an AWS Account.” Again, we must provide an email id for the new AWS account. Choose this carefully, as discussed above. Give a name to this account — Development. This will take a few minutes to create. Once it is done, you can start developing using this account.
You will surely mess up its configuration in the development process. It is best to keep that mess isolated. You can create testing, staging, and production accounts when your application matures.
Multiple Accounts per Stage
Splitting it into multiple accounts per stage makes sense if you build a more extensive application.
1. Security:
Different parts of the enterprise have different security requirements. Some are inherently more sensitive than others. We can simplify an auditor’s (and hence our own) job by separating them. For example, if all the credit card data can be held in an isolated account, that reduces the auditor's work. We are happy if the auditor is happy.
Isolating data in different AWS accounts can simplify the process and ownership. An account boundary isolates different units of the application. Thus, any risk or loss can be contained within the isolated unit.
2. Multiple Teams:
More often than not, enterprise-scale applications are developed and maintained by multiple teams. Each of them has different responsibilities. They require different resources, and they follow different processes.
Business units or products often have entirely different purposes and processes. Individual accounts can be established to serve business-specific needs.
If we have multiple accounts for the different teams, we can reduce the chaos of teams interfering with one another. Each team can own its policies and processes without sacrificing anything for account-level discipline.
3. Billing:
An account is the simplest way to split the bills among components. We can have billing tags that help us identify the cost breakdown. However, that requires more effort than the benefit it provides. Of course, it is easy to “forget” a tag to stay within the budget.
Multiple accounts make it easy to enforce discipline. We can have quotas and detailed tracking. At the same time, we can use organizations to have centralized billing.
Please watch this video to understand organizations in detail.
Identity Center (Single Sign-on)
When you have too many accounts, naturally, you have too many passwords to remember. Naturally, we are tempted to reduce the stress on our memory by choosing simpler passwords, which is wrong. Also, it is a difficult task for the administrator to track and manage multiple users in multiple accounts. It gets messy very soon.
Single Sign-on is a convenience feature that helps us solve this problem. We can create granular groups and users with specific access to individual AWS accounts.
It requires some effort for the admin to set up. However, like most of the points above, you should invest in this. A few hours at this time will save you a lot of trouble and risk in the years to come.
AWS Activate
If you are a startup working on a promising application, AWS would be happy to help you begin. They are very generous with startups. Once you have the basics ready, apply for help from AWS Activate.
Not just free credits, you will get help on several other fronts. It makes sense to understand the opportunities they provide. Invest a few hours reading through it, and chat with them to understand more possibilities.
They are eager to hand-hold you to success.
