AWS Notes
Qwiklabs
How can I learn about more AWS topics using Qwiklabs?
- Create an account at https://run.qwiklabs.com
- Search for labs that look interesting
- Many labs here are free
3D Diagrams of AWS Stacks
How can I draw neat 3D diagrams of AWS stacks?
Predictive Auto Scaling
Is it possible to use machine learning to do intelligent predictive auto scaling?
- Scryer is Netflix’s engine for using machine learning for predictive auto scaling—read about how Netflix uses Scryer.
- http://techblog.netflix.com/2013/11/scryer-netflixs-predictive-auto-scaling.html
Auto Scaling Containers
How can I auto scale containers on ECS?
Containers on Spot Instances
How can I run ECS containers on spot instances?
WorkSpaces and VPC
Can I spin up WorkSpaces in my pre-existing VPCs?
- Set up AD directory service in your VPC. Then attach WorkSpaces to this directory. The WorkSpaces instance will be in the same VPC.
- For more details on this follow the steps for the “Advanced Setup” in this guide to set up WorkSpaces in your pre-existing VPC.
- http://docs.aws.amazon.com/workspaces/latest/adminguide/getting_started.html
AD Connectors
How can I set up AD connectors on AWS?
EMR Sizing
How can I size my EMR cluster based on my data?
Parallel EMR Steps
How can I run EMR Steps in parallel?
- Use EMR Activity in Data Pipeline
- http://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-object-hadoopactivity.html
Data Pipeline and Existing EMR
How can I use an existing EMR cluster with Data Pipelines?
Spark and YARN
How can I optimize Spark on YARN?
- Here is the link to the talk by Sandy Ryza about how to optimize Spark on YARN.
- https://spark-summit.org/2014/spark-on-yarn-a-deep-dive/
- https://spark-summit.org/2014/wp-content/uploads/2014/07/Spark-on-YARN-A-Deep-Dive-Sandy-Ryza.pdf
Redshift and DynamoDB
How can I parse JSON when I upload data into Redshift?
- You can parse JSON in COPY command
- http://docs.aws.amazon.com/redshift/latest/dg/copy-usage_notes-copy-from-json.html
- http://docs.aws.amazon.com/redshift/latest/dg/r_COPY_command_examples.html
SSH Timeout
How can I prevent my SSH connection to EC2 from timing out?
-
To prevent SSH from terminating after 60 seconds put this line in
~/.ssh/config
ServerAliveInterval 50
Viewing EC2 Instance Logs
How can I see EC2 instance syslogs in CloudWatch?
- Set up a CloudWatch Logs agent on the EC2 instance
- This agent will push your logs into CloudWatch Logs
- Quick walkthrough steps: https://aws.amazon.com/blogs/aws/cloudwatch-log-service/
- Details: http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_GettingStarted.html
DynamoDB Encryption
Can you encrypt data in DynamoDB?
- DynamoDB supports client-side encryption and does not support server-side encryption
- Details: https://aws.amazon.com/blogs/developer/client-side-encryption-for-amazon-dynamodb/
S3 Key Salting
Why doesn’t S3 automatically salt the key names with hash prefixes instead of making customers do this?
- The keys are stored in contiguous partitions based on lexicographic order
- Lexicographic order allows S3 to easily return all the files inside a folder (everything with the same prefix).
Billing and Cost Management
How can I find out my billing reports?
- Trusted Advisor
- Click on Services → Trusted Advisor
- This will show you if you have any idle EC2 instances or RDS instances
- It runs across all regions
- Billing & Cost Management
- Click on: Your user name (top right corner) → Billing & Cost Management
- Or go directly to: https://console.aws.amazon.com/billing/home#/
- This will show you your current bill and projected bill for next month
- You can enable more detailed billing by storing data into S3
Lambda Hello World
How can I write a simple hello world function using Lambda that is accessible from the web?
- Configure IAM permissions
- Use the code below for the function
- Configure API gateway to point to the function
def lambda_handler(event, context):
print "Hello world from lambda"
return {
'statusCode' : 200,
'headers' : {
'x-custom-header' : 'my custom header value'
},
'body' : 'Hello, world!'
}
Minecraft on AWS
How can I set up Minecraft on AWS quickly using a CloudFormation template?
- This is an example Minecraft CloudFormation template
- https://raw.githubusercontent.com/zcrowell/paxdevdemo/master/CloudFormation/MineCraft_Server.json
Networking Internals
How do the internals of AWS networking work?
- A Day in the Life of a Billion Packets
- https://www.youtube.com/watch?v=Zd5hsL-JNY4
Referencing Security Groups Across VPCs
Can I use security groups across peered VPCs to allow groups of instances from one VPC to access particular ports in another VPC?
- You can reference security groups across VPCs
- Details: http://docs.aws.amazon.com/AmazonVPC/latest/PeeringGuide/working-with-vpc-peering.html#vpc-peering-security-groups
- Announcement from March 2016: https://aws.amazon.com/about-aws/whats-new/2016/03/announcing-support-for-security-group-references-in-a-peered-vpc/
Public IPs and ifconfig
Why does ifconfig -a not show the public IP address of an EC2 instance and only the private IP address?
- Public addresses are set on the IGW.
- Excerpt from docs: “The Internet gateway logically provides the one-to-one NAT on behalf of your instance”
- Details: http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Internet_Gateway.html
Security Groups and VPCs
Can a security group span multiple VPCs?
- Security groups belong to a VPC so they cannot span multiple VPCs
Static Private IPs
How can I make sure that my EC2 instance maintains the same private IP address even if it crashes?
- Enable auto-recovery on your EC2 instance
- If the instance terminates this restart it with the same IP addresses, same instance ID, and the same metadata
Route 53 and DNSSEC
Does Route 53 support DNSSEC?
- Amazon Route 53 supports DNSSEC for domain registration but does not support DNSSEC for DNS service
- If you want to configure DNSSEC for a domain that is registered with Amazon Route 53, you must use another DNS service provider.
- http://docs.aws.amazon.com/Route53/latest/DeveloperGuide/domain-configure-dnssec.html
VMware Templates and AMIs
Can I convert a VMware template into an AMI?
- You can import VM images as AMIs
- Use the
vm-import
service to do this - http://docs.aws.amazon.com/vm-import/latest/userguide/vmimport-image-import.html
- https://aws.amazon.com/ec2/vm-import/
Restricting EC2 Permissions by VPC
Is it possible to use IAM permissions to allow users to spin up EC2 instances inside a Dev VPC while preventing them from spinning up EC2 instances inside the Prod VPC?
- https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-supported-iam-actions-resources.html
- Scroll down to “RunInstances”; it shows you what resources/keys you can use in the policy
arn:aws:ec2:region:account:subnet/*
is one of the resources you can restrict on
RDS and HIPAA
What RDS databases can I use if I want to be HIPAA compliant and have a Business Associate Agreement with Amazon?
- You can use MySQL and Oracle (databases must use encryption): https://aws.amazon.com/compliance/hipaa-compliance/
- Details on how to write HIPAA compliant applications on AWS: https://d0.awsstatic.com/whitepapers/compliance/AWS_HIPAA_Compliance_Whitepaper.pdf