AWS Notes

Qwiklabs

How can I learn about more AWS topics using Qwiklabs?

• Create an account at https://run.qwiklabs.com
• Search for labs that look interesting
• Many labs here are free

3D Diagrams of AWS Stacks

How can I draw neat 3D diagrams of AWS stacks?

• https://cloudcraft.co

Predictive Auto Scaling

Is it possible to use machine learning to do intelligent predictive auto scaling?

• Scryer is Netflix’s engine for using machine learning for predictive auto scaling—read about how Netflix uses Scryer.
• http://techblog.netflix.com/2013/11/scryer-netflixs-predictive-auto-scaling.html

Auto Scaling Containers

How can I auto scale containers on ECS?

• http://docs.aws.amazon.com/AmazonECS/latest/developerguide/cloudwatch_alarm_autoscaling.html

Containers on Spot Instances

How can I run ECS containers on spot instances?

• https://aws.amazon.com/blogs/compute/powering-your-amazon-ecs-clusters-with-spot-fleet/

WorkSpaces and VPC

Can I spin up WorkSpaces in my pre-existing VPCs?

• Set up AD directory service in your VPC. Then attach WorkSpaces to this directory. The WorkSpaces instance will be in the same VPC.
• For more details on this follow the steps for the “Advanced Setup” in this guide to set up WorkSpaces in your pre-existing VPC.
• http://docs.aws.amazon.com/workspaces/latest/adminguide/getting_started.html

AD Connectors

How can I set up AD connectors on AWS?

• https://aws.amazon.com/blogs/security/how-to-connect-your-on-premises-active-directory-to-aws-using-ad-connector/

EMR Sizing

How can I size my EMR cluster based on my data?

• http://docs.aws.amazon.com//ElasticMapReduce/latest/ManagementGuide/emr-plan-instances.html

Parallel EMR Steps

How can I run EMR Steps in parallel?

• Use EMR Activity in Data Pipeline
• http://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-object-hadoopactivity.html

Data Pipeline and Existing EMR

How can I use an existing EMR cluster with Data Pipelines?

• http://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-how-task-runner-user-managed.html

Spark and YARN

How can I optimize Spark on YARN?

• Here is the link to the talk by Sandy Ryza about how to optimize Spark on YARN.
• https://spark-summit.org/2014/spark-on-yarn-a-deep-dive/
• https://spark-summit.org/2014/wp-content/uploads/2014/07/Spark-on-YARN-A-Deep-Dive-Sandy-Ryza.pdf

Redshift and DynamoDB

How can I parse JSON when I upload data into Redshift?

• You can parse JSON in COPY command
• http://docs.aws.amazon.com/redshift/latest/dg/copy-usage_notes-copy-from-json.html
• http://docs.aws.amazon.com/redshift/latest/dg/r_COPY_command_examples.html

SSH Timeout

How can I prevent my SSH connection to EC2 from timing out?

• To prevent SSH from terminating after 60 seconds put this line in ~/.ssh/config

    ServerAliveInterval 50
  

Viewing EC2 Instance Logs

How can I see EC2 instance syslogs in CloudWatch?

• Set up a CloudWatch Logs agent on the EC2 instance
• This agent will push your logs into CloudWatch Logs
• Quick walkthrough steps: https://aws.amazon.com/blogs/aws/cloudwatch-log-service/
• Details: http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_GettingStarted.html

DynamoDB Encryption

Can you encrypt data in DynamoDB?

• DynamoDB supports client-side encryption and does not support server-side encryption
• Details: https://aws.amazon.com/blogs/developer/client-side-encryption-for-amazon-dynamodb/

S3 Key Salting

Why doesn’t S3 automatically salt the key names with hash prefixes instead of making customers do this?

• The keys are stored in contiguous partitions based on lexicographic order
• Lexicographic order allows S3 to easily return all the files inside a folder (everything with the same prefix).

Billing and Cost Management

How can I find out my billing reports?

• Trusted Advisor
  • Click on Services → Trusted Advisor
  • This will show you if you have any idle EC2 instances or RDS instances
  • It runs across all regions
• Billing & Cost Management
  • Click on: Your user name (top right corner) → Billing & Cost Management
  • Or go directly to: https://console.aws.amazon.com/billing/home#/
  • This will show you your current bill and projected bill for next month
  • You can enable more detailed billing by storing data into S3

Lambda Hello World

How can I write a simple hello world function using Lambda that is accessible from the web?

• Configure IAM permissions
• Use the code below for the function
• Configure API gateway to point to the function

def lambda_handler(event, context):
    print "Hello world from lambda"
    return {
        'statusCode' : 200,
        'headers' : { 
          'x-custom-header' : 'my custom header value' 
        },
        'body' : 'Hello, world!'
    }

Minecraft on AWS

How can I set up Minecraft on AWS quickly using a CloudFormation template?

• This is an example Minecraft CloudFormation template
• https://raw.githubusercontent.com/zcrowell/paxdevdemo/master/CloudFormation/MineCraft_Server.json

Networking Internals

How do the internals of AWS networking work?

• A Day in the Life of a Billion Packets
• https://www.youtube.com/watch?v=Zd5hsL-JNY4

Referencing Security Groups Across VPCs

Can I use security groups across peered VPCs to allow groups of instances from one VPC to access particular ports in another VPC?

• You can reference security groups across VPCs
• Details: http://docs.aws.amazon.com/AmazonVPC/latest/PeeringGuide/working-with-vpc-peering.html#vpc-peering-security-groups
• Announcement from March 2016: https://aws.amazon.com/about-aws/whats-new/2016/03/announcing-support-for-security-group-references-in-a-peered-vpc/

Public IPs and ifconfig

Why does ifconfig -a not show the public IP address of an EC2 instance and only the private IP address?

• Public addresses are set on the IGW.
• Excerpt from docs: “The Internet gateway logically provides the one-to-one NAT on behalf of your instance”
• Details: http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Internet_Gateway.html

Security Groups and VPCs

Can a security group span multiple VPCs?

• Security groups belong to a VPC so they cannot span multiple VPCs

Static Private IPs

How can I make sure that my EC2 instance maintains the same private IP address even if it crashes?

• Enable auto-recovery on your EC2 instance
• If the instance terminates this restart it with the same IP addresses, same instance ID, and the same metadata

Route 53 and DNSSEC

Does Route 53 support DNSSEC?

• Amazon Route 53 supports DNSSEC for domain registration but does not support DNSSEC for DNS service
• If you want to configure DNSSEC for a domain that is registered with Amazon Route 53, you must use another DNS service provider.
• http://docs.aws.amazon.com/Route53/latest/DeveloperGuide/domain-configure-dnssec.html

VMware Templates and AMIs

Can I convert a VMware template into an AMI?

• You can import VM images as AMIs
• Use the vm-import service to do this
• http://docs.aws.amazon.com/vm-import/latest/userguide/vmimport-image-import.html
• https://aws.amazon.com/ec2/vm-import/

Restricting EC2 Permissions by VPC

Is it possible to use IAM permissions to allow users to spin up EC2 instances inside a Dev VPC while preventing them from spinning up EC2 instances inside the Prod VPC?

• https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-supported-iam-actions-resources.html
• Scroll down to “RunInstances”; it shows you what resources/keys you can use in the policy
• arn:aws:ec2:region:account:subnet/* is one of the resources you can restrict on

RDS and HIPAA

What RDS databases can I use if I want to be HIPAA compliant and have a Business Associate Agreement with Amazon?

• You can use MySQL and Oracle (databases must use encryption): https://aws.amazon.com/compliance/hipaa-compliance/
• Details on how to write HIPAA compliant applications on AWS: https://d0.awsstatic.com/whitepapers/compliance/AWS_HIPAA_Compliance_Whitepaper.pdf