INFRASTRUCTURE AS CODE: AWS EDITION

As a follow on to my script that deploys a cluster of two load balanced Windows servers, installs IIS, and deploys a website for Azure, I created a similar script to do so in AWS. A few things of note that I feel makes AWS’s script better.

  • Certain tasks are non blocking and do not wait for the action to complete. I added wait states in the script to make sure time comparisons are true.
  • AWS actions are much faster. On average in my script it takes Azure 65 seconds to add a VM to a load balancer where in AWS its an average of 2 seconds.
  • AWS’s CLI allows for multiple instance IDs to be provided per command to increase efficiency even though my script doesn’t really take advantage of this which provides a more true comparison since I don’t think Azure’s CLI or PowerShell module allows for this.

Here is the script:

#This script creates a number of Windows VMs, installs IIS, a simple webpage, and places them behind a load balancer
#run this if needed
#aws configure
function elapsedTime {
    $CurrentTime = $(get-date)
    $elapsedTime = $CurrentTime - $StartTime
    $elapsedTime = [math]::Round($elapsedTime.TotalSeconds,2)
    Write-Host "Elapsed time in seconds: " $elapsedTime -BackgroundColor Blue
}
#Captures start time for script elapsed time measurement
$StartTime = $(get-date)

#Sets "Constants" to be used to create VMs
$imageID = "ami-0182e552fba672768" #Amazon's provided windows 2019 datacenter base
$subnet = "subnet-000000000" #my subnet for us-east-2a
$securityGroup = "sg-00000000" #My network security baseline
$instanceType = "t2.medium" #Instance size, 2 vCPUs, 4 GB RAM
$keyPair = "ServerHobbyist" #Keypair to retreive administrator password
$instanceName = "WebWinApp" #sets base name
$class = "disposable" #sets class tag as disposable for easier identification and cleanup

#creates load balancer
$lbName = "WinWebLB1"
Write-Host "Creating Loadbalancer $($LbName)"
aws elb create-load-balancer `
    --load-balancer-name $lbName `
    --listeners "Protocol=HTTP,LoadBalancerPort=80,InstanceProtocol=HTTP,InstancePort=80" `
    --subnets $subnet `
    --security-groups $securityGroup
aws elb add-tags --load-balancer-name $lbName --tags Key=Class,Value=$class #tags elb with disposable class
aws elb configure-health-check --load-balancer-name $lbName --health-check Target=HTTP:80/,Interval=5,UnhealthyThreshold=2,HealthyThreshold=2,Timeout=3 #sets a lower threshold for health checks
Write-Host "Load Balancer $($Lbname) created"
elapsedTime


$serverCount = 2 #how many VMs to deploy
$instancesDeployed =  New-Object System.Collections.Generic.List[System.Object] #creates array list that will contain instance IDs deployed
for ($i=1; $i -le $serverCount; $i++){
    
    $instanceNameTag = $instanceName + $i
    Write-Host "Creating VM $($instanceNameTag)"
    $instance = aws ec2 run-instances `
        --image-id $imageID `
        --count 1 `
        --instance-type $instanceType `
        --key-name $keyPair `
        --security-group-ids $securityGroup `
        --subnet-id $subnet | ConvertFrom-Json
    aws ec2 create-tags --resources $instance.instances.InstanceId --tags Key=Name,Value=$instanceNameTag #tags instance with name
    aws ec2 create-tags --resources $instance.instances.InstanceId --tags Key=Class,Value=$class #tags instance with name

    $instancesDeployed.Add($instance.Instances.InstanceId)
    Write-Host "VM $($instanceNameTag) created"
    elapsedTime
}
Start-Sleep -Seconds 15

#Checks to make sure each instance deployed from above is in a running state, otherwise it can't recieve the IAM role.
foreach ($instanceDeployed in $instancesDeployed){
    $instance = aws ec2 describe-instances --instance-ids $instanceDeployed | ConvertFrom-Json
    $InstanceTags = $Instance.Reservations.Instances.Tags
    $InstanceName = $InstanceTags | Where-Object {$_.Key -eq "Name"}
    $InstanceName = $InstanceName.Value
    Write-Host "Checking if instance $($InstanceName)  is ready to receive IAM role for SSM"
    while ($instance.Reservations.Instances.State.Name -ne "running") {
        Write-Host "Instance $($InstanceName) not ready. Waiting to check again"
        sleep 5
        Write-Host "Checking if instance $($InstanceName) is ready to receive IAM role for SSM"
        $instance = aws ec2 describe-instances --instance-ids $instanceDeployed | ConvertFrom-Json
    }
    Write-Host "Instance $($InstanceName) is now ready, assigning role"
    elapsedTime
    aws ec2 associate-iam-instance-profile --instance-id $instanceDeployed --iam-instance-profile Name=AmazonSSMRoleForInstancesQuickSetup
    
}
Start-Sleep -Seconds 15
#Checks to make sure each instance deployed from above has the SSM agent. Otherwise commands can't be sent through AWS's orchestration system
foreach ($instanceDeployed in $instancesDeployed){
    $instance = aws ec2 describe-instances --instance-ids $instanceDeployed | ConvertFrom-Json
    $InstanceTags = $Instance.Reservations.Instances.Tags
    $InstanceName = $InstanceTags | Where-Object {$_.Key -eq "Name"}
    $InstanceName = $InstanceName.Value
    Write-Host "Checking if instance $($InstanceName) has receieved the system management agent"
    $ssmTest = aws ssm list-inventory-entries --instance-id $instanceDeployed --type-name "AWS:InstanceInformation" | ConvertFrom-Json
    while ($ssmTest.Entries.AgentType -ne "amazon-ssm-agent"){
        $ssmTest = aws ssm list-inventory-entries --instance-id $instanceDeployed --type-name "AWS:InstanceInformation" | ConvertFrom-Json
        Write-Host "Instance $($InstanceName) has not yet received the SSM agent"
        start-sleep -Seconds 5
    }
    Write-Host "Instance $($InstanceName) has received the SSM agent. Proceeding to next instance or step."
    elapsedTime
}

#Installs IIS and deploys website
foreach ($instanceDeployed in $instancesDeployed){
    Write-Host "Sending command to install IIS and deploy website on $($instanceDeployed)"
    aws ssm send-command `
        --document-name "AWS-RunPowerShellScript" `
        --parameters commands=['Add-WindowsFeature Web-Server; Invoke-WebRequest -Uri "https://serverhobbyist.com/deployment/index.html" -OutFile "c:\inetpub\wwwroot\index.html"'] `
        --targets "Key=instanceids,Values=$($instanceDeployed)" `
        --comment "Installs IIS"
    Write-Host "Command sent to $($instanceDeployed)"
    elapsedTime
}

#adds VMs to load balancer
foreach ($instanceDeployed in $instancesDeployed){
    Write-Host "Registering instance $($instanceDeployed) with LB"
    aws elb register-instances-with-load-balancer --load-balancer-name $lbName --instances $instanceDeployed #registers instance with load balancer
    Write-Host "Registered instance $($instanceDeployed) with LB"
    elapsedTime
}

Write-Host "Checking if website is ready to be served from load balancer"
$lbURL = aws elb describe-load-balancers --load-balancer-name $lbName | ConvertFrom-Json
$lbURL = "http://" + $lbUrl.LoadBalancerDescriptions.CanonicalHostedZoneName
$check = $false
while ($check -eq $false){
try {
    $check = $true
    $result = invoke-webrequest -uri $lbURL -UseBasicParsing -TimeoutSec 20
}
catch {
    $check = $false
    Write-Host "Website failed to load. Trying again"
    Start-Sleep -Seconds 5
}}
Write-Host "Website is now loading at $lbURL"
elapsedTime

Write-Host "Script completed" -BackgroundColor Blue
elapsedTime

Infrastructure as Code: Azure Edition

Since I’ve been using AWS as a hobbyist for about a decade it is the public cloud I am most comfortable with. Lately to expand my horizons I’ve been learning about Microsoft’s take on it with Azure. I hope I’m not too bias since its easier for me to favor AWS since I’ve been using it for so long however my initial take on Azure is not positive. It is slow. I’m working on a comparison in terms of speed between AWS and Azure with the goal of standing up a 2 node load balanced IIS cluster. While I continue to work on making a good comparison write-up here is the code that deploys out the Azure resources. It is written in PowerShell and includes a function that measures completion time.

Import-Module Az
function elapsedTime {
    $CurrentTime = $(get-date)
    $elapsedTime = $CurrentTime - $StartTime
    $elapsedTime = [math]::Round($elapsedTime.TotalSeconds,2)
    Write-Host "Elapsed time in seconds: " $elapsedTime -BackgroundColor Green
}
#Run this to connect to Azure account if needed
#Connect-AzAccount
#Captures start time for script elapsed time measurement
$StartTime = $(get-date)
#Sets "Constants" to be used throughout script
$resourceGroup = "DisposableLab"
$location = Get-AzLocation | Where-Object {$_.DisplayName -like "North Central US"}
$vnet = "vnet1"
$subnet = "default"
$securityGroup = "DisposableLabSecurityGroup"
$secpasswd = ConvertTo-SecureString "password" -AsPlainText -Force
$credential = New-Object System.Management.Automation.PSCredential ("username", $secpasswd)
$lbname = "WebAppWinLB"
$availSetName = "WinWebappAvailabilitySet"
#Creates Availability Set to allow both servers to be load balanced
New-AzAvailabilitySet `
   -Location $location.Location `
   -Name $availSetName `
   -ResourceGroupName $resourceGroup `
   -Sku aligned `
   -PlatformFaultDomainCount 2 `
   -PlatformUpdateDomainCount 2

$publicIp = New-AzPublicIpAddress -Name 'LB1PublicIP' -ResourceGroupName $resourceGroup -AllocationMethod Static -Location $location.Location
#sets up the inbound IP pool for the load balancer
$feip = New-AzLoadBalancerFrontendIpConfig -Name 'myFrontEndPool' -PublicIpAddress $publicIp
$bepool = New-AzLoadBalancerBackendAddressPoolConfig -Name 'myBackEndPool' 
#creates health check for load balancer
$probe = New-AzLoadBalancerProbeConfig `
 -Name 'myHealthProbe' `
 -Protocol Http -Port 80 `
 -RequestPath / -IntervalInSeconds 360 -ProbeCount 5
#creates load balancing rule
$rule = New-AzLoadBalancerRuleConfig `
  -Name 'webInbound' -Protocol Tcp `
  -Probe $probe -FrontendPort 80 -BackendPort 80 `
  -FrontendIpConfiguration $feip `
  -BackendAddressPool $bepool
#creates new LB from settings gathered so far
 $lb = New-AzLoadBalancer `
  -ResourceGroupName $ResourceGroup `
  -Name $lbname `
  -Location $location.Location `
  -FrontendIpConfiguration $feip `
  -BackendAddressPool $bepool `
  -Probe $probe `
  -LoadBalancingRule $rule 

elapsedTime

$serversCount = 2
for ($i=1; $i -le $serversCount; $i++) {
  elapsedTime
$VMName = "WebappWin" + $i
Write-Host "Creating VM " + $VMName
#Generates new public IP for the new load balancer to be created
$VM = Get-AzVM -Name $VMName
$NIC = Get-AzNetworkInterface -Name $VMName
#creates new VM
New-AzVm `
    -Credential $credential `
    -ResourceGroupName $resourceGroup `
    -Name $VMName `
    -Location $location.Location `
    -VirtualNetworkName $vnet `
    -SubnetName $subnet `
    -SecurityGroupName $securityGroup `
    -PublicIpAddressName "$($VMName)PublicIP" `
    -AvailabilitySetName $availSetName
Write-Host "VM $($VMName) has been created"
elapsedTime
Write-Host "Installing IIS for " + $VMName
$PublicSettings = '{"commandToExecute":"powershell Add-WindowsFeature Web-Server"}'
#Waits a few seconds for the VM to become available to recieve 
Start-Sleep -Seconds 5
Set-AzVMExtension -ExtensionName "IIS" -ResourceGroupName $resourceGroup -VMName $vmName `
  -Publisher "Microsoft.Compute" -ExtensionType "CustomScriptExtension" -TypeHandlerVersion 1.4 `
  -SettingString $PublicSettings -Location $location.location
Write-Host "IIS Installed for $($VMName)"
elapsedTime
Write-Host "Deploying website for " + $VMName

Invoke-AzVMRunCommand -ResourceGroupName $resourceGroup -VMName $VMName -CommandId "RunPowerShellScript" -ScriptPath "C:\pathhere\WebsiteTest\deployWebsite.ps1"
Write-Host "Website deployed on $($VMname)"
elapsedTime
#Gets load balancer object based on name
$lb = Get-AzLoadBalancer -Name $lbname
$backendConfig = Get-AzLoadBalancerBackendAddressPoolConfig -LoadBalancer $lb
#Get's NIC from virtual machine
$NIC = Get-AzNetworkInterface -Name $VMName
#Removes VM from LB
#$nic.Ipconfigurations[0].LoadBalancerBackendAddressPools=$null
#Adds VM to LB
Write-Host "Adding $($VMName) to Loadbalancer " + $lbname
$nic.IpConfigurations[0].LoadBalancerBackendAddressPools=$lb.BackendAddressPools[0]
Set-AzNetworkInterface -NetworkInterface $nic
Write-Host "VM $($VMName) added to the load balancer"
elapsedTime
}

Write-Host "Script completed" -BackgroundColor Blue
elapsedTime

Azure!

I tried moving this website to CentOS 8 but I somehow managed to break YUM/DNF in my initial configuration of the LAMP stack. I don’t want to admit I screwed up so I’ll just say that CentOS 8 is not ready for production quite yet, at least not in my world. As a result of the remediation I have moved this server to Azure. I’m learning more and more about Microsoft’s way of running a cloud provider..

CI/CD

I learn best by doing. I had heard continuous integration and continuous deployment thrown around for a while, often as buzz words. My current employer doesn’t really do either of these though we do use Jenkins as a deployment server. I’ve never seen it in action but from what I understand it has hooks into some of our java container hosts like WebLogic. It copies files over and understands compilation errors. Again I’ve never actually used it so this is just my understanding from what I’ve heard at work. GitLab gives you 2000 CI minutes which I never used. With my learning of Go it and pushing my code into it I started getting template suggestions for testing. The template provided needed a few edits and it started compiling my code and testing it for errors. It would place the built binary in a kind of hidden directory. It would do this on every push and took about 5 minutes. It used environment variables to protect credentials being in the config file.

Deployment was more difficult. I had trouble finding a good example out there on how to deploy my web app written in Go. Finally I found an alright example with a shell script that copied files. I augmented it with a service created on my web server.

Here’s my gitlab-ci.yml:

# This file is a template, and might need editing before it works on your project.
image: golang:latest

variables:
  # Please edit to your GitLab project
  REPO_NAME: gitlab.com/murphyslaw4267/cool_go_app
  APP_NAME: cool_go_app
  S3_BUCKET_NAME: "serverhobbyistohio"
  AWS_ACCESS_KEY_ID: $AWSID
  AWS_SECRET_ACCESS_KEY: $AWSSecret

# The problem is that to be able to use go get, one needs to put
# the repository in the $GOPATH. So for example if your gitlab domain
# is gitlab.com, and that your repository is namespace/project, and
# the default GOPATH being /go, then you'd need to have your
# repository in /go/src/gitlab.com/namespace/project
# Thus, making a symbolic link corrects this.
before_script:
  - mkdir -p $GOPATH/src/$(dirname $REPO_NAME)
  - ln -svf $CI_PROJECT_DIR $GOPATH/src/$REPO_NAME
  - cd $GOPATH/src/$REPO_NAME

stages:
  - test
  - build
  - deploy

format:
  stage: test
  script:
    - go get github.com/aws/aws-sdk-go
    - go fmt $(go list ./... | grep -v /vendor/)
    - go vet $(go list ./... | grep -v /vendor/)
    - go test -race $(go list ./... | grep -v /vendor/)

compile:
  stage: build
  script:
    - go get github.com/aws/aws-sdk-go
    - echo $CI_PROJECT_DIR
    - go build -race -ldflags "-extldflags '-static'" -o $CI_PROJECT_DIR/build/$APP_NAME
  artifacts:
    paths:
      - build/
  
deploy:
  stage: deploy
  only:
  - master
  image: ubuntu
  before_script:
  - apt update -y
  - apt install rsync -y
  - 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client git -y )'
  - eval $(ssh-agent -s)
  - echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add - > /dev/null
  - mkdir -p ~/.ssh
  - chmod 700 ~/.ssh
  - ssh-keyscan webapp1.serverhobbyist.com >> ~/.ssh/known_hosts
  - chmod 644 ~/.ssh/known_hosts
  - rsync -ae ssh ./build/* root@webapp1.serverhobbyist.com:/var/www/go/cool_go_app/
  - rsync -ae ssh ./*.html root@webapp1.serverhobbyist.com:/var/www/go/cool_go_app/
  script:
  - bash .gitlab-deploy.sh


Learning About Lambda

Amazon Web Service’s serverless compute has always fascinated me. A lot of my pursuits of learning the DevOps methodology involves scripting but not full blown programming. When it comes to automation I can accomplish my goals with scripting. Now I just found out that you can run PowerShell in AWS Lambda but I think that is pretty recent. So before I knew that I decided I wanted to re-engage my brain centers for learning a programming language. I see a lot of hype and support for Google’s Go language. I started going through the tutorials and creating things. An idea came to me for a creation which I will talk about in a later post. For now I wanted to share my understanding of Lambda.

Lamba uses requests and events. The request contains the data it needs to function in kind of a HTTP post, at least that’s what it reminds me of since you can call it through a HTTP request and the Lambda library pulls out headers and the body. Then it returns in an HTTP fashion as well. Here’s a sample piece of code that leverages AWS Simple Email Service to take a name out of a HTTP body.

package main

import (
	"errors"
	"fmt"
	"log"

	"github.com/aws/aws-lambda-go/events"
	"github.com/aws/aws-lambda-go/lambda"
	"github.com/aws/aws-sdk-go/aws"
	"github.com/aws/aws-sdk-go/aws/awserr"
	"github.com/aws/aws-sdk-go/aws/credentials"
	"github.com/aws/aws-sdk-go/aws/session"
	"github.com/aws/aws-sdk-go/service/ses"
)

const (
	// Replace sender@example.com with your "From" address.
	// This address must be verified with Amazon SES.
	Sender = "lambda@serverhobbyist.net"

	// Replace recipient@example.com with a "To" address. If your account
	// is still in the sandbox, this address must be verified.
	Recipient    = "Recipient@Recipient.com"

	// Specify a configuration set. To use a configuration
	// set, comment the next line and line 92.
	//ConfigurationSet = "ConfigSet"

	// The subject line for the email.
	Subject = "Lambda Function Go!"

	// The character encoding for the email.
	CharSet = "UTF-8"
)

var (
	// ErrNameNotProvided is thrown when a name is not provided
	ErrNameNotProvided = errors.New("no name was provided in the HTTP body")
)

// Handler is your Lambda function handler
// It uses Amazon API Gateway request/responses provided by the aws-lambda-go/events package,
// However you could use other event sources (S3, Kinesis etc), or JSON-decoded primitive types such as 'string'.
func Handler(request events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) {
	var TextBody = "Hello " + request.Body
	var HtmlBody = "Hello " + request.Body
	sess, err := session.NewSession(&aws.Config{
		Region:      aws.String("us-east-1"),
		Credentials: credentials.NewStaticCredentials("Removed", "Removed", ""),
	})

	// Create an SES session.
	svc := ses.New(sess)

	// Assemble the email.
	input := &ses.SendEmailInput{
		Destination: &ses.Destination{
			CcAddresses: []*string{},
			ToAddresses: []*string{
				aws.String(Recipient),
			},
			BccAddresses: []*string{
				aws.String(BCCRecipient),
			},
		},
		Message: &ses.Message{
			Body: &ses.Body{
				Html: &ses.Content{
					Charset: aws.String(CharSet),
					Data:    aws.String(HtmlBody),
				},
				Text: &ses.Content{
					Charset: aws.String(CharSet),
					Data:    aws.String(TextBody),
				},
			},
			Subject: &ses.Content{
				Charset: aws.String(CharSet),
				Data:    aws.String(Subject),
			},
		},
		Source: aws.String(Sender),
		// Uncomment to use a configuration set
		//ConfigurationSetName: aws.String(ConfigurationSet),
	}

	// Attempt to send the email.
	result, err := svc.SendEmail(input)
	log.Println(result)

	// Display error messages if they occur.
	if err != nil {
		if aerr, ok := err.(awserr.Error); ok {
			switch aerr.Code() {
			case ses.ErrCodeMessageRejected:
				fmt.Println(ses.ErrCodeMessageRejected, aerr.Error())
			case ses.ErrCodeMailFromDomainNotVerifiedException:
				fmt.Println(ses.ErrCodeMailFromDomainNotVerifiedException, aerr.Error())
			case ses.ErrCodeConfigurationSetDoesNotExistException:
				fmt.Println(ses.ErrCodeConfigurationSetDoesNotExistException, aerr.Error())
			default:
				fmt.Println(aerr.Error())
			}
		} else {
			// Print the error, cast err to awserr.Error to get the Code and
			// Message from an error.
			fmt.Println(err.Error())
		}

	}

	// stdout and stderr are sent to AWS CloudWatch Logs
	log.Printf("Processing Lambda request %s\n", request.RequestContext.RequestID)

	// If no name is provided in the HTTP request body, throw an error
	if len(request.Body) < 1 {
		return events.APIGatewayProxyResponse{}, ErrNameNotProvided
	}
	log.Println(request.PathParameters)
	return events.APIGatewayProxyResponse{
		Body:       "Hello " + request.Body,
		StatusCode: 200,
	}, nil

}

func main() {
	lambda.Start(Handler)
}

Monitoring with Grafana

Monitoring is one of those things I’ve seen get overlooked. Throwing money at it doesn’t always make it better. Taking the time to sit down and understand and doing proper event management is key. Of course I say all of that and I don’t always practice it, especially on my own personal infrastructure. I think I go beyond the basics however. Originally I would just set up nagios and get emails and texts when a server didn’t respond to ping. Finding something that can monitor and alert on performance was more challenging. I ended up settling using the telegraf agent and having it send to an influxdb instance. Then grafana interprets the influxdb data and makes it into pretty visuals.