https://github.com/nathants/libaws Skip to content Sign up * Product + Features + Mobile + Actions + Codespaces + Packages + Security + Code review + Issues + Integrations + GitHub Sponsors + Customer stories * Team * Enterprise * Explore + Explore GitHub + Learn and contribute + Topics + Collections + Trending + Learning Lab + Open source guides + Connect with others + The ReadME Project + Events + Community forum + GitHub Education + GitHub Stars program * Marketplace * Pricing + Plans + Compare plans + Contact Sales + Education [ ] * # In this repository All GitHub | Jump to | * No suggested jump to results * # In this repository All GitHub | Jump to | * # In this user All GitHub | Jump to | * # In this repository All GitHub | Jump to | Sign in Sign up {{ message }} nathants / libaws Public * Notifications * Fork 5 * Star 247 aws should be easy nathants.com/projects/libaws License MIT license 247 stars 5 forks Star Notifications * Code * Issues 0 * Pull requests 0 * Actions * Projects 0 * Wiki * Security * Insights More * Code * Issues * Pull requests * Actions * Projects * Wiki * Security * Insights This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. master Switch branches/tags [ ] Branches Tags Could not load branches Nothing to show {{ refName }} default View all branches Could not load tags Nothing to show {{ refName }} default View all tags 1 branch 0 tags Code Latest commit @nathants nathants readme ... 6927f8c May 24, 2022 readme 6927f8c Git stats * 705 commits Files Permalink Failed to load latest commit information. Type Name Latest commit message Commit time cmd completions.d examples gif lib .gitignore Dockerfile Makefile go.mod go.sum license.txt main.go readme.md test.sh tox.ini View code [ ] libaws why how what aws sdk, pulumi, terraform, cloudformation, and serverless readme index install cli go api tldr define an infrastructure set ensure the infrastructure set trigger the infrastructure set update the infrastructure set delete the infrastructure set usage explore the cli explore a cli entrypoint explore the go api explore simple examples explore complex examples explore external examples infrastructure set typical usage design tradeoffs infra.yaml environment variable substitution name s3 dynamodb sqs keypair vpc security group instance profile lambda entrypoint attr policy allow env include require trigger trigger types api websocket s3 dynamodb sqs schedule ecr bash completion outgrowing testing readme.md libaws why aws is amazing, but it's hard to see the forest for the trees. aws should have fewer knobs. aws should be easy to use, and hard to screw up. aws should be fast, and fun. aws should have a tldr. how easily CREATE and DEPLOY useful systems to aws: * that contain: + lambdas + s3 buckets + dynamodb tables + sqs queues + vpcs + vpc security groups + ec2 instance profiles + ec2 keypairs * that react to lambda triggers: + http apis + websocket messages + s3 bucket writes + dynamodb table writes + sqs queue puts + cron schedules + ecr docker pushes what a simpler way to declare aws infrastructure that is easy to use and extend. there are two ways to use it: * yaml and the cli * go structs and the go api the primary entrypoints are: * infra-ls: view deployed infrastructure sets. * infra-ensure: deploy an infrastructure set. * infra-rm: remove an infrastructure set. infra-ensure is a positive assertion. it asserts that some named infrastructure exists, and is configured correctly, creating or updating it if needed. many other entrypoints exist, and can be explored by type. they fall into two categories: * mutate aws state: >> libaws -h | grep ensure | wc -l 19 >> libaws -h | grep new | wc -l 1 >> libaws -h | grep rm | wc -l 26 * view aws state: >> libaws -h | grep ls | wc -l 33 >> libaws -h | grep describe | wc -l 6 >> libaws -h | grep get | wc -l 16 >> libaws -h | grep scan | wc -l 1 aws sdk, pulumi, terraform, cloudformation, and serverless compared to the full aws api, systems declared as infrastructure sets : * have simpler examples. * have fewer knobs. * are easier to use. * are harder to screw up. * are almost always enough, and easy to extend. * are more fun. if you want to use the full aws api, there are many great tools: * aws sdk for go * pulumi * terraform * cloudformation * serverless readme index * install + cli + go api * tldr + define an infrastructure set + ensure the infrastructure set + trigger the infrastructure set + update the infrastructure set + delete the infrastructure set * usage + explore the cli + explore a cli entrypoint + explore the go api + explore simple examples + explore complex examples + explore external examples * infrastructure set * typical usage * design * tradeoffs * infra.yaml + environment variable substitution + name + s3 + dynamodb + sqs + keypair + vpc o security group + instance profile + lambda o entrypoint o attr o policy o allow o env o include o require o trigger # api # websocket # s3 # dynamodb # sqs # schedule # ecr * bash completion * outgrowing * testing install cli go install github.com/nathants/libaws@latest export PATH=$PATH:$(go env GOPATH)/bin go api go get github.com/nathants/libaws@latest tldr define an infrastructure set >> cd examples/simple/go/s3 && tree . +-- infra.yaml +-- main.go name: test-infraset-${uid} s3: test-bucket-${uid}: attr: - acl=private lambda: test-lambda-${uid}: entrypoint: main.go attr: - concurrency=0 - memory=128 - timeout=60 policy: - AWSLambdaBasicExecutionRole trigger: - type: s3 attr: - test-bucket-${uid} package main import ( "context" "fmt" "github.com/aws/aws-lambda-go/events" "github.com/aws/aws-lambda-go/lambda" ) func handleRequest(_ context.Context, e events.S3Event) (events.APIGatewayProxyResponse, error) { for _, record := range e.Records { fmt.Println(record.S3.Object.Key) } return events.APIGatewayProxyResponse{StatusCode: 200}, nil } func main() { lambda.Start(handleRequest) } ensure the infrastructure set [ensure] trigger the infrastructure set [trigger] update the infrastructure set [update] delete the infrastructure set [rm] usage explore the cli >> libaws -h | grep ensure | head codecommit-ensure - ensure a codecommit repository dynamodb-ensure - ensure a dynamodb table ec2-ensure-keypair - ensure a keypair ec2-ensure-sg - ensure a sg ecr-ensure - ensure ecr image iam-ensure-ec2-spot-roles - ensure iam ec2 spot roles that are needed to use ec2 spot iam-ensure-instance-profile - ensure an iam instance-profile iam-ensure-role - ensure an iam role iam-ensure-user-api - ensure an iam user with api key iam-ensure-user-login - ensure an iam user with login explore a cli entrypoint >> libaws dynamodb-ensure -h ensure a dynamodb table example: - libaws dynamodb-ensure test-table userid:s:hash timestamp:n:range stream=keys_only required attrs: - NAME:ATTR_TYPE:KEY_TYPE optional attrs: - ProvisionedThroughput.ReadCapacityUnits=VALUE, shortcut: read=VALUE, default: 0 - ProvisionedThroughput.WriteCapacityUnits=VALUE shortcut: write=VALUE default: 0 - StreamSpecification.StreamViewType=VALUE, shortcut: stream=VALUE - LocalSecondaryIndexes.INTEGER.IndexName=VALUE - LocalSecondaryIndexes.INTEGER.Key.INTEGER=NAME:ATTR_TYPE:KEY_TYPE - LocalSecondaryIndexes.INTEGER.Projection.ProjectionType=VALUE - LocalSecondaryIndexes.INTEGER.Projection.NonKeyAttributes.INTEGER=VALUE - GlobalSecondaryIndexes.INTEGER.IndexName=VALUE - GlobalSecondaryIndexes.INTEGER.Key.INTEGER=NAME:ATTR_TYPE:KEY_TYPE - GlobalSecondaryIndexes.INTEGER.Projection.ProjectionType=VALUE - GlobalSecondaryIndexes.INTEGER.Projection.NonKeyAttributes.INTEGER=VALUE - GlobalSecondaryIndexes.INTEGER.ProvisionedThroughput.ReadCapacityUnits=VALUE - GlobalSecondaryIndexes.INTEGER.ProvisionedThroughput.WriteCapacityUnits=VALUE Usage: dynamodb-ensure [--preview] NAME ATTR [ATTR ...] Positional arguments: NAME ATTR Options: --preview, -p --help, -h display this help and exit explore the go api package main import ( "github.com/nathants/libaws/lib" ) func main() { lib. (TAB =>) |--------------------------------------------------------------------------------| |f AcmClient func() *acm.ACM (Function) | |f AcmClientExplicit func(accessKeyID string, accessKeySecret string, region stri| |f AcmListCertificates func(ctx context.Context) ([]*acm.CertificateSummary, erro| |f Api func(ctx context.Context, name string) (*apigatewayv2.Api, error) (Functio| |f ApiClient func() *apigatewayv2.ApiGatewayV2 (Function) | |f ApiClientExplicit func(accessKeyID string, accessKeySecret string, region stri| |f ApiList func(ctx context.Context) ([]*apigatewayv2.Api, error) (Function) | |f ApiListDomains func(ctx context.Context) ([]*apigatewayv2.DomainName, error) (| |f ApiUrl func(ctx context.Context, name string) (string, error) (Function) | |f ApiUrlDomain func(ctx context.Context, name string) (string, error) (Function)| |... | |--------------------------------------------------------------------------------| } explore simple examples * api: python, go, docker * dynamodb: python, go, docker * ecr: python, go, docker * includes: python, go * s3: python, go, docker * schedule: python, go, docker * sqs: python, go, docker * websocket: python, go, docker explore complex examples * s3-ec2: + write to s3 in-bucket + which triggers lambda + which launches ec2 spot + which reads from in-bucket, writes to out-bucket, and terminates explore external examples * new-gocljs * aws-rce * aws-ensure-route53 infrastructure set an infrastructure set is defined by yaml or go struct and contains: * stateful services + s3 + dynamodb + sqs * ec2 services + keypairs + instance profiles + vpcs o security groups * lambdas + lambda triggers o api o websocket o s3 o dynamodb o sqs o schedule o ecr typical usage * use infra-ls to view deployed infrastructure sets. * use infra-ensure to deploy an infrastructure set. * use infra-rm to remove an infrastructure set. * use lambdas in the infrastructure set to: + respond to triggers. + manage fleets of ec2 instances. + monitor services. + do anything. * rapidly iterate by responding to file changes: + on changes run: infra-ensure infra.yaml --quick LAMBDA_NAME + this updates the lambda code without making other changes. design * there is no implicit coordination. + if you aren't already serializing your infrastructure mutations, lock around dynamodb. * there are only two state locations: + aws. + your code. * aws resources are uniquely identified by name. + all aws resources share a private namespace scoped to account /region. use good names. + except s3, which shares a public namespace scoped to earth. use better names. * mutative operations manipulate aws state. + mutative operations are idempotent. if they fail due to a transient error, run them again. + mutative operations can --preview. no output means no changes. * ensure are mutative operations that create or update infrastructure. * rm are mutative operations that delete infrastructure. * ls, get, scan, and describe operations are non-mutative. * multiple infrastructure sets can be deployed into the same account/region. tradeoffs * no attempt is made to avoid vendor lock-in. + migrating between cloud providers will always be non-trivial. + attempting to mitigate future migrations has more cost than benefit in the typical case. * ensure operations are positive assertions. they assert that some named infrastructure exists, and is configured correctly, creating or updating it if needed. + positive assertions CANNOT remove top level infrastructure, but CAN remove configuration from them. + removing a trigger, policy, or allow WILL remove that from the lambda. + removing policy, or allow WILL remove that from the instance-profile. + removing a security-group WILL remove that from the vpc. + removing a rule WILL remove that from the security-group. + removing an attr WILL remove that from a sqs, s3, dynamodb, or lambda. + removing a keypair, vpc, instance-profile, sqs, s3, dynamodb, or lambda WON'T remove that from the account/region. o the operator decides IF and WHEN top level infrastructure should be deleted, then uses an rm operation to do so. o as a convenience, infra-rm will remove ALL infrastructure CURRENTLY declared in an infra.yaml. infra.yaml use an infra.yaml file to declare an infrastructure set. the schema is as follows: name: VALUE lambda: VALUE: entrypoint: VALUE policy: [VALUE ...] allow: [VALUE ...] attr: [VALUE ...] require: [VALUE ...] env: [VALUE ...] include: [VALUE ...] trigger: - type: VALUE attr: [VALUE ...] s3: VALUE: attr: [VALUE ...] dynamodb: VALUE: key: [VALUE ...] attr: [VALUE ...] sqs: VALUE: attr: [VALUE ...] vpc: VALUE: security-group: VALUE: rule: [VALUE ...] keypair: VALUE: pubkey-content: VALUE instance-profile: VALUE: allow: [VALUE ...] policy: [VALUE ...] environment variable substitution anywhere in infra.yaml you can substitute environment variables from the caller's environment: * example: s3: test-bucket-${uid}: attr: - versioning=${versioning} the following variables are defined during deployment, and are useful in allow declarations: * ${API_ID} the id of the apigateway v2 api created by an api trigger. * ${WEBSOCKET_ID} the id of the apigateway v2 websocket created by a websocket trigger. name defines the name of the infrastructure set. * schema: name: VALUE * example: name: test-infraset s3 defines a s3 bucket: * the following attributes can be defined: + acl=VALUE, values: public | private, default: private + versioning=VALUE, values: true | false, default: false + metrics=VALUE, values: true | false, default: true + cors=VALUE, values: true | false, default: false + ttldays=VALUE, values: 0 | n, default: 0 * schema: s3: VALUE: attr: - VALUE * example: s3: test-bucket: attr: - versioning=true - acl=public dynamodb defines a dynamodb table: * specify key as: + NAME:ATTR_TYPE:KEY_TYPE * the following attributes can be defined: + ProvisionedThroughput.ReadCapacityUnits=VALUE, shortcut: read =VALUE, default: 0 + ProvisionedThroughput.WriteCapacityUnits=VALUE, shortcut: write=VALUE, default: 0 + StreamSpecification.StreamViewType=VALUE, shortcut: stream= VALUE, default: 0 + LocalSecondaryIndexes.INTEGER.IndexName=VALUE + LocalSecondaryIndexes.INTEGER.Key.INTEGER= NAME:ATTR_TYPE:KEY_TYPE + LocalSecondaryIndexes.INTEGER.Projection.ProjectionType=VALUE + LocalSecondaryIndexes.INTEGER.Projection.NonKeyAttributes.INTEGER =VALUE + GlobalSecondaryIndexes.INTEGER.IndexName=VALUE + GlobalSecondaryIndexes.INTEGER.Key.INTEGER= NAME:ATTR_TYPE:KEY_TYPE + GlobalSecondaryIndexes.INTEGER.Projection.ProjectionType= VALUE + GlobalSecondaryIndexes.INTEGER.Projection.NonKeyAttributes.INTEGER =VALUE + GlobalSecondaryIndexes.INTEGER.ProvisionedThroughput.ReadCapacityUnits =VALUE + GlobalSecondaryIndexes.INTEGER.ProvisionedThroughput.WriteCapacityUnits =VALUE * schema: dynamodb: VALUE: key: - NAME:ATTR_TYPE:KEY_TYPE attr: - VALUE * example: dynamodb: stream-table: key: - userid:s:hash - timestamp:n:range attr: - stream=keys_only auth-table: key: - id:s:hash attr: - write=50 - read=150 sqs defines a sqs queue: * the following attributes can be defined: + DelaySeconds=VALUE, shortcut: delay=VALUE, default: 0 + MaximumMessageSize=VALUE, shortcut: size=VALUE, default: 262144 + MessageRetentionPeriod=VALUE, shortcut: retention=VALUE, default: 345600 + ReceiveMessageWaitTimeSeconds=VALUE, shortcut: wait=VALUE, default: 0 + VisibilityTimeout=VALUE, shortcut: timeout=VALUE, default: 30 * schema: sqs: VALUE: attr: - VALUE * example: sqs: test-queue: attr: - delay=20 - timeout=300 keypair defines an ec2 keypair. * example: keypair: VALUE: pubkey-content: VALUE * example: keypair: test-keypair: pubkey-content: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICVp11Z99AySWfbLrMBewZluh7cwLlkjifGH5u22RXor vpc defines a default-like vpc with an internet gateway and public access . * schema: vpc: VALUE: {} * example: vpc: test-vpc: {} security group defines a security group on a vpc * schema: vpc: VALUE: security-group: VALUE: rule: - PROTO:PORT:SOURCE * example: vpc: test-vpc: security-group: test-sg: rule: - tcp:22:0.0.0.0/0 instance profile defines an ec2 instance profile. * schema: instance-profile: VALUE: allow: - SERVICE:ACTION ARN policy: - VALUE * example: instance-profile: test-profile: allow: - s3:* * policy: - AWSLambdaBasicExecutionRole lambda defines a lambda. * schema: lambda: VALUE: {} * example: lambda: test-lambda: {} entrypoint defines the code of the lambda. it is one of: * a python file. * a go file. * an ecr container uri. * schema: lambda: VALUE: entrypoint: VALUE * example: lambda: test-lambda: entrypoint: main.go attr defines lambda attributes. the following can be defined: * concurrency defines the reserved concurrent executions, default: 0 * memory defines lambda ram in megabytes, default: 128 * timeout defines the lambda timeout in seconds, default: 300 * logs-ttl-days defines the ttl days for cloudwatch logs, default: 7 * schema: lambda: VALUE: attr: - KEY=VALUE * example: lambda: test-lambda: attr: - concurrency=100 - memory=256 - timeout=60 - logs-ttl-days=1 policy defines policies on the lambda's iam role. * schema: lambda: VALUE: policy: - VALUE * example: lambda: test-lambda: policy: - AWSLambdaBasicExecutionRole allow defines allows on the lambda's iam role. * schema: lambda: VALUE: allow: - SERVICE:ACTION ARN * example: lambda: test-lambda: allow: - s3:* * - dynamodb:* arn:aws:dynamodb:*:*:table/test-table env defines environment variables on the lambda: * schema: lambda: VALUE: env: - KEY=VALUE * example: lambda: test-lambda: env: - kind=production include defines extra content to include in the lambda zip: * this is ignored when entrypoint is an ecr container uri. * schema: lambda: VALUE: include: - VALUE * example: lambda: test-lambda: include: - ./cacerts.crt - ../frontend/public/* require defines dependencies to install with pip in the virtualenv zip. * this is ignored unless the entrypoint is a python file. * schema: lambda: VALUE: require: - VALUE * example: lambda: test-lambda: require: - fastapi==0.76.0 trigger defines triggers for the lambda: * schema: lambda: VALUE: trigger: - type: VALUE attr: - VALUE * example: lambda: test-lambda: trigger: - type: dynamodb attr: - test-table trigger types api defines an apigateway v2 http api: * add a custom domain with attr: domain=api.example.com * add a custom domain and update route53 with attr: dns= api.example.com + this domain, or its parent domain, must already exist as a hosted zone in route53. + this domain, or its parent domain, must already have an acm certificate with subdomain wildcard. * schema: lambda: VALUE: trigger: - type: api attr: - VALUE * example: lambda: test-lambda: trigger: - type: api attr: - dns=api.example.com websocket defines an apigateway v2 websocket api: * add a custom domain with attr: domain=ws.example.com * add a custom domain and update route53 with attr: dns= ws.example.com + this domain, or its parent domain, must already exist as a hosted zone in route53-ls. + this domain, or its parent domain, must already have an acm certificate with subdomain wildcard. * schema: lambda: VALUE: trigger: - type: websocket attr: - VALUE * example: lambda: test-lambda: trigger: - type: websocket attr: - dns=ws.example.com s3 defines an s3 trigger: * the only attribute must be the bucket name. * object creation and deletion invoke the trigger. * schema: lambda: VALUE: trigger: - type: s3 attr: - VALUE * example: lambda: test-lambda: trigger: - type: s3 attr: - test-bucket dynamodb defines a dynamodb trigger: * the first attribute must be the table name. * the following trigger attributes can be defined: + BatchSize=VALUE, shortcut: batch=VALUE, default: 100 + ParallelizationFactor=VALUE, shortcut: parallel=VALUE, default: 1 + MaximumRetryAttempts=VALUE, shortcut: retry=VALUE, default: -1 + MaximumBatchingWindowInSeconds=VALUE, shortcut: window=VALUE, default: 0 + StartingPosition=VALUE, shortcut: start=VALUE * schema: lambda: VALUE: trigger: - type: dynamodb attr: - VALUE * example: lambda: test-lambda: trigger: - type: dynamodb attr: - test-table - start=trim_horizon sqs defines a sqs trigger: * the first attribute must be the queue name. * the following trigger attributes can be defined: + BatchSize=VALUE, shortcut: batch=VALUE, default: 10 + MaximumBatchingWindowInSeconds=VALUE, shortcut: window=VALUE, default: 0 * schema: lambda: VALUE: trigger: - type: sqs attr: - VALUE * example: lambda: test-lambda: trigger: - type: sqs attr: - test-queue schedule defines a schedule trigger: * the only attribute must be the schedule expression. * schema: lambda: VALUE: trigger: - type: schedule attr: - VALUE * example: lambda: test-lambda: trigger: - type: schedule attr: - rate(24 hours) ecr defines an ecr trigger: * successful image actions to any ecr repository will invoke the trigger. * schema: lambda: VALUE: trigger: - type: ecr * example: lambda: test-lambda: trigger: - type: ecr bash completion source completions.d/libaws.sh source completions.d/aws-creds.sh source completions.d/aws-creds-temp.sh outgrowing drop down to the aws go sdk and implement what you need. extend an existing mutative operation or add a new one. * make sure that mutative operations are IDEMPOTENT and can be PREVIEWED. you will find examples in cmd/ and lib/ that can provide a good place to start. you can reuse many existing operations like: * lib/iam.go * lib/lambda.go * lib/ec2.go alternatively, lift and shift to other infrastructure automation tooling. ls and describe operations will give you all the information you need. testing run all integration tests aws with tox: export LIBAWS_TEST_ACCOUNT=$ACCOUNT_NUM tox run one integration test aws with tox: export LIBAWS_TEST_ACCOUNT=$ACCOUNT_NUM tox -- bash -c 'cd examples/simple/python/api/ && python test.py' About aws should be easy nathants.com/projects/libaws Topics golang aws lambda serverless Resources Readme License MIT license Stars 247 stars Watchers 3 watching Forks 5 forks Used by 3 * @nathants @nathants / go-dynamolock * @nathants @nathants / new-gocljs * @nathants @nathants / aws-rce Languages * Go 98.4% * Shell 1.3% * Other 0.3% * (c) 2022 GitHub, Inc. * Terms * Privacy * Security * Status * Docs * Contact GitHub * Pricing * API * Training * Blog * About You can't perform that action at this time. You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.