Alex's Undercover Blog

For the geekier side of me.

Creating a Blog Platform with Amazon Web Services - Getting Caught

Posted: Sun, 01 Dec 2019 by

Getting Caught Up

In my last post I talked a bit about my desire to give recreating my blog using AWS, and in particular using a serverless setup (API Gateway -> Lambda, etc). One of the things i failed to mention is that I’ve already been working on this off an don for a while and already gotten deep into what’s needed to get it done. My general intention for these posts is a sort of development journal, documenting what I’m doing, the issues that I ran into, ways that I ended up fixing those problems, and general thought process I went through to get things done.

Since I’ve already started working on this, I’d like to share what I’ve gone through so far, to get more or less up to date, and then start doing proper journal-like entries. I’ve already started to take notes for some of my future posts, but for the time being, let’s get started on what I’ve done so far.

Getting started with the AWS CDK

While I’ve worked with AWS services on and off for years, the number of services that I’ve been exposed to have been minimal; generally S3, DynamoDB, and a smattering of others, none of which I would claim to have any sort of serious expertise in (especially given how quickly the services are built upon and improved). My current team at Amazon spent the last year building up a service using mainly the serverless model that I’m going for with my blog platform using Cloud Formation (essentially configuration to define AWS infrastructure). I’ve had some opportunities to make changes to the CloudFormation configuration, but I didn’t find updating the YAML to be very friendly or fun.

When the CDK’s GA was announced in July, I was excited by the prospect of putting a user-friendly layer on top of Cloud Formation and started to experiment with it a bit to understand how the APIs work in general and to setup some infrastructure as well.

If you found the idea of writing code to create infrastructure, with all the bells and whistles you expect from an IDE experience intriguing, then a great place to start would be the CDK Workshop that the CDK team put together. It will walk you through the prerequisites for which ever language you choose (though I expect to be doing all of the implementation in Java), as well as walk you through setting up your first application. This makes for a good jumping off point for playing around with creating and updating new resources. This experimenting has also helped me to understand the different AWS services that I’ll be using at a much more fundamental level. Being able to use auto-complete to make suggestions is awesome.

Starting hear will allow you to get up to speed on things in preparation for the next post.

What’s Coming Up?

This whole project is intended to be a thorough deep-dive on the entire process from beginning to end. I’ll be sharing as much as I can as I go. I may go on some tangents here or there if I find that I keep doing things the hard way. I basically want to automate as much of the process as possible. I’ll try and keep any tangents I go on as separate posts so you can know ahead of time what you want to read and what you want to avoid, though I hope that all the posts will be helpful in some way.

In my next post, I’ll go through some example code for defining an API in API Gateway, and why doing so with out-of-the-box CDK code won’t scale.

Creating a Blog Platform with Amazon Web Services

Posted: Sun, 17 Nov 2019 by

In The Beginning

Way back in January 2012, when I started a new role at Amazon, I needed to learn how to code using Ruby and Rails. As a personal exercise, I decided to put together a simple blog platform using the language and framework not only to get my feet wet, but also because I didn’t like the idea of using another platform like Wordpress; I wanted to have intimate knowledge of how it worked and be able to make changes to it on a whim if I wanted.

The requirements I had were pretty straightforward: * Written entirely in Ruby and Rails. * Learn something new to bolster my “tool box” of skills. * Be able to create, read, update and delete posts. * Be able to tag posts so they’d be able to be categorized. * Use a single blogging platform for multiple domains and have the posts that are shown for each domain be relevant for only that domain, including being able to show the same post across multiple blogs. * Use media wiki to format the text. I used this format on a daily basis so it made sense at the time to use this.

Thus, my blogs were born: one for personal stuff (family and friends), and another for more technical stuff that I want to share with the world. Let’s be honest though, it’s usually just for me, apart from the occasional recruiter that contacts me on LinkedIn that mention my blog. I haven’t written to them very often (my last post was in September of 2018, and before that it was 6 years earlier), but I do aspire to write more often than I do, which has caused me to give more thought into my blogs lately.

Defining the problem

There are a number of things that get in the way my writing posts, the least of which is having many commitments in my day-to-day (work and family mostly). It’s also the lack of accessibility in writing them, however. Like every other hobby-like thing in my life, I need to be able to maximize the little time that I have which means planning and being able to make gradual progress; sometimes a few minutes here and a few minutes there. In a nutshell, being able to write posts where ever any free time presents itself. In order to do that, I would ideally have an app on my phone that would let me keep track of post ideas, and put them together and post them whenever I can.

Another, perhaps more important thing, is having something compelling to write about. You don’t just start a blog for the sake of doing so. You do it because you have some knowledge or experience to share. While I’ve been able to write a thing or two over the years that satisfy that requirement, I’ve never been able to do it consistently, both because I wasn’t spending a lot of my free time doing things that I felt like sharing, and because I wasn’t in and found it difficult to get into the habit.

Finally, there are constantly so many cool things being done in the tech space, and being able to experiment with those things and do things that generally take me out of my comfort zone, will help keep me fresh and not stagnate in my career. The urge in me to continually learn new and interesting things is enough that rolling my own software is the best option. Being able to share those things with people is really exciting to me. If I can help show somebody the ropes, or to help plant a seed in somebody’s head for their own ideas, then I feel very rewarded.

The Solution

While putting together a blog with Rails was a great exercise for me at the time, my career has been heading in a different direction the last 3 years or so. While my current role does include some front-end work, it’s minimal at best. Most of what I do is back-end work, and it’s starting to move into the pure AWS space, as opposed to some AWS, and some internal Amazon stuff (though there’s still a lot of that). With my team starting to head in that direction, I’d like to be able to learn more about AWS services and how they can help solve the problems I’m keen on solving (personally and professionally). This seems like a good opportunity to rewrite my blog platform using AWS services and other frameworks, languages, etc. to put together the full stack. This is not going to be a small amount of work, but I think it will be interesting enough that it will keep me engaged through to the end, and I’m hoping that my experience can help other people putting together similar projects. I’d also be keen on putting all the source code up on Github so others can benefit as much as possible.

I’ve already started on some of this work, and the rabbit hole is a bit deeper than this, but I’ll get to the other details in subsequent posts.I should note that I’m trying to design the full stack in such a way that it can essentially scale infinitely, though I realize that my blog in particular doesn’t need to. As this point, most of the stuff that I have planned should be easily accommodated by the AWS free tier. To that end, here’s my high-level plan for putting this all together.

Java

Except where otherwise mentioned, I’ll be writing most of the code for this project in Java, which is the language I’m most comfortable with. When using Java, most of the new stuff I’m learning will be related to AWS services or different frameworks. There will be other components later on that will provide new learning in different languages. I’ll also be using Maven for dependency management.

AWS CDK

The AWS CDK (Cloud Development Kit) was introduced in July, 2019. It’s a framework that allows you to write code in Java, TypeScript, and others that generate Cloud Formation configuration to deploy your CFN stacks. It takes care of a lot of the heavy lifting for you, and adds very user-friendly APIs to manage permissions and other aspects of CloudFormation, not to mention the ability to auto-complete your code in whatever IDE that you use. I’ve used this a bit for a small project at work and am smitten.

API Gateway

I’ll start by defining the APIs that will be needed to create blogs, articles, tags, etc. via AWS API Gateway. Defining how you want to interact with a system is a great first step, which will help define the implementation details more formally. I want to use the OpenAPI specification to define the APIs and create the API Gateway end points to make the process as easy and maintainable as possible. While the group that defined OpenAPI has tools to generate code based on those specifications, I wasn’t able to find one that works with API Gateway, and certainly not the CDK as it’s fairly new. It’ll require writing some custom code to generate the APIs using the CDK.

Dynamo DB

I’ll be using DynamoDB to store all the data needed for the blogs. While I’ve thought about blogs from a relational perspective, pushing the data into Dynamo will force me to think in a non-relational way.

Elm (for the web front-end)

I’ve not had a huge amount of experience with functional languages and I’m especially intrigued by Elm, which is a functional language for front-end development that compiles down to JavaScript. The plan is to write the full read-only front end in Elm to call the API I’ve defined in API Gateway, and serve up the site via S3 and CloudFront.

SwiftUI (for write access)

Ever since SwiftUI was announced at WWDC earlier this year, I’ve been really excited to jump into it to build an app. I think the blog platform will be the perfect opportunity to do so as it should be fairly straight-forward once the API is defined and implemented.

High-level Architecture

Bonus Stuff

Along the way, I’m willing to wager that there are several other tools or technologies that I’ll need to learn in order to implement the platform as effectively as possible. The fact that I already have a blog platform in place means I don’t have to rush through this, but rather I can do it right. Beyond that there may be some other not needed but nice to have things I could learn such as setting up a continuous integration environment using AWS Code Commit, Build and Deploy. As the project grows, any sort of automation that I can implement may help save time in the long run (help me utilize my time as best as possible), and also be far more fun than doing things manually.

Writing Cadence

This is a pretty big project, and has a lot of bits and pieces that need to come together in order to make the whole. Turning anything into a habit can take quite a while; not the 21 days that I’ve previously heard. I don’t want to overcommit myself, which I’ve shown is something I typically do, especially when it comes to writing. To that end, I’m going to commit to writing an average of 12 articles over the next year that will gradually document the effort I go through to get all this stuff setup, including migrating all my previous posts from my Rails/PostgreSQL stack. Some of them will be longer than others, and I may even do more than that as I run into problems, or new tech that I want to take a stab at to solve a particular problem. Expect some tangents as I go.

In the mean time…

As a final note, since I’m most comfortable writing in Markdown, but my current blog implementation only understands mediawiki, I’ll be using Pandoc to convert to that format, until I get the current platform up and running. I’ll likely also use it to convert all of my existing articles over to Markdown. We’ll cross that bridge when I get there.

Updated: 2019-11-17 14:59:52 -0800

Checking in...

Posted: Sun, 16 Sep 2018 by

Every now and then I'll open up my blog (if you can call it that) to see when the last time I wrote something of use. I've always liked writing, but have been horrible at getting in the habit of doing so. This is clearly the case since I haven't written since early 2013 (more than 5 years ago at this point) and, embarrassingly, was me proclaiming I was going to get "this and that" done by the end of the year. While setting goals is good, I've learned from experience that the goals that I need to set need to generally be smaller in scope and be shorter term. I set a bunch of goals for the year, but those goals weren't broken down into smaller chunks or time frames which made it hard to track my progress with them (not that I was tracking their progress).

In my previous post I wrote about wanting to read a laundry list of books, some technical, some not. The list seems fine enough, but the fact is the mood that I'm in shifts regularly enough that thinking I would be interested enough in those books, whenever I got around to them, was wishful thinking. Especially if I was going to be reading them over the course of the year. Over the last several years I've started to get into the habit of reading one technical book, to read when I have an hour or more by myself to spare, and one non-technical book, to read before bed or while I'm driving (in audio format), at a time. Basically I let whatever mood I'm in drive what it is I'm reading. It seems simple, I know.

With respect to reading, I've been trying to use Good Reads to track my reading progress (using my Kindle) and to keep a list of books that I want to read at some point in the future. That way I should be able to jump straight into the next book, once I've completed one. This is much more visible then keeping all the books I want to read on my kindle and then trying to find the one I'm after. It's not perfect but it's a step in the right direction.

As far as technical books go, my ability to tackle them has been a bit hit or miss, especially now having a toddler running around the house. In order for me to really absorb what it is I've been reading, I need to be able to apply it on a regular basis. There are so many things that I want to learn, but the fact is if I don't come up with a project to apply what I'm learning as I go, I essentially lose it. The books that I've been the most successful with have been those that have a book-long project that is built up and improved as you read the book. Two books that come to mind are:

Both of these start you off with the very simplest knowledge you need to get some basic functionality implemented, and then build on top of the initial codebase, a lot of the times starting you out with less than stellar coding patterns, but backing those things out in favor of more improved ways of doing things once you have the knowledge to do so.

Both of these books I've been able to stick with because the projects are compelling.

I've also found success reading books on those subjects that I use on a daily basis at work. The most recent being Functional Programming in Java. Java 8 introduced lambda functions and functional programming facilities to make code cleaner, more efficient, etc. Because the codebase I was working on was starting to use these facilities, it was important that I be able to use them effectively. It was easy to read a chapter or a half, and then apply it the next day at work.

So what am I doing to try and actually accomplish things that I want to accomplish?

  • Jotting down ideas as soon as I have them, and taking notes whenever I think of new and interesting details. In this way I can capture enough details to help kick start whatever it is that I'm doing.
  • Planning things as much as I can. Organizing the notes above and coming up with a general plan of execution and then tracking these tasks somewhere. I've been using Trello.com for this as I can define swim lanes for each step of the development process. The planning step is hard but worth it as it allows me to focus what little time I have on specific tasks.
  • Doing my best to get a project that I'm working on to a stage where it's actually working so I'm not left with a bunch of half-implemented code. This blog is a good example. Written from scratch and has enough functionality to be useful but could easily be extended in a number of different ways to make it better.
  • Making sure the project is useful to me in some way. This will give me more incentive to actually get it to a working state.

I've only found success with this process once so far, but I'm happy with the results. I have lots of notes written for a number of project ideas to pursue at some point, and also have a mostly finished Alexa skill that provides me some benefit. My hope is to keep working on it until it's fit for consumption by other Alexa-using customers (right now it's hard-coded for just me, but the base of the skill is there and working well).

One last thing. I want to pursue writing a bit more, technically and personally; a journal of sorts. Even if they're only small bits of writing, it's nice to be able to look back and see what it was I was doing at the time, and I'm not super keen on Facebook or Twitter anymore, what with all the privacy-related crap that's been going down.

This was definitely a hodge-podge of stuff and not super organized but just wanted to get something on the books.

Updated: 2018-09-16 15:40:48 -0700

Goals for 2013

Posted: Thu, 03 Jan 2013 by

Almost 4 years ago, around this time I wrote about doing new year's resolutions differently. It's funny to look back that far and read about my good intentions in that single post but not reading anymore about my goals that I mention again. This year I'm going to attempt to identify what I've done wrong in the past and do things differently so that I can feel like I accomplish more in the year and avoid insanity.

Lets have a look at the goals I set 4 years ago and see how I've ended up:

  • Writing more: Technically I did write more (from 0 to some) but certainly far less than what I had intended by saying "more".
  • Continue to improve healthy habits: This is probably the one I've done the best at. Though I was hesitant in the original post I did end up running my first half-marathon in 2009 and have run several races since, including a full marathon in 2011.
  • Accomplish more personal projects: There have been a few minor things that I've done under this high-level goal though the only specific thing that I can think of is writing my own blog platform which I only just accomplished in 2012 (and am using now to share this post).

What exactly was it that was missing from the goals I set back then? First of all they were too vague. Take the writing goal for instance. The goal that I set didn't provide any specific means for measuring its success or failure. Again, technically I met the goal given that I wrote more than nothing but I don't feel that the goal was met. If I had come up with something more specific like "Write at least one blog post a month" it would've been much easier to measure the success or failure.

The second thing I should have provided for the goals was a time frame for accomplishing them. The "once a month" goal above would've have worked well for the writing goal. Really any timeframe other than "by the end of the year" would've been a big help in preventing procrastinating with regards to the goals.

One last thing that I can think of that would've helped is occasionally checking in to see how much progress I was making towards the goals to see if the goals should be adjusted, if they ended up being too unrealistic or weren't specific enough.

Now that I have these things in mind what goals am I going to set for 2013 you ask? While I have some specific goals in mind but I want to give some time to thinking about what those goals are and put together a plan for how I want to accomplish them. So really my first goal for the year is to come up with goals for the year. Sounds kind of silly but if I take the time to plan them out I think I have a much better chance of accomplishing them in 2013. I'm committing to coming up with more specific plans by Sunday evening (Jan. 6). That's not to say I'm going to have the whole year planned but at least have a few specific plans in mind to get started on.

Hopefully by this time next year I'll be able to look back and feel like I've accomplished what I set out to.

Updated: 2013-01-03 09:42:16 -0800

The Cynical Voter

Posted: Tue, 02 Nov 2010 by

They get you through television, radio, mail, and by calling you. For the last few months we've all been bombarded with half (or less) truths from an array of candidates and groups supporting one initiative or another. With every election I find myself becoming more and more cynical. So cynical in fact I've threatened to not vote just out of spite though I know that won't solve anything. Each of the candidates spends more time putting down their opponent(s) then presenting exactly what it is they stand for and why they would make the best next whatever. Not to mention the fact that millions upon millions of dollars are spent to get the word out for the different campaigns. It's gotten to the point (and has been that way for a long time) where regular Joes can't run for an office of their choice simply because they don't have the money.

Some might say that that's what fund raising is for, but I say why should it be necessary to spend so much money when it could be put to much better use. I really think it's time that we overhaul our political system. I'm certainly not holding my breath though.

Tagged: propoganda vote
Updated: 2012-02-25 15:51:26 -0800

The Trouble with Project 365

Posted: Tue, 11 May 2010 by

For those of you that I'm close to you know that I recently picked up a camera and committed to doing Project 365. That is taking and sharing a picture a day for a year. So far the undertaking has been a lot of different things. At times it's been exciting and inspiring and at other times frustrating and desperate.

Unfortunately for the last few weeks (or month even) most of my "efforts" have been pure desperation to get photos for each day. At just over 100 days in this is, from what I understand, I pretty classic place for project-goers to get stuck. I've spent the first few months of this project exploring my neighborhood and around where I work. It's been really great to see these two areas from a different perspective but it's become more and more clear that, as far as my inspiration is concerned, I've almost completely exhausted the possibilities of this limited scope. At least for the time being.

While I've grown in my photography a lot in the last few months I truly believe that my biggest growth is yet to come. I think that if I can really get myself in the habit of visiting some new and different places my desire to take photographs will return. Even visiting some familiar places and trying to get a different perspective on them would be helpful. Of course this is easier said than done given that my weekdays are generally filled with work and running.

Updated: 2012-02-25 15:52:56 -0800

And All of a Sudden I'm an (Amateur) Photographer

Posted: Thu, 31 Dec 2009 by

I've always loved photography. I got some of that love from my father and sister who are both incredible photographers. Until just recently however 100% of what I've done has been simple point and shoot. I've been able to get some great quality pictures but really wanted to take it to the next level.

A couple of months ago Chase Jarvis, a professional photographer, came to Amazon to promote his latest book "The Best Camera is the One You Have With You". He was clearly very passionate about his work and wanted to get everybody else in the room fired up about photography too. That's really when I started to  give photography a bit more consideration. I hatched a plan to save up for a new DSLR (a Nikon D90), signed up for a beginning photography class at the nearby community college (which I start Tuesday) and have started to make photography part of my everyday life.

More recently I started reading Digital Photography School and came across an article about the concept of Project 365. The idea is that you commit to taking and sharing one photograph a day for a year so you can see how you progress with your skill and also to integrate photography into your every day life. Since I'm starting to take the art more seriously I thought I'd give it a try and joined the Project 365! group on Flickr and took my first photo this morning:

Since I'm only just getting into the details of photography I'm excited to see how quickly my skill and technique improve.

Updated: 2012-02-25 19:29:28 -0800

Rejoining The World Of Warcraft

Posted: Fri, 27 Mar 2009 by

I've been playing World of Warcraft off and on since before it's initial release in 2004. I've never been a huge fan of MMORPG's in the past either because they require far too much grinding or they place too much of a focus on the multi-player aspect and essentially ignoring the solo player (because I have such a busy schedule I tend to be more of a solo player in general). When WoW came along I was intrigued by the small barrier to entry, it's simple and effective questing system and it of course had the advantage of the Warcraft history built in. Because of my lack of focus when it comes to games (I'm far too easily distracted by other things) I would generally play for a few months and then take several months off from WoW. My latest hiatus lasted about a year. It took a new expansion to get me back into the game.

I've been playing again now for about 4 months or so and am delighted to say that I'm having a great time and not getting bored nearly as easily as I have in the past. Previously the highest level that I achieved for a character was 44. At that point I would either start a new character of a different class or just quit altogether. My current main character is a level 58 rogue. I'm glad I've really stuck with the game this time as I'm only now starting to discover the higher level content; content which I haven't seen before. Here's why I think WoW is such a great game and why it always keeps me coming back for more....

  • As mentioned before the barrier to entry is minimal. You can easily jump into a new character and figure out how to play it pretty quickly.
  • The interface is customizable. There are some fantastic add ons out there that make the game more fun and efficient to play all because Blizzard decided to make the interface scriptable.
  • It's feasible to solo. Although some classes are better than others for this purpose WoW is accessible for the solo player.
  • The stories and history of Azeroth that ate presented to you throughout the game is nothing short of astounding. There are definitely some truly creative minds at Blizzard.
  • There is a ton of content. The world is just huge. If you get bored questing in one area there are likely several others that will be suited for your character's level.
  • Multi-player specific content is reasonably accessible. The UI makes it easy to find other people to group with for the games instances and the rewards are worth it. Also the 5 man instances can be run through in a few hours so it's not a big time commitment.
  • The new achievements system adds an interesting twist. If after all the main content you start to get bored they have some very interesting and sometimes strange things to do for accomplishment points (and bragging rights of course).
  • Leveling isn't quite as difficult as it used to be. This is another aspect that makes soloing possible.
  • You're not penalized for not playing for a while. When logged out you actually become rested. When your character is rested you gain twice the experience when killing creatures for a while which can make up for some lost time.
  • Though I mainly solo I've started a small guild with a few friends and we run instances together occasionally. That only adds to the fun. Since I finally got a character past 55 I've been playing the new death knight class from the latest release and have also been able to visit Outland.
I'm still discovering new and cool things about the game. It's easy enough to suck you in and deep enough to keep you playing for a long time.

Updated: 2012-02-25 15:15:34 -0800

The Historian

Posted: Fri, 23 Jan 2009 by

Based on a recommendation I got at a party I recently picked up a copy of The Historian at the library. All I really knew about the book going into it was that it was a story about people attempting to ascertain the true origin of Vlad the Impaler (Dracula). I've been really into vampire/zombie books lately so I though it might be a good read.

It turns out the book is more historical fiction than science-fiction. The reader is left on many a precipice of unknowns. As one question is answered many more are left unanswered and you are "forced" to venture through more of the book if you want to get at those answers. I'm about half way through it so far and can honestly say it's a great book. It's a perfect combination of historical fact with a great adventure story on top of it. It's definitely one of my new favorites and I would highly recommend it to anyone.

Updated: 2012-02-25 16:16:26 -0800

Interplay relaunches site, rehires original Fallout dev

Posted: Wed, 24 Sep 2008 by

Say what now? How cool is this?! I worked for both Babbages and Egghead Software during and after high school and played just about all of the games that Interplay published. It seems to have taken quite an effort on their part but hopefully this is just the beginning of the good news for this company. The first project to be an MMORPG based on the Fallout universe is very exciting. It certainly has a ton of potential. Hiring Chris Taylor was a great move also.

Updated: 2012-02-24 11:54:28 -0800