Today, the team I’m a part of at TargetSmart is releasing our first open source project, a bit of Python I like to call “cloudchain”. cloudchain is designed to make it easy to store and retrieve secrets using AWS. cloudchain relies on the AWS Identity and Access Management (IAM) Key Management Service (KMS) to securely store and manage access to encryption keys, and stores the encrypted secret in a DynamoDB table.

Part of the reason, if not the biggest reason, we are open sourcing this project is to request feedback from the community. cloudchain itself is only a few lines of glue plugging together a few AWS services, but its the idea itself that I’d like vetted. We are using this in a few projects internally, and so far it’s worked out. However, I know that there are things I haven’t thought of, and ways to improve the process, so I’m hoping others will be able to look at the project with fresh eyes and see things we haven’t.

There are three steps in the process. First, cloudchain retrieves an encryption key from KMS and uses it to encrypt the plain text secret. The boto3 library used returns a dictionary with a “Ciphertext” entry containing the encrypted key. cloudchain then base64 encodes the encrypted key into a string, and saves that string to a DynamoDB table named, by default, “safedb”.

Setup

pip install cloudchain

A new encryption key should be created in KMS. Using the console makes this easy, and sets up permissions to the key using IAM users or Roles. IAM users should be given permission individually, while instances launching in AWS should be identified by a role.

A new DynamoDB table should be created as well. Run this command using the AWS CLI tools:

aws dynamodb create-table \
--table-name safedb \
--attribute-definitions \
AttributeName=Service,AttributeType=S \
AttributeName=Username,AttributeType=S \
--key-schema \
AttributeName=Service,KeyType=HASH \
AttributeName=Username,KeyType=RANGE \
--provisioned-throughput \
ReadCapacityUnits=1,WriteCapacityUnits=1 

This will create the DynamoDB table with two attributes: Service and Username. cloudchain assumes that the combination of a service and a username will require a unique secret. The first time a secret is written to the table the third “Secret” attribute is created.

Configuration

The cloudchain cli, cchain, looks for a configuration file at ~/.cchainrc. This should be a standard Python ConfigParser compatible file with the following format:

[dynamo]
region_name = us-east-1
endpoint_url = https://dynamodb.us-east-1.amazonaws.com
tablename = safedb

[IAMKMS]
keyalias = alias/key

The “keyalias” should be the name of the KMS encryption key created during the setup, prefixed by “alias/”. The “endpoint_url” should point at the closest HTTPS endpoint, or at localhost if using a local development environment.

Import cloudchain as a Module

Both the test.py unit tests and the cchain cli import cloudchain.py. After importing, cloudchain expects four variables to be set:

• region_name
• endpoint_url
• tablename
• keyalias

Reasonable defaults are mentioned in the configuration section above, but the keyalias must be unique.

After importing, cloudchain can be called on to encrypt and decrypt secrets:

To Encrypt:

cloudchain.savecreds(args['service'], args['user'], args['save'])

To Decrypt: cloudchain.readcreds(args['service'], args['user'])

Where:

• service = The service name the username and secret are associated with
• user = The username
• save = The unencrypted secret to encrypt

Command Line Use

The command line script supports five arguments:

  -h, --help            show this help message and exit
  -u USER, --user USER  User name
  -e SERVICE, --service SERVICE
						Service or application
  -s SAVE, --save SAVE  Save password to the safe
  -r, --read            Read password from the safe

• The --save and --read arguments are mutually exclusive, and cannot be used at the same time.
• --save expects the unencrypted secret as an argument, and requires both --user and --service flags.
• --user expects the username as an argument.
• --service expects the service name as an argument.
• --read requires no arguments, and requires both --user and --service flags.

Examples

To save a secret:

./cchain -u testuser --service testservice --save testsecreet

To retrieve a secret:

./cchain -u testuser --service testservice --read

We hope this is useful, and that we can continue to make cloudchain better, easier to use, and more secure as development continues.

For the past two months I’ve been working, on and off, with a Rocelco Height Adjustable Standing Desk Riser, a less expensive choice for working at a standing desk than the popular VARIDESK. The Rocelco is a solid alternative for budget conscious workers, but as with most products, the drop in price comes with a set of trade-offs.

Having worked for several months with a VARIDESK, and the past two with the Rocelco, my opinion is that the VARIDESK is simply a better product, and will probably stand up better over the course of several years. The Rocelco does what it advertises, it raises the monitor and keyboard tray up to a reasonable level that feel appropriate to my height. However, the pistons it uses to raise the desk are so strong that you can’t simply pull on the top to raise it and stand away while the desk raises itself. If you do the top shoots up with enough force that when it reaches it’s full height it stops suddenly and shakes.

The first time this happened I was a bit afraid for my monitor. It even managed to shake the desk out of position slightly. If I had a cup of coffee on the desk I’m sure it would have splashed out. The pistons are strong. Once I realized this I remembered from then on to guide the desk to the standing position.

There are no alternative desk heights with the Rocelco, not without engaging the desk locks on each side of the structure. Where the VARIDESK has set points along the path of the raise, the Rocelco has one smooth transition from collapsed to fully expanded, although at any point along the way the locks could, theoretically, be engaged to lock the desk at a specific height, with the mechanics of how the desk raises it would be awkward at best. I’ve not bothered to try.

I’m a bit worried about the long-term prospects of the keyboard tray. The tray seems to be sitting an eighth of an inch lower than it was when I first unpacked the desk, and pulling up on the tray shows that it’s developed a bit of play to it. After two months of on and off use I would expect it to remain solid, I’m not sure what shape it will be in after a year or two. Also, neither the tray nor the desk seem solid enough to support me leaning on it, which, honestly, is a good thing. I shouldn’t be leaning on the desk while working anyway.

Since switching to a sanding desk last year I’ve become accustomed to long periods of standing, and walking around my office to think and work through problems. While I think the Rocelco is a fine starter desk, neither the aesthetics nor the mechanics of it make me happy enough not to start planning it’s replacement. For the next version I’m leaning heavily towards The Wirecutter’s recommendation of the Jarvis Bamboo, but I’m also considering a drafting desk like Dr. Bunsen’s.

Starting a new job working from home gave me the opportunity to evaluate exactly what I wanted from my work environment. I knew I was getting a new Mac (13” Retina MBP, as you do), but I also knew that I needed a new monitor and standing desk.

I worked for a few weeks using nothing but a 15” rMBP on my desk at home, and found that my right wrist was starting to hurt after only a few hours. I’ve had run-ins with RSI before, and had to wear a brace and adjust my posture while typing to relieve it. While my desk works well for the occasional work day at home, spending a few weeks sitting at it proved to me that it’s insufficient for serious extended periods of concentrated work.

However, I was ruined after looking at the beautiful retina screen for as long as I did. At my previous job I used an Apple Thunderbolt Display, which was beautiful and functional, but had lower DPI than the MacBook display, and was not very adjustable. The Thunderbolt could power the MacBook, and acted as a hub for anything that I needed to plug in, like network cable or external hard drive. The Thunderbolt is also $1,000. Given that I look at text all day, and sometimes into the night, I wanted a display that equaled my MacBook, had adjustable height, and wouldn’t break the bank. After researching a few alternatives, I settled on the Dell Ultra HD 4K Monitor P2415Q 24-Inch Screen LED-Lit Monitorat less than half the price.

I’ve read about Dell’s Ultrasharp line before, and always came away with the impression that they were very high quality. The 24” screen has a resolution of 3840 x 2160, the same as the next step 27”, which means the pixels are at a much higher density. After using this one for the past week, I can say that the screen itself is superb. There are no dead pixels, text is crisp and clear, colors are sharp, and the matte finish means that I’m not inadvertently staring at my reflection when working in a dark color scheme.

The screen adjusts to a height level with my eyes, so I’m not hunched over or looking down while working. It also rotates 90°, which I tried briefly and discarded. Just too weird.

While I’m perfectly satisfied with the screen, I do miss quite a few of the niceties of the Thunderbolt Display. There are no integrated speakers, no FaceTime camera, no ethernet port, and no power adaptor for the MacBook. It’s just a monitor. It does have a USB hub, and apparently the display port is capable of carrying audio, but neither feature is integrated into the Mac enough to be useful. For example, if my Time Machine drive was plugged into the monitor when the monitor went to sleep, the drive would be lost to OS X and I’d get that annoying alert about not ejecting a drive. If I need to do video or audio conferencing I’m either going to have to take the Mac out of it’s Twelve South BookArc or I’m going to need to plug in a USB camera, microphone, and speakers like it’s 2003.

I mentioned that the display port can carry audio. The monitor has a speaker jack on it, and I at first plugged in my desk speakers into that, but when I did the Mac lost all capability to control the volume output to the speakers. Hitting F11 or F12 showed an image on the screen indicating that the Mac was just shrugging it’s shoulders. Nothing it could do. Plugging the speakers back into the Mac and setting the headphone jack as the default audio out port solved this issue. I had to remind myself… it’s just a monitor.

The last part of the puzzle is still being shipped. I ordered an adjustable standing desk similar to a VariDesk, but made by a Canadian company for, again, about half the cost. I have the 32” Rodelco ADR on the way, and I imagine I’ll once again be standing for about three-quarters of the day. I spent several months standing while working at my previous job, and I miss that more than the Thunderbolt display. I’ll post again after I’ve worked with it for a week or so with my first impressions.

The 13” is a big change from the 15”, but I don’t feel like I’m missing anything. The 15” seemed an odd size to me. Too big to comfortably use apps full screen, but too small to be able to see more than two windows at the same time. The 13” is perfect for concentrating on a single task, is super light and easy to carry, and has enough power to push the incredible number of pixels on the Dell screen. So far I’m pretty happy with my new setup.

I was listening to Core Intuition a few weeks back and Manton said that if anyone was interested in his project to let him know, and he would send a couple of stickers. I was interested, so I emailed him a quick note, and quickly forgot about it.

Yesterday an envelope arrived in the mail with two stickers and a handwritten note. I’m looking forward to his new project, that Manton Reece seems like a stand-up guy.

Monday I was offered a new position, yesterday I accepted it. I’m hoping that this is the last time I’ll have to look for a job for a very, very long time. Having an unexpected change in your career and having to search for a new job is one of the most stressful things a person can do. It was hard, I didn’t sleep well.

I did get to talk to a lot of interesting and smart people though, and got a view of what their challenges were and what their company was like. Most of the people I spoke with were at a company that was growing fast, and they were looking for help scaling and automating their infrastructure, something I’ve come to specialize in. I applied for thirty-four positions, got back twenty-one responses, spoke with twelve companies, and found three positions where I was a near perfect fit.

Of the companies I spoke with, one of the common themes I heard again and again was “I read your blog.” Through my writing here they were able to take a look back through my history, understand how I convey information, and generally get to know me better. One of the first things I did after I started the search was write two blog posts explaining how I came to be where I am in my career, and what I believe DevOps to be.

Writing these posts served two functions. Most importantly it forced me to think through what I was trying to say, to understand what I think about the topic, and convey that in a way that’s concise and understandable. After writing the posts, when the questions inevitably came up during an interview, I was better able to answer without rambling.

I’ve often wondered if keeping this site up was worth my time, my experience during the past month prove, at least to me, that time spent blogging is time well spent. It’s an investment in my future, a calling card to the world. This site is my little corner of the Internet.

Everyone’s professional path follows a slightly different trajectory. We are each a unique recipe of skills, experience, and interests, which shape who we are and how we come to be in the careers that we have. My experience in moving from a systems administrator to a devops role is unique, because, well, we are all unique.

Somewhere around January of 2010 I saw a shift in the future of the systems administrator role. Automation systems and cloud services were going to change things, and I knew that to stay relevant I was going to have to change too. At the time I felt a bit of an entrepreneurial pull, and decided I was going to be an independent Mac developer. I bought the book, set my alarm for an hour earlier each morning, and spent that hour digging through the book learning Cocoa and Objective-C. I did this for months, reading every page, every chapter, and finishing every programming challenge. At the end of the book I started working on my first app, an adoption of a shell script I called “go”.

Go was basically a bookmark manager, but it let you bookmark anything on your Mac. I used it to quickly SSH to servers, mount NFS shares, launch internal management web apps, and other administrative tasks. I was a bit enamored by the delicious generation at the time, and made some rather, well, regrettable UI choices. When the Mac App Store was announced, I knew that I wanted to get my app in there on day one, so I started working right away on version two, or, Go2. Go2 launched in the MAS in 2011, but never sold enough to make it worthwhile. So, after taking a break from development, I started working on the next app, a static blogging app I called “Paragraphs”. I worked on this app for months, launched it, watched it fizzle out, and decided I needed to rethink my professional goals.

During this same period I went to grad school for Human-Computer Interaction. I still saw that I needed to change my professional career path, I was fascinated by design and the science of how we use these machines, and felt it a personal goal of mine to have a graduate degree. So, nights and weekends for two years I was neck deep in a mix of psychology, Adobe Illustrator, OmniGraffle, Silverback, and Python. I graduated with a Masters degree in HCI in 2012, spent a few months at a start up in Des Moines, went back to CDS, and the following year shut down my development company. All while holding down a full time job and raising a family. To say I was stressed doesn’t even come close.

At CDS I was a Linux sysadmin, but I was also very deeply involved in the design and deployment of the architecture. After my boss quit I applied for and was appointed to a new position titled “Systems Architect Consultant”, which was kind of a fancy term for special projects and development. I had been studying automation, cloud architecture, high availability, and the myriad of other topics surrounding devops, but as the systems architect I was solely devoted to researching those topics and designing solutions based on my research. Around the end of my time there CDS sent me to AWS re:Connect, where I was kickstarted into learning just how big AWS was, and the reality of what I foresaw years before hit me with full force.

I could tell that things were not going well at the company though, and two months before a big round of layoffs I made the move to work for a small software shop named Future Health Software as a systems automation engineer. Unfortunately, between the time I interviewed for the position and the time I was hired, the company was acquired by an investment firm that also acquired Future Health’s biggest competitor, Chirotouch.

Not wanting to concern myself with any of the internal politics or rivalries involved in merging two previously competing companies, I dug into my work and learned everything I could about running production workloads on AWS. I kept an open mind, and learned that the AWS systems were meant to be used like a development framework, bootstrapping an application in the most efficient way possible. Of course, most places hadn’t used the service like this, especially the ones that had been using AWS since all that was available was EC2 and S3. So, I set about rebuilding the entire architecture, six years of it, in CloudFormation code, and developing a deployment system based on CodeDeploy.

After several months of setbacks, wrong turns, and distractions of managing the legacy environment, I had a system that could rebuild the entire infrastructure from scratch. I’m quite proud of that work, and hope to someday put it under a production workload. Unfortunately, before I got the chance, the company closed our office and announced that we were going to be let go. Which brings us to today.

Over the years as I’ve worked out where I want to be in my professional life I’ve learned many different technologies. I’ve spent time digging through the Linux kernel source and reading RFCs, I’ve learned to think pragmatically, read and understand code, and build systems from the ground up. I’ve also learned how the people who use the systems I build actually use them, and how the design choices I make affect the usability and reliability of the system.

For me, the path to get from a sysadmin to a devops engineer travelled from Unix to Linux, through Cocoa and Objective-C, past Python and PHP, stopping in HCI, design, and UX, back to Linux, Docker, and now on to the cloud. And you know, what? It’ll never stop. The tech industry moves fast, you’ve got to keep learning your entire life to stay on the right side of that wave. You’ll either ride it, or be crushed by it, but it’ll never stop.

So, be open to learning new things, try something out of the ordinary, push yourself to try something you haven’t done before, and always do your absolute best. I promise it’ll be a heck of a ride.

The phrase “DevOps” gets thrown around quite a bit, so I thought it might be helpful for me to write down exactly what it means to me. DevOps is the evolution of systems administration. A few years ago, I noticed that the SysAdmin field was finally starting to change, after years of being relatively static. For decades, A sysadmin would set up the hardware, install the operating system, setup SSH (or, telnet in the bad old days), install your application, and get it running. Even when virtualization became more mainstream and worked its way into production workloads, it didn’t change the core tasks of a sysadmin. There were simply more boxes to manage, and without appropriate configuration management, each virtual machine became a unique little snow flake. A few tools became more commonplace like CFEngine, Puppet, or Chef to ease the burden of virtual machine sprawl, but it wasn’t until cloud computing came along that the role of a sysadmin really started to change.  

I realized that the traditional role of a sysadmin was going to change over next few years, and I needed to be on the right side of the this wave of change. Today’s sysadmin needs to know code – because the entire infrastructure is based on code. DevOps as a concept means to me that the mindset of a traditional sysadmin in a data center wasn’t going to cut it for the kind of skills needed for the new environment. Hardware doesn’t matter, operating systems don’t matter, the only thing that matters is the ability to run the application, and run it in a way that is stable, scaleable, and secure. 

One has to completely change the way they think about infrastructure. A common flaw that I’ve seen is trying to treat AWS like your data center. AWS will let you do that, gladly, but it will be expensive. It’s utility computing, like water. You get your water from the local water company, and you pay for what you use. The water company will let you turn on your faucet and just run water all day, but it will charge you for it and you will pay for far more than what you actually need. That’s why you have to have tooling, in this example a faucet to turn the water off and on. In a similar manner, you need automation systems in your infrastructure to scale up and down as needed.  

The other part of the evolution is that you have to stop thinking of the infrastructure as a systems administration task, and instead think of AWS and other cloud providers more like an application programming framework. Tying into building blocks like legos, and picking the appropriate parts to run your application. 

Systems administration is evolving in the same way programming languages have, building up to ever higher layers of abstraction. DevOps is simply working with the developers to build the best possible system to run the application.

And everything is changing for me again. The CTO of the company I work for spoke with me yesterday, our office is being shut down and they are laying off the staff. I’ve got till March 1st to find something new.

If anyone is looking for a good devops guy, let me know!

My brief experiment with mutt ended mostly how I expected it would. With me turning on my email in Mail.app again and carrying on as normal. I try to understand the draw to using such an archaic tool as mutt, but there’s simply nothing about it that appeals to me. Not at this stage of my life anyway.

This time someone emailed me a PDF that I needed to sign and send back to her. A simple process with Mail.app, but I knew that I was going to have to jump through hoops and spend an hour reading the documentation to figure out how to respond with an attachment with mutt. Come to think of it, I should amend my previous statement on how well I know the app. I know how to read and organize my email in mutt, and I can use it to build an offline mirror of my hosted email. I can build a search index for it, and send plain text email. I know how to view attachments and how to use a command line web browser to view HTML email. But, the one thing I didn’t know how to do was the one thing I needed to get done. Something trivially simple with the built in mail client.

So, to the wayside it went. While I was at it, I decided to do some cleaning up of the rest of my command line tools as well. I uninstalled everything I had installed with brew with a simple command:

brew uninstall brew list

Then I reinstalled the few command programs that I knew I needed for work.

brew install jq jsmin rbenv

This mass uninstall also removed MacVim. Something I’ve had conflicting feelings about for a while now. As I said in my last post, MacVim excels at editing text, but I realize I set up a bit of a straw man comparing it with TextEdit. Obviously TextEdit was never the competition with MacVim, instead, I should have been comparing it with BBEdit and Ulysses, my other two text editors of choice.

Like Vim, BBEdit and Ulysses are power tools, but unlike Vim they are Mac power tools. They are built specifically for the Mac environment, by small teams who are dedicated to their craft, and who charge a reasonable, sustainable rate for their product. They approach their jobs in different ways, and are very different applications. BBEdit is built for developers and sysadmins, and has depths of integration and feature set that I’m only just beginning to explore. Ulysses is built for writing prose, and it’s where I’m writing this. Honestly, it’s beautiful.

We live in a blessed era of Mac productivity. We have almost an embarrassment of riches when it comes to incredibly well crafted third party applications, and we still have access to all the low-level Unix tools that attracted me to the platform thirteen years ago. I’ve waffled back and forth over using the old tools and adopting new ones, but I’m getting too old for that. I simply want my tools to work for me. As of right now, I don’t imagine I’ll be going back to mutt, vim, or anything else command line. Not when there are fantastic Mac apps that do the job either just as well, or better.

After reading through Matt Gemmell’s latest post on mutt and the good doctor’s response, I fired up my old mutt config and gave it another run through. Well, after being a bit snarky on Twitter, of course.

I’m always of two minds when it comes to using Unix tools on a Mac. On the one hand, I want to live in the future, where everything is beautiful and powerful, simple and easy to use, but functional enough to do everything I want. That future might never come, but sometimes it feels as if its right around the corner. On the other hand, I know Unix. I make my living on the command line, and have for years. I’ve got a customized .vimrc file that’s setup just the way I like it. I know how to zip around mutt. I could even setup mpd to play music instead of iTunes if I felt like it. That constant friction of how I wish things were hits up against how I know things are on a daily basis.

I know how to get things done quickly and efficiently, but sometimes I just don’t want to do it that way. I want to live in the future, not the past. But, once you know how to use the power tools, the knowledge will forever be present in the back of your mind, a quiet voice that says “you know how to do this, you know how to make this better”. However, it’s important to know when to take advantage of a power tool, and to avoid unnecessarily fetishizing the command line.

Living in the Apple ecosystem has benefits. Pictures I put in the Photos app are available everywhere. Likewise with music in iTunes Match. Any document I edit in TextEdit is automatically versioned without my needing to think about it. Apple designs their products to work well together, but they also design them to appeal to the largest audience possible, which may or may not include me.

Take TextEdit as an example. It’s a basic text editor, it can edit a Markdown document just fine, but when I edit Markdown in MacVim I’ve got access to my years of keyboard navigation and key combos for doing things like inserting links and switching between inline links and reference links. As previously mentioned, TextEdit automatically versions it’s documents, but with a couple of modifications, so does MacVim. TextEdit is a general purpose tool, MacVim is a power tool, they are not really in the same category, except they can both edit a text document.

Maybe that’s the differentiation I’ve been searching for. TextEdit can, as it’s name implies, edit text files, but MacVim excels at it. Mail can send and receive email, it’s a general tool best for most people. Mutt is a power tool, best for specific use cases when you already have the background knowledge to be able to take advantage of it.

Knowing how to use the power tools will make you a better computer user. Knowing when to use a particular tool, and knowing which tool is the best one for the job, makes you an expert.