AppRunner - The Good, Bad and the Ugly

Background

In May of 2021, AWS released AppRunner to the public.

AWS App Runner is an AWS service that provides a fast, simple, and cost-effective way to deploy from source code or a container image directly to a scalable and secure web application in the AWS Cloud. You don’t need to learn new technologies, decide which compute service to use, or know how to provision and configure AWS resources.

App Runner connects directly to your code or image repository. It provides an automatic integration and delivery pipeline with fully managed operations, high performance, scalability, and security.

What makes AppRunner so compelling are these important features:

• Creates the entire infrastructure required for hosting a scalable web application
• Connects directly to your code or image repository AND can automatically redeploy the application when assets are updated
• Can scale up or down based on traffic
• Very cost effective - you only pay for resources consumed when your application is in use
• Associate your custom domain with an AppRunner endpoint.
• It’s much less expensive than provisioning all of the resources necessary to host your application yourself!

My Use Case

Back in 2012, I started a SaaS application (Treasurer’s Briefcase) for providing recordkeeping services for small non-profit organizations like PTOs, PTAs and Booster clubs. Back then, I cobbled together the infrastructure using the console, then started to explore CloudFormation and eventually re-architected everything using Terraform.

The application is essentially based on a LAMP stack - well sort of since I use a different templating web framework rather than PHP. The stack consists of an EC2 that hosts the Apache server, an EC2 that hosts some backend services, an ALB, a MySQL RDS instance and a VPC. There are a few other AWS services used like S3, SQS and EFS, but essentially the stack is relatively simple. Even so, provisioning all of that infrastructure using Terraform alone and creating a development, test, and production environments was a bit daunting but a great learning experience.

Starting with the original infrastructure, I reverse engineered it using terraforming and then expanded it using terraform.

The point being, it wasn’t necessarily easy to get it all right the first time. Keeping up with Terraform was also a challenge as it evolved over the years too. Moreover, maintaining infrastructure was just another task that provided no incremental value to the application. Time spent on that task took away from creating new features and enhancements that could provide more value to customers.

Enter AppRunner…with the promise of taking all of that work and chucking it out the window. Imagine creating a Docker container with your application and handing it to AWS and saying “host this for me, make it scalable, create and maintain an SSL certificate for me, create a CI/CD pipeline to redeploy the application when I make changes and make it cheap.” I’m in.

Not So Fast Skippy

AppRunner has evolved over the years and has become much more mature. However, it still has some warts and pimples that might make you think twice about using it. Back in 2021 it was an interesting new service, an obvious evolutionary step from Fargate Tasks which provide some of the same features as AppRunner. Applications that utilized Fargate Tasks as the basis for running their containerized web applications still had to provision a VPC, load balancers, and manage scaling on their own. AppRunner bundles all of those capabilities and creates a compelling argument for moving Fargate based apps to AppRunner.

Prior to October 2022 AppRunner did not support the ability to access resources from within a VPC. That made it impossible for example, to use a non-publicly accessible RDS instance. With that addition in October of 2022, it was now possible to have a web application that could access your RDS in your VPC.

The fall of 2023 has seen several changes that make AppRunner even more compelling:

Some of the limitations of AppRunner currently include:

• Inability to mount file systems (like EFS)
• While you can associate a custom domain, it cannot provision www from the console (in fact I have been unable to get it to work properly from the CLI either although the documentation indicates it should work)
• Still unavailable in some regions
• Cannot use security groups to limit access

The first limitation is a bit of show-stopper for more than a few web applications that might rely on mounted file systems to access assets or provide a stateful storage environment. For my application I use EFS to create session directories for logged in users. Using EFS I can be assured that each EC2 in my web farm accesses the user’s session regardless of which EC2 serves the request. Without EFS, I will be forced to re-think how to create a stateful storage environment for my web app. I could use S3 as storage (and probably should) but EFS provided a “quick-shot-on-goal” at the time.

The second limitation was just frustrating as associating a custom domain sort of kinda works. If I associate a domain managed by AWS (in the same account as my AppRunner application) then I was able to get the TLD to resolve and work as expected. AppRunner was able to associate my appliation to the domain AND provide an SSL certificate. It will redirect any http request to https. Unfortunately, I could not associate www sub-domain using the CLI as documented. In fact I could not even get the CLI to work without trying to enable the www sub-domain. Working with AWS support confirmed my experience and I still have a ticket pending with support on this issue. I’m confident that will be resolved soon(?) so it should not limit my ability to use this service in the future.

Conclusion

AppRunner is an exciting new service that will make application development and deployment seamless allowing developers to focus on the application not the infrastructure.

You can find the AppRunner roadmap and current issues here.

I’m So Done With Agile

Every development project ultimately has a goal of providing some kind of value to the organization that has decided to initiate a software development project.

The bottom line of any software development project is the bottom line. Does the cost of the project AND the maintenance of the project create a profit?

I know what you are thinking. Not all software applications are designed to produce profit. Untrue. Even applications we call “internal” create value or contribute to the creation of value.

What is Failure?

Let’s talk about and characterize failure first. Because its much easier to define (as anyone who has had the misfortune of working with a product development team that cannot define “done” knows). And I’ve been told that that most software development projects fail.

1. The project is canceled.

   This is the “first order broke” condition of projects. It took too long, it went over budget and looked to continue to be a money pit (someone understood the fallacy of sunk costs), the environment changed making the application moot or a new CEO decided to replace all internal applications with some SaaS, PaaS, or his own pet project.

2. The application was launched and did not meet the goals of the project.

   This can mean a lot of things: the project does not solve enough of the business problems to justify the continued cost of maintenance. Or perhaps the application did not generate enough revenue to justify its existence because of poor market acceptance. People just hate using it.

3. The project is in use, people use it, but the ROI is too far in the future or perhaps indeterminate.

   The project becomes a drag on the organization. No one wants to pull the plug because they have no alternative (or believe they don’t). There’s no appetite to rewrite, refactor or reimagine the application. It becomes a huge boat anchor that a handful of engineers keep running by kicking it in the ass whenever it stalls.

What is Success?

1. The project launches on time and under budget.

   Keep in mind that this is (mostly) a necesasry, but insufficient condition for success. Yes, there are some successful projects that are over budget or late, but its sort of like starting Monopoly owing everyone money. You need to catch up and catch up fast.

2. The application completely solves the business problem.

   Again, a necessary but insufficient condition for success. If the application is difficult to maintain and requires constant attention that costs more than it saves or produces, it’s not a success.

3. The application just works

   …and is a critical component in a complex workflow - without it nothing else would - its cost to develop and maintain is easily justified by the the nature of its job. It successfully completes its mission every single day.

This Was About Agile Right?

Oh yeah, Agile. I read articles about Agile and people’s experience with it all the time. I suspect most opinions are based on few data points and mostly from one person’s negative (or rarely positive) experience with Agile. My opinions (and that’s all they are…YMMV) are based on working with some fairly large clients that I am not at liberty to divulge. One FANG, one Fortune 50 company, one major manufacturer of phones and multiple companies with more than 5000 employees. I’m not opining based on one ride on the merry-go-round. I’m the kind of person that always believes that I just don’t get it, and I need to learn more, read more and accept more to overcome my ignorance and lack of experience. It’s a viewpoint that has allowed me to grow in my career and learn a lot of very useful things that have conspired to make me, if not wealthy, not concerned about money.

I am now having a lot fun going back to my roots of being a software developer. While I have been on the management side of projects employing the Agile process I am now in the belly of the beast. It smells bad, feels wrong and kills productivity. But, again, YMMV.

Why Does Agile Suck?

1. Product Owners - All “product owners” are not created equal. They have varying degrees of understanding of their own domain. Some even believe developers have ESP. To be fair, some expect developers (and rightly so) to “ask questions”. The problem is, what happens when the developer does not understand the domain. What questions should they ask? They are clueless.

   Product owners should assume nothing (in my opinion) and determine the level of domain expertise developers have. It is their responsibility to make that assessment - if they don’t they must be explicit with requirements, otherwise you’ll almost certainly end up with a project or feature that does not meet your needs.

2. Scrum Masters - are generally useless. The Vanna White’s of software development. I’d almos say Vanna had a harder job. Turning letters that have been lit up for you puts a lot of pressure on you to turn the “right” letter.
3. Ceremonies - Most developers hate meetings. There is of course the rare “meeting moth” that has decided it is easier to pontificate, hash things out, and “parking lot” ideas than actually code. But let’s talk about the programmers we all know and love. Leave them alone with a problem and some good requirements, decent tools and they’ll produce something for you. Developers generally loathe people. To hold meetings apparently you need to round some people up and torture them for 30 to 45 minutes every day. Oh, so let’s make “stand-ups”! We’ll cap them at 15 minutes and 20 people will get a fraction of a minute to recap what they did yesterday. Only there’s that one guy. Yeah, you know who you are. You think we care about your busy day and the fact that you worked through the night solving problems you created? Nope. Please, please, continue babbling so the time runs out on this farce!
4. Pointing - Sadly, there’s a poor Italian mathematician turning over in his grave as we speak. His claim to fame usurped by idiots who use numbers without understanding what they represent or the context for their use. That’s right - points mean diddlysquat. I’ve seen them used for billing by the scammy outsourcing companies that point their own stories and then bill accordingly. Or there is the pointing that uses them to determine how much “work” someone can do in 2 weeks. Ha! Gotcha! I’m pointing everying as 13! Is that Fibonacci?
5. Retrospectives - Fuck that shit. Let’s just do the Festivis pole, air our list of grievances and then proceed to the feats of strength. It would be more useful and of course highly entertaining. Hoochie Mama!
6. Sprints - We don’t know where we’re going but let’s sprint there! Isn’t that the irony of Agile? We don’t know wtf the requirements are until we start coding (according to some people’s definition of Agile) and yet we want to get it done quickly. It’s like watching my 3 year old son on his Big Wheel motoring down the street careening off of curbs and smashing this absolute wonder of plastic technology into my neighbors bushes. No one is seriously hurt, it’s a bit amusing but we really haven’t gotten anywhere.

So, here’s the bottom line. Any idea worth something greater than 0 that also has a wee bit of marketing behind it quickly becomes an opportunity for gypsies, tramps and thieves to exploit the ignorant masses. Take Christianity for example. Need I say more? Agile has become the Chrisitianity of corporate America. No one dare mention that is doesn’t solve our problems or make us feel any better. Fuck Agile, the ceremonies, the training, the roles the practice…it is the most unproductive enviroment one can devise for developing software. Look it up…Bill Gates wrote an entire BASIC interpreter and shoved it into 4K of a ROM. He then worked on a 32K version that was essentially a complete OS. He didn’t need Agile to do that.

So, let’s be clear. Agile is social engineering. An attempt to organize human beings in order to create something that no one of them could do alone (or so it goes). Somehow I don’t think Agile works. Some will say, yeah, well not every project should use Agile. Yes, that’s true, but the sad fact is that corporate America is not nuanced. They are binary. They want single solutions to complex problems and do not want to hear…it depends. And so they consume the entire bottle of aspirin.

There will be a day when people look back at the unproductive, waste and utter insansity that is “Agile”. They will marvel at the way that a single, possibly good idea for some things, was transformed into a dogma that haunted software development for a decade.

I’m hopeful however that really smart companies know that instituting things like Agile are the bellweather of their demise. They will avoid trying to fit round pegs into square holes. They will embrace the idea that you can plan things properly, but plans can change without embracing a chaotic, highly disorganized process that actually masquerades as a structured protocol.

You have been warned. When some consultant you hire to justify the outsourcing of your development team says that they can replace your current processes with an Agile team from Elbonia and a scrum master from Bumblefuck…be afraid…be very afraid. There is no free lunch.

One final thought…why is software development so hard? And why do we struggle so to create applications?

It’s not a hard question actually. The goal of software development is to codify a solution to a problem. But first…and here is the reveal…you have to define the problem. That is, in and of itself the most difficult thing in the development process. Missed requirements are, in my experience, the biggest reason for “re-work”. Note I did not say “bugs” or “defects”. Most maintenance on systems is because of missed requirements, not because programmers make mistakes. Oh, for sure, they do. But really? Think. Look back at your tickets and do a root cause analysis.

There are other reasons software development is hard. First, people do not communicate well. The do not communicate precisely and they do not communicate accurately. Next, the tools to express the solutions to our problems are complex and incomplete. Better ingredients make better pizzas. Papa Johns!

Okay, I have to wrap this up…Agile sucks. I hate Agile. I want to mute myself when I’m in stand-ups just to say every day “Oh, I was on mute.” and torture everyone that thinks this ceremony is useful.

Oh,I’m having issues with my internet so I may have to drop soon….open the pod bay doors Hal?