Deploying your code to production doesn't have to hurt

Posted in CI/CD, Automation, Deployment

This blog post is at the back of a conversation I had on Twitter with a few people about how (not) to deploy code to an environment.

It all started after reading a blog post by Jesse Liberty with the title Azure For Developers – Part 3: Deploying from Visual Studio. I have huge respect for Jesse and I know he's been around for much longer than I've been. However, I felt that he omitted something very important from his blog post. The fact that although deploying from Visual Studio works great and is super convenient, this is something that should be avoided at all costs, especially in production or an enterprise environment.

Many have said that this is not so bad and that developer should use common sense and best practices when deploying code. Unfortunately, if reality has taught me one thing, is that convenience and laziness will trump recommended practices or industry advice.

Before we continue, let me just say that I've been there and I've done all the stupid mistakes. I've worked on many projects with silly deployment processes and where security and consistency or automation is not in anyone's vocabulary. Sadly, this has been (and still is) the reality for many developers and organizations even in 2017! Since I joined Microsoft I've had the opportunity to experience how many different organizations work and how they approach change and release management. I've seen the good and the bad and I often relate with some of the developers working on these project, because I see myself going through the same pains, issues, late nights and wild firefights with code run in production. And it all starts with the deployment. I'll leave monitoring and logging for another post.

What's so bad with pushing from Visual Studio?
Imagine a team of 10 developers, all working on the same project, all with the ability to push changes to an environment (dev, test, qa or prod). Each developer has full rights to all the environments and the ability to push code unrestricted. Imagine this. What could go wrong, right? Some of the things that could go wrong include (but are not limited to):

  • security: pushing the wrong config files with the wrong credentials. Or even having access to production credentials. Or the fact that any developer has permissions to connect and push to an environment and in some cases even to production! This information, readily available in clear text, scattered through config files, post it notes, virtual post-its and notepads on developer's machines are a hacker's paradise and an enterprise's worse nightmares
  • accountability: who was the last person to push to production? Was his code vetted by the rest of the team (I won't even mention release and change management here)? Which version of the code was pushed? Did he test the code before pushing it out to an environment?
  • repeatability/consistency: does every developer publish and push the code in the same way? is there a deployment document which enumerates the steps? How do you keep track if all the steps have been followed? What happens if you skip a step? Is it easy to reproduce the same outcome and the same artifacts each time (assuming the code hasn't changed)?

Usually, the lack of proper CI/CD means that there are other problems with the overall ALM in the team. Lack of testing (automated or manual) is usually the first one to pop out. Lack of good security practices is also quite common, with hard coded secrets all over the place (config files, inline etc). In some rare cases, some projects may lack source control, though it's been very long since the last time I came across such a case. Finally, change management, versioning etc are also a common theme, which is sad.

Can people still get deployments wrong in 2017?

Do people still struggle with code deployments in 2017? Unfortunately, yes they do. Deploying code is complex, but developers like to make their lives miserable either due to lack of time, interest, knowledge etc. If you're reading this and you're nodding your head, stop and think:

Is this really the best way to do this???

For every single thing you do that costs you time and effort, use this very important question and then do your search. The examples I have below are things that we still see today in production environments, sometimes even in reputable organizations. So, what kind of bad deployment practices exist in 2017?

  • Stick the executable to a USB stick to distribute the application to the end users
  • VS Deploy to an environment
  • VS Deploy to disk, copy the artifacts to the target environment.
  • Zip the code and ship to production along with a 60-page word document with manual steps and checklists
  • FTP to production
  • sFTP to production (at least we're secure)
  • A bunch of PoSH, Bash, Batch scripts that may do all of the above

Full Disclosure: At some point or another, I used some of these and I'm ashamed to admit it, but I had to learn the hard way. Thankfully I also had the chance to work with some really clever and experienced people that pointed me in the right direction. It's not until you're out of the "disaster zone" that you look back and realize how bad it was.

So why are these practices bad? There's been a recent example where a new employee managed to wipe out the production database as he was setting up his local dev environment. The same manual process, scripts, and manuals used to setup the local dev environment was the same ones used to deploy to production. Just different credentials. Now, imagine you being that employee, on your first day at work. Horror stories like this are written every day but they're not always made public.

Now let me stop you here and tell you that there's no right way or one true way for deploying code to production. Each team, each project, each company has different requirements, different procedures, and different tools. But there are definitely some common themes and recommended practices when it comes to deploying code outside your own local box.

What does a proper CI/CD have to offer?

You don't have to own an army of developers or be part of a complex team to consider CI/CD a necessity.

Even as a solo developer I use various tools to keep me in line. What do I mean? Raise your hand if you've lost code because you didn't use source control. Even for your pet/side projects that you wrote at 11pm. Thanks for your honesty. I've been there, done that and I learned my mistake. These days one of the first things I do on any project is to run:

git init
git add .
git commit
git push

Never before have 4 commands saved my skin as have these. They come highly recommended :)

Source Control

Every project needs solid Source Control. Whether you prefer Git, BitBucket, TFVC, Subversion etc, I don't care. I'm not precious about tools or the platform, as long as there's something. If your team prefers Git then great. If your team prefers VSTS, then super awesome. Just make sure you use the right tools for the job. Whether there's only one developer or 100. There should be consistency.

Source control can also help improve code quality, security, and performance. Pull requests, for example, provide visibility of changes to the whole team and should be used as a learning tool for less experienced developers. In addition, they can help improve communication within the team and offer the chance to provide constructive feedback. Open Source projects have been operating like this for a long time with proven success, so why should enterprises not adopt the same successful model?

I've been asked to work on a project that has no source available! Seriously. I had to use ILSpy and Reflector and that was only the start of my painful journey. The developer who wrote the application left and his machine was wiped out. The code was gone with him/her and only the production copy was left behind. The company hadn't moved, upgraded or patch that application in 5 years! And all this because they didn't believe (yes, right) in source control. If you're not using source control today, I implore you to start using it ASAP. GitHub is free for public repositories and has a cost of 2 Starbucks coffees for private repos. VSTS has unlimited private repos etc. Just find the tool that best fits your needs.

Continuous Integration

As soon as you commit your code to source control, you want to build it and publish the artifacts (what's needed to run your application). This is the CI bit of CI/CD and it comes with great benefits. For starters, you'll quickly realize that you forgot to commit half of the files that are necessary to run your application! Or that you have specific tasks that don't run outside your local environment. And since we tend not to ship individual developer machines to production, we need a reliable and repeatable way to establish that the code compiles and produces the desired output in a controlled environment. The continuous integration process is usually split into various parts and stages. Usually, there are build tasks that run during the day and tend to be quick and targetted and there are nightly build tasks that tend to take longer and perform longer slower tests.

An integral part of Continuous Integration is running code tests of any form (integration, functional, unit, UX etc). Apart from the reassurance that your code can build outside your local environment, you also need to ensure that your latest changes haven't introduced a bug and meet the acceptance criteria. I hope that you're still with me. If not, I would suggest that you invest serious time on how to write reliable, testable code.

As you think about how to best implement your continuous integration you should also be thinking about the portability of your code. How does your application consume configuration settings and where do these settings live. How do they change across various environments and who controls these settings, especially secrets such as API keys, database connection string, cloud resource keys etc. A good CI/CD system should allow you to tokenize or provide them at build time and depending on the target environment, different people should have different access levels. Developers should never have access to production credentials (which, by the way, should be frequently rotated).

I've recently been to a company to review their existing processes. One of the questions was around deployment credentials to their cloud provider. It turns out that the Ops team has been using the same credentials (one username and password) for years. People that have left the company and the team, still have access to the production environment and the ability to wipe out the whole subscription!

Continuous Deployment

This is the part that deploys our code to an environment. Although we say continuous deployment, the process is a lot more controlled and ideally staged to allow for release management. Some deployments can be automated and continuous, to help validate that the code works as expected. Other deployments can be automated but triggered manually (think of production). However, there are companies that deploy 10-20 times a day to production, which is mighty impressive. Most teams I deal with are nowhere near that cadence, but having the ability to push once a week is still a massive win for many, considering some of them only deploy every 6-9 months.

Deployments should be fully automated, where possible. If automation is not feasible, look at your workflow and identify pain points that can be eliminated through scripting or deployment tasks. Each manual step in your deployment process is a step that things can go horribly wrong. It's also a step that prevents you and your team from having a "One Button Deployment to Production" as a former boss of mine used to say. You need that magic "button" and anything that stands in your way needs to be eliminated.

Amazon's AWS recently experienced a severe downtime in production when a manual step failed due to human error. You can read all about it [here](https://www.recode.net/2017/3/2/14792636/amazon-aws-internet-outage-cause-human-error-incorrect-command" target="_blank). You should now be able to understand that no matter the organization or the team, it's important to eliminate manual steps from the release process.

Automation has another benefit: repeatability. The release process should be reliable and repeatable. No matter how many times you run the process the expected outcome should always be the same. This, in effect, should provide you with confidence that you can repeatedly deploy your application to an environment and the application will run as expected. The release process should have the same steps across all target environments with the configuration settings being the only differentiator.

Another example that I know many in the .NET world will have experienced. Raise your hand if you had to manually manage web.config files in when releasing to production. Although web.config files support transformation and environment specific properties, many teams are still unable to manage these files properly. I can't remember the amount of time and effort I spent trying to sync web.config files when doing a release due to bad practices. It usually went like this:

  • ask for a copy of the production web.config (i didn't have access due to security! LOL #securityfail)
  • add the new properties and config settings
  • make sure you have a copy in case it all goes horribly wrong
  • usually the prod file was out of sync and out of order with dev, which made editing a proper nightmare
  • use 3 different editors to validate that the XML is valid
  • attach the new web.config file to the release candidate
  • keep 10s of copies of said file around because Source Control doesn't apply here
  • cross fingers and legs when deploying to production and hope for the best
  • in case of a failure, run as fast as you can
  • sometimes people would edit web.config files directly in production to fix the problem, forgetting to propagate this change back to dev or check it in source control.

If all of the above sounds familiar it's because in 2017, this is still a thing. side note at least the .NET Core team made it easier to manage these changes with environment variables, but people can still get it wrong.

What tools should I use?

I've used most of the tools that are out there. I'm confident that most of you, my readers, have too. I'm really thankful for how the industry has evolved and the tools we have at our disposal today. I mentioned earlier that there's no one right answer nor one perfect tool. There are tools with pros and cons and each one of us has its own preferences. My advice would be to test the tools and frameworks that are out there and come up with the right combination that meets your needs and budget:

  • Visual Studio Team Services
  • GitHub
  • AppVeyor
  • TeamCity
  • Jenkins
  • Octopus Deploy
  • BitBucket
  • Etc

I would like to leave you, my dear reader, with one message and one challenge:

Imagine if all the developers out there, instead of fixing mundane bugs and configuration issues were able to work together to propel humanity to next industrial revolution. Automation is the first step in the right direction.

So now that you what we could achieve, your challenge is this: take a good hard look at your code and process today and try to identify areas that can be improved. Ask that question: Is this the best way to do this? Then go out and find the best way and implement it. Then use your newly found knowledge to teach others. We can all be better than this

  • Share this post on