Resilient / High Availability

05 How do I make my app resilient?

This content is copyright of CloudCredo. © CloudCredo 2015. All rights reserved.

---

Feature

As a CF hero
I want my app to be resilient
So that random failures won't take it offline

---

Let's ship it

# From the training home directory:
$ cd 05-resilience/imperfect-app
$ cf push

...

urls: imperfect-app-votive-seeress.cfapps.io

...

---

You've shipped your new app!

The static website is handling the traffic wonderfully

Everyone wants to use your new app, but...

---

It crashes

Version 1 Sucks, But Ship It Anyway

Note: Coding Horror wisdom

---

How to make an app resilient?

Embrace failure & run many instances of the same app

$ cf scale imperfect-app -i 3
$ cf apps

name            state     instances   memory   disk   urls
imperfect-app   started   2/3         64M      256M   imperfect..

Note: Because CF is running your app, running many instances of it is just a command away

Notice the different IPs that your app instances are running on...

Even though each DEA runs on a different host, multiple app instances can end up on the same host

The more app instances you have, the less likely that they will be running on the same host

The next generation CF runtime - a.k.a. Diego - does a much better job in regards to evenly distributing the app instances

---

Many crashed instances, 
app still available

$ cf app imperfect-app

     state     since       cpu    memory         disk
#0   running   2015-11-02  0.0%   25.3M of 32M   66.9M of 128M
#1   down      2015-11-02  0.0%   0 of 0         0 of 0
#2   down      2015-11-02  0.0%   0 of 0         0 of 0

$ watch cf apps # Watch app instances restart in real-time

---

What restarts crashed apps?

Health Manager in DEA v2 (a.k.a. HM9K)

Health Check in Diego v3

Note: When an app instance crashes, the Health Manager dubbed HM9K will notice this and restart the app instance

---

My app needs more memory

$ cf events imperfect-app

description
index: 1, reason: CRASHED... Exited with status 255 (out of memory)

$ cf scale imperfect-app -m 256M

---

My app needs more disk space

Do not use the app's disk for persistence

$ cf app imperfect-app

     state     since        cpu    memory           disk
#0   running   2015-11-02   0.0%   24.4M  of 256M   95.2M  of 256M
#1   running   2015-11-02   0.0%   112.4M of 256M   224.2M of 256M
#0   running   2015-11-02   0.0%   24.3M  of 256M   95.2M  of 256M

$ cf logs imperfect-app --recent

Errno::EDQUOT - Disk quota exceeded @ io_write - infinite-file:

$ cf scale imperfect-app -k 1G

Note: Use a service for persistence, do not store any files in the container that runs the app

---

Scale instances, disk & memory

Combine multiple options in a single command

$ cf help scale

NAME:
   scale - Change or view the instance count, disk space limit...

USAGE:
   cf scale APP_NAME [-i INSTANCES] [-k DISK] [-m MEMORY] [-f]

OPTIONS:
   -i       Number of instances
   -k       Disk limit (e.g. 256M, 1024M, 1G)
   -m       Memory limit (e.g. 256M, 1024M, 1G)
   -f       Force restart of app without prompt