About HOOQ

HOOQ is one of the leading video-on-demand streaming services in Southeast Asia. It was formed in 2016 from a joint venture between Sony Pictures Entertainment, Warner Bros and Singapore Telecommunications. HOOQ delivers a wide range of Hollywood & local content that’s ushering in an era of convenience for the digital consumer. As of today, HOOQ operates in the Philippines, Thailand, India, and Indonesia with a population footprint of over 1.6 billion people.

Challenges

In 2015, HOOQ entered the OTT market in the Philippines and gradually expanded its presence in Southeast Asia. HOOQ differentiated itself from its rivals in the OTT space by offering the largest catalog of local content, and this heavily contributed to its popularity in the region. HOOQ’s need at this point was to redesign its infrastructure to support the increasing demand. Since their deployments were handled manually; the maintenance window was high (~ 40 mins) and prone to human errors. With their current Monolithic architecture, any change in the older module would require the entire application to be deployed again. As a result, scaling became extremely challenging.

Goals

HOOQ realised that with some infrastructure improvements they would be able to meet customer demands quickly and scale massively. CloudCover came into the picture in 2016 to offer them completely managed infrastructure solution on Amazon Web Services. CloudCover has been evaluating and studying HOOQ’s current architecture since and moved HOOQ to microservices model that has emerged from the world of domain-driven design, continuous delivery, platform and infrastructure automation, scalable systems, polyglot programming, and persistence.

The end goal of this project is to automate the process, save HOOQ a substantial amount of time and help them achieve King Kong scalability in a highly available and reliable environment.

Infrastructure Components - AWS

Third-Party Tools

Strategy and approach

HOOQ’s infrastructure has more than 15–20 microservices running at any given time. Each of these services uses various AWS services such as Amazon S3, Amazon RDS, DynamoDB, and Redis. Configuring each service manually is a mammoth task. CloudCover is changing the way the entire infrastructure is managed and helping them leverage the power of Infrastructure as Code & Terraform. By introducing different templates of Terraform code for multiple services, CloudCover can easily easy bring up any HOOQ service with a click of a button. The idea here is to encapsulate complicated pieces of infrastructure behind simple & easy-to-use APIs and create reusable infrastructure with Terraform modules. This, in turn, results in a fully automated process.

Solution

A Journey to Automated Deployments

CloudCover has been effectively monitoring the current deployment process at HOOQ. The results showed that a considerable amount of their Developers’ time is spent on validating a manual deployment. By introducing a fully automated deployment process, HOOQ team has been able to devote their time in building the next awesome features for their application.

A simple yet powerful combination of Jenkins and Ansible is helping them in scheduling the deployments. CloudCover is automating all their Jenkins jobs (using in-house bot ‘CeCe’) for every service based on AWS. In certain cases where deployments fail, rollback requests are automatically initiated. Twilio sends notifications and alerts when a deployment fails & triggers a rollback.

CloudCover seamlessly integrates Jenkins and CloudWatch using AWS Lambda with Slack. This is helping in fetching deployment updates and important alarms in a timely manner. One can easily use Slack to schedule and monitor deployments. None of the technical resources have to go through the trouble of waking up in the middle of the night to deploy.

Achieving King Kong Scalability

CloudCover is helping HOOQ move from a Monolithic architecture to microservices model to improve their scalability. HOOQ is growing really fast and it is extremely crucial to meet the demand. They were witnessing situations where one application feature experienced a rise in demand and the entire architecture had to be scaled; this, in turn, affected other services’ performance. To address this, CloudCover is isolating the services for HOOQ into different services and reducing the risk of single point of failure.

CloudCover applies ECS containerization technique here and is using Application containers for running a microservices application architecture. Thanks to the seamless integration between Amazon ECS & AWS services, HOOQ is currently running in a highly scalable, performant, and self-managed infrastructure.

To ensure scalability is achieved at all times, CloudCover finely tunes all the Auto Scaling group scaling policies based on the load test results obtained & ensures that each service autoscaling group is configured to scale on the basis of their different load type.

Launching new services: A cakewalk

For launching any new service in HOOQ, CloudCover is utilizing Terraform to provision resources, ranging from a service (which includes, Amazon EC2, ELB, RDS, Amazon S3) to an entire infrastructure on AWS. By setting up Modules containing the essential code, a single codebase can be used for multiple services. One simply needs to variablize the core Terraform file with correct names as per service description and it is successful. To make the Terraform code more resilient, the codebase is securely stored on Github private repos, which is accessible only to a few users. The state files of the infrastructure provisioned via Terraform is stored in Amazon S3, which is centrally accessible.

Optimizing Savings

CloudCover regularly performs inventory checks that helps identify idle and unused resources. This, in turn, helps generate a report that specifies the areas where cost can be optimized. With the help of CloudWatch alarms (integrated with Slack and email), services can be monitored effectively. All the application level monitoring is easily achievable via NewRelic.

In order to ensure optimal use of resources and reduction in cost, AWS Instance Scheduler is set up in a way that the EC2 and RDS Instances are turned ON/OFF for Dev, staging, and testing environments based on a defined schedule.

Architecture Diagram

Results/Benefits

All of HOOQ’s deployments are not only fully automated but can be easily scheduled and monitored via Slack. This gives them a substantial amount of time to focus on their application and add new improvements for the growing user base.

Introducing automation in a full-fledged manner had removed the need for human intervention. The maintenance window has come down drastically and the time to deploy a new service has reduced as well.

Today, HOOQ’s user base in Southeast Asia growing immensely and they are successfully able to meet the demand and deliver exceptional content at lightspeed.