Automation and Architecture
The Customer Challenge
Supplying sufficient development and testing environments in order to deal with the high volume of output of new games has historically been a challenge. Coupled with this, the typical application environment is inherently complex due to the many moving parts of the current application architecture and the critical nature of the impact that development changes can have on data quality.
Security is paramount when working with new games and the existing intellectual property.
The Approach
Derivco were under significant pressure to meet delivery deadlines for the release of a new games and the existing VMware based infrastructure simply did not have sufficient capacity in order to deal with the current testing requirements.
The process began with a Proof of Concept in order to deal with a real capacity related issue that Derivco were experiencing during the testing of the payout percentages of a new games that were being developed.
Payout testing is conducted by running a large number of transactions through the system in order to build a dataset that is large enough to predict the statistical outcome of a particular game, with a high degree of confidence.
In order to facilitate the testing of the new games TechConnect worked with Derivco to replicate the “Gold Masters” into Amazon Web Services (“AWS”). The Gold Masters, approximately 30 underlying systems, form the basis of all testing and development environments within Derivco and are updated frequently with the latest builds of each of the many applications that constitute the application stack. The many applications that constitute an environment are a combination of .Net applications, database systems and testing client applications.
Replicating the environments into AWS was achieved by using a combination of the AWS Management Portal for VCenter (which is an offline copy) and a real-time replication agent that was installed on the systems that were being updated more frequently. The real-time replication agent allowed for changes to be transferred into the AWS environment as they were being made on the Gold Master images to reduce the amount of effort required in order to maintain the environment in AWS.
Once the images were successfully copied or replicated into AWS, an Amazon CloudFormation script was run in order to create the multiple testing environments that were required for the testing. The cloudformation scripts created all of the required components from networking, security and logging requirements through to installing some specific custom applications and settings that were required in order to perform the testing. The diagram below shows a high level design.
Additional tooling, developed by TechConnect, enabled additional automation through self service configuration and deployment of testing systems.
The Outcomes
Once the images had been copied into AWS and the CloudFormation scripts generated TechConnect was in a position to deliver 10 environments totalling approximately 150 instances and over 50 TB of OLTP based storage in a few hours. The project timelines were reduced from weeks and potentially months to days.
Higher testing throughput was achieved and Derivco were able to build a suitable dataset in half the anticipated time. This was achieved without making any further optimisations to the environment by leveraging additional AWS capability.
Once the testing is completed all the resources are terminated to avoid incurring any unnecessary expenses, resulting in a significant reduction in costs to deliver the service.
