Gold Coast Business Awards Logo Winner

Gold Coast Business Excellence Awards 2021

The Gold Coast Business Excellence Awards launched in 1996 and in 2021 we are celebrating our 26th year. Primarily beginning as a vehicle to recognise family business, the Gold Coast Business Excellence Awards have developed to now encompass a category for almost every type of business enterprise.

Today, the Awards seek to contribute to the self-image and identity of Gold Coast business and to foster pride in this business community. They also seek to highlight the growing importance of the Gold Coast as a business hub and recognise leading businesses with a view to encouraging others. The Awards and all intellectual property fall under the ownership of the Gold Coast Business Excellence Awards Association Inc.

This is the second year that TechConnect has entered this award and are thrilled to have won the first month (May) in category of IT & Digital Business. We would like to extend a congratulations to all the other winners and a massive thank you to the sponsors and organisers. We look forward to the annual awards later in the year and competing with some of the amazing businesses on the Gold Coast.CEO of TC receiving the award

The IT and Digital Business Award, sponsored by Griffith University, is the recognition of Gold Coast businesses demonstrating outstanding knowledge management based, information technology products and services. Applicants should highlight their smart technology product or service and how it benefits everyday life and impacts the businesses bottom line. Today we live in the Information Age. Knowledge management underpins almost all services today, and the effective management of knowledge distinguishes companies in IT, supply chain management and logistics. Excellent information technology, knowledge management and digital services based businesses may exemplify this in their own business, or enable and enhance the practices of other businesses through their IT, logistics or supply chain solutions.

Migrating Relational Data into an Amazon S3 Data Lake

5 V's of Big Data

The concept of a data lake is not new, but with the proliferation and adoption of cloud providers the capacity for many companies to adopt the model has exploded. A data lake is a centralised store for all kinds of business data:

  • unstructured – images, videos, PDFs, Word documents
  • semi-structured – JSON, XML, spreadsheets
  • structured – CSVs, RDBMS tables, tabular spreadsheets

typically stored in a format as close to the raw source as possible. This is to take advantage of the elastic storage and to minimise bugs arising from transforming the source data before landing in permanent storage. Of these, many organisations will find that a large proportion of their business-critical data will be hosted within one or more relational database management systems (RDBMS). These sources, therefore, will be high on any priority list to be extracted and landed into the data lake to break down the data silos built around them and to democratise access to the data contained within.

In any data engineering exercise, it is important to make assessments of the 5Vs for each data set. The 5Vs are:

Velocity how quickly the data is created and changed and what are the latency requirements that your your pipelines must satisfy?

Volume how much data there is both in total and per item?

Value what is the value that the data provides the business and how does this change as the data ages?

Variety how many different shapes of data are there (for RDBMS the variety should be very low with the occasional NULL field or free text column) and how does this change over time (how do you deal with slowly changing dimensions)?

Veracity how consistent and understood is your data? is the process by which it is created, moved and transformed documented, validated and repeatable?

With these attributes understood for the dataset in question the particular pattern of ingest into the lake should be straight forward to determine.

If the data is high velocity (requires low latency) and/or every change to every item represents value to the business, then a replication solution like AWS DMS (data migration service) is appropriate.  The challenge with such an approach is that the data will land in Amazon S3 in CSV or Parquet format with the operation ([I]nsert, [U]pdate or [D]elete), table name, database schema name followed by the column values for the new row state (or the deleted row data). In general, this format will not be ideal for consumption via BI services or for direct import into downstream data warehouses or data marts. Consequently, some form of “hydration” into a point-in-time state will be necessary.

If the velocity is lower (hourly, daily or above) then batch processing is a valid option. This can take several forms:

If the volume is low, then a simple and effective approach is to ingest the entire table according to the schedule. This ensures that historical updates are captured in addition to new data. This can be achieved using AWS DMS, AWS Glue or the databases native export capabilities.

If the volume or velocity is such that complete table copy is ineffective then full backup followed by incremental delta capture is necessary. How the deltas are defined will depend on the data available within the table or from domain knowledge of how the upstream table changes.

If the data in the table is immutable then the deltas can be captured by keeping the last compound key for the table. AWS Glue with Job booking marking enabled is an implementation of this pattern.

However, many tables are transactional and will support updates which tracking a table watermark will not identify. If all of the intermediate states of records don’t matter at the scheduled granularity then a column in the source table that captures the date-time each record was last updated can be used.  Then the delta is defined by rows with an updated time after the last data ingest. With appropriate indexing this lookup can be made efficient and only changed rows will be imported. The challenge then is how these deltas are applied and how data hydration is achieved. Using Apache Hudi via Amazon EMR or via a custom AWS Glue connector enables this transparently provided the Amazon S3 destination is defined as an Apache Hudi table.

If “updated at” columns are not already present in the source table then business rules of thumb can be utilised instead. For instance, if it is known that certain business processes run on a known schedule (e.g. financial reconciliations) such that the data is frozen after review then this can be used to define the deltas. No data for ids existing prior to the last freeze need to be brought over. Any records with ids after are replaced according to the schedule. The challenge, like for updates generally, is how to integrate them into the existing Amazon S3 storage. In this instance, if the Apache Hudi solutions cannot be brought to bear then Amazon S3 table partitioning can ensure the deltas can be applied efficiently. By partitioning the data in Amazon S3 according to the data freeze schedule then only the most recent partition of data needs to be overwritten. The downside to this approach is that the freeze schedule will not result in a partition scheme that aligns with the data query patterns arising from downstream analysis services. To overcome this additional data transforms can restructure the data as required.

These ingest patterns should cover most of the combinations of value, variety, velocity, veracity and volume data scenarios arising from importing relational data into an Amazon S3 data lake.

TechConnect IT Solutions_Technology

What a hybrid workforce in 2021 means for your security and data strategy

Many workplaces spent 2020 suddenly working from the home. With this large shift to a remote workplace, many organisations were able to witness first-hand the ability to maintain productivity even when geographically dispersed.

Where previously there had been a fear that separating the workforce would create a variety of insurmountable problems, 2020 showed us that the people are sufficiently adaptable and motivated to keep working even through tough times.

This year, many organisations have started to bring staff back to the office, at least part-time. But with people accustomed to a new kind of work-life routine, these new habits and behaviours have given rise to the so-called ‘hybrid workforce’. This is where staff work regularly split their time between home and office and have the agility to decide not only where, but when they work. This gives staff the flexibility to fit work around the rest of their lives, rather than fitting their lives around work.

And it appears the hybrid workforce is here to stay. Research from Radware shows that 83 per cent of C-level executives expect the changes they made in the areas of people, processes, and applications as a response to the COVID-19 pandemic to become permanent. Meanwhile, a global study conducted by Lenovo showed 52 per cent of workers want to continue to work from home more than they did pre-COVID-19, even after social distancing is eased.

Data Security

Despite the apparent success of this kind of hybrid workforce, there remain challenges that need to be overcome before it completely enables secure and effective collaboration for employees.

Key among these is the aspect of data security. Covid-19 triggered an avalanche of cloud migrations as business pushed apps and infrastructure to the cloud to quickly shift operations out of office. 76 per cent of companies adopted cloud services faster than they had planned.

At the same time, a McAfee Labs COVID-19 threat report found that threats against cloud services increased by 630 per cent over the course of the pandemic thus far. Attackers are now using credentials uncovered via phishing campaigns to exploit the anonymous, decentralised nature of cloud applications. The focus has shifted from IT infrastructure and hackers are exploiting employees.

To secure data and devices, businesses need to maintain a 360-degree view of devices, to keep them protected from all threats. Because there isn’t a one-size-fits-all app, system or service that will fully protect your devices, you need make sure that there is interoperability and integration between your security solutions.

While this might feel like an added layer of complexity, the overlapping areas of protection provided by multiple solutions will keep your employees data and devices better protected.

Data Diffusion

With traditional approaches to cyber-security not applicable in a hybrid workforce, organisations need to protect remote and mobile users consuming cloud apps outside the protection of your business network.

As data is shared across encrypted internal, external, and third-party accounts, it’s easy for businesses to lose sight of what is being sent and where it’s going. Threats from the cloud are multiple and add another potential angle for bad actors to conduct phishing and ransomware attacks on unsuspecting businesses.

That threat is forcing organisations to look at holistic security solutions that protect across SaaS, IaaS, and the web.

Cloud-Native Security

Migration to a cloud-native security solution can deliver a significant risk reduction and an improvement to user experience, with relatively low effort.

Adopting cloud-native next-generation gateways grant businesses the ability to control web access, provide data and threat protection, and decode thousands of cloud services and apps to provide extensive policy control.

Data-centric cloud security protects you across managed and unmanaged cloud services, apps, and web traffic. All while it continuously assesses public cloud instances for possible data exposure.

Protection Anywhere, Everywhere

Many companies reacted quickly and provisioned staff with the technology they needed while maintaining support for customers. But being reactive only works in the short term. Businesses need to work on a long-term plan that provides security to employees and protects data and devices in a new and vulnerable location.

The time has come for a more strategic approach to security as companies settle into new hybrid workforces that support remote work.

Get in touch with us to discuss how we can help you to govern, secure and protect your mobile workforce.


Geostream TM

How can businesses improve mobile customer experience?

Geostream TM


A couple of years ago one of our clients approached us with a difficult question:

“How can we understand and improve mobile game performance and player experience in targeted locations around the World?”

This was not easy to answer.

Simulating current game performance

The first step was figuring out how to simulate game performance on mobile devices across different geographical regions and mobile networks.

We considered simulating latency and throughput via selected data centre virtual machines with traffic shaping, to introduce errors and real-world experiences, but we quickly shelved this idea as it does not account for the many factors that may impact mobile network performance.

You may ask, “Why is this a challenge?”, so think about these problem statements:

What is a person’s experience in Accra Ghana or Novosibirsk Siberia?
Why are some geographies or locations performing poorly?
Which mobile networks indicate worst player experience and why?
How do I measure any improvements once implemented?

We concluded that there was no solution available for our customer. We promptly set about building one.

The idea, created by the team, was to build a device which would mimic a mobile phone, a customer and that could be deployed on site at each targeted location. Using physical mobile phones was not a viable option due to battery, heat and other reliability issues.

The mobile devices needed to fit the following criteria:

  • Capable of running 24/7 on a continuous basis.
  • Be compatible with local SIM cards for the mobile networks.
  • Processor capacity should be close to that of a mobile phone.
  • Must be small, transportable and robust.
  • The device would need to be completely automated, with almost no user intervention.

Our team tested a number of microcomputer options and finally settled on a customised version of the Raspberry Pi4. Additional 4G modules, external aerials, a cooling system and a custom designed aluminium housing were added to the package. Delivering the capacity of 4 mobile networks (SIMs) per housing, now named GeoStream™.
Geostream TM

Each GeoStream™ enables 4 mobile networks to be performance tested and monitored per location.

Data-driven decisions are the core of TechConnect’s capability, delivering solutions to our customers and unlocking value from data. It, therefore, goes without saying that we must collect and analyse the data. GeoStream™ includes the ability to:

  • Capture the data and send it back to a centralised datalake in Australia.
  • Deploy multiple devices per location for broader testing and data collection.
  • Securely house and cool devices to deliver robustness in remote locations.
  • Reporting and analytics platform to analyse the data and enable data-driven decisions.
  • A scheduling platform for scheduling activity and runbooks.

Collecting the data

Custom Python scripts automatically connect the devices back to the datalake and control system in Sydney. Data is analysed via a web-based front-end application.

The GeoStream™ front-end is built on Angular and utilises a number of Amazon Web Services capabilities. The front-end gives administrators the ability to assign role-based access controls for roles such as admin, tester, scheduler and analytics.

The first revision of GeoStream™ was released late 2019, with the very first unit being deployed to Ghana. GeoStream™ has assisted to deliver improvements across both the MTN and Vodafone networks in Accra. The Ghana unit continues serving analytics and test result data to the datalake in Sydney. GeoStream™ revision 2 was released in July 2020 and is destined for regions throughout Canada, New Zealand, India, Taiwan, London and Estonia later in 2020, with many more expected to follow.

GeoStream™ has delivered a solution to mobile network performance analysis, from within country and satisfying many of the challenges facing mobile network analysis. The product provides a commercially viable tool that gives our client an accurate view of their mobile users experience on all their applications, no matter where they are and at any time of day.

“We love to innovate and when innovation intersects customer value we have made a real impact.” Mike Cunningham – TechConnect CEO.

TechConnect plans to roll out many GeoStream™ devices in conjunction with our private content delivery network, named Slipstream, to improve mobile player experience for our customer’s customer.

Gold Coast Excellence - Queensland

Gold Coast Excellence Awards 2020

Gold Coast Excellence Awards 2020 - IT and Digital

The Gold Coast Business Excellence Awards launched in 1996 and in 2020 are celebrating our 25th year! During this time the Awards have grown to be recognised as the region’s most comprehensive and prestigious business awards scheme, offering specific and meaningful benefits to the wider Gold Coast business community.

TechConnect is super proud to support the Gold Coast community, growing locally, nationally and internationally.  With a presence in Brisbane, Melbourne, Perth and South Africa we have grown from our humble base in the Gold Coast, Queensland.

This was the first year that TechConnect has entered this award and are thrilled to have won the category of IT & Digital Business. We look forward to the annual awards and competing with some of the amazing businesses on the Gold Coast.

Gold Coast Excellence Awards 2020 Queensland


Pictured below is Mike Cunningham – CEO of TechConnect (left) and Fabrizio Carmignani – Professor of Economics Griffth University (right).

Gold Coast Excellence Awards

TechConnect IT Solutions_Data

Digital Transformation with Data

TechConnect IT Solutions_Data

Harnessing Data to drive effective digital transformation

The COVID-19 pandemic has made clear that businesses need to be prepared for flexible, remote working practices.

As lockdowns forced offices to close and people headed home to limit the potential spread of the virus, many organisations found they weren’t prepared to provision the necessary work from home (WFH) technology and processes for their staff to continue with business as usual.

As a result, businesses have been required to undertake (or accelerate) a significant digital transformation journey to get up to speed. As these transformation journeys roll out, the need to harness data effectively becomes more critical than ever for a successful, long term change. Here’s what you need to consider.

A strategic approach

Before beginning a digital transformation, it’s critical to have a strategy in place to explain how you will manage, store, secure and use your data. Yet, this is a step that’s often forgotten in the rush to transform and digitise processes.

A data strategy should be driven by the needs of your business. Your strategy will also define how to make decisions about the use of data, more capably manage data flow, and secure information effectively.

Any successful plan will identify realistic goals along with a road map for rolling it out. This ensures that you’re properly prepared for every step of the journey.

Beginning the journey

A digital transformation unshackles an organisation from the past. It empowers you to move into the future, free of outdated technology and slower manual processes.

For example, take mobile and cloud technology. While we were once restricted to an office environment for productive working, it’s now possible for geographically diverse teams to collaborate as efficiently as they would in a traditional office setting. Files, apps, and other resources can all be accessed remotely, and meetings held virtually, giving workplaces and workforces the ability to be truly flexible.

However, the reality of a digital transformation is that with staff spread across locations, there are a range of new infrastructure management issues to consider. Chief among these is data security.

Keeping data safe is vital as users access business networks and devices remotely, often without the protection provided by robust on-site architecture. It’s important to decide how you’ll service and secure company devices, and how to make sure users and the data they handle and generate will be protected, and implementing those systems early.

Harnessing the power of data

With a clear data strategy in place and your digital journey underway, you can start to take advantage of the power of your data and use it to drive improved decision-making internally and externally.

Artificial intelligence (AI) and machine learning (ML) can be used to sort your unstructured data, learning as they go to uncover valuable insights. Once the data has been cleansed, you can enrich it by adding third-party data or public datasets to uncover more hidden insight.

The adoption of AI and ML also frees your people for bigger picture tasks. Instead of manually sorting through stacks of data, they can concentrate on delivering valuable and creative work powered by the insights you’ve identified – ultimately working towards the goals outlined in your data strategy.

Gathering data helps to deliver external benefits to your business too. It can improve customer service by identifying current pain points or uncover new customer segments for targeting – the possibilities are endless!

The lesson in the journey

Businesses shouldn’t underestimate the change that needs to be undertaken in digital transformation journeys. They require significant planning and thought before beginning. However, while the challenge is large, data can make the journey less difficult and more successful.

Accessible, accurate and relevant data enables businesses to make better informed decisions and deliver actionable insights. And by establishing a data strategy up front, you can better understand, apply and secure your data to meet the needs of your organisation.

If you’re asking yourself questions such as “Are we doing things the right way?” or “Can we do this better?” why not get in touch and let’s explore how TechConnect can deliver results for your business as you undertake a digital transformation.

TechConnect achieves AWS Data and Analytics Competency

TechConnect Directors - Amazon Web Services - Data and Analytics Competency

AWS Data and Analytics Competency

Proves technical proficiency, operational excellence, security, reliability and 360-degree customer delivery capability; Cites major client projects with Virgin’s Velocity Frequent Flyer and IntelliHQ.

TechConnect IT Solutions (TechConnect), a leading provider of cloud services and an Amazon Web Services (AWS) Advanced Consulting Partner, today announces it has been awarded AAWS Data and Analytics Competency certification; the only Advanced Consulting Partner in Australia to achieve this prestigious competency level.

The AWS Competency Program recognises partners who demonstrate technical proficiency and proven customer success in specialised solution areas. TechConnect undertook a rigorous partner validation process to be awarded the certification, including an independent audit of its technical, organisational, governance and customer capabilities; along with scrutiny of large scale, in-production customer deployments.

Customer case studies that were reviewed as part of the certification process include a customer insights project with Velocity Frequent Flyer; a predictive medicine data platform for IntelliHQ that uses heart rate variability to predict patient outcomes; and a big data analytics project with Kamala Tech that gave technical users data to form insights across the business including areas such as data science, machine learning, marketing systems, reporting and self-service capabilities.

As healthcare comes under more and more pressure to deliver quality personalised care under constrained budgets the healthcare industry is seeing innovation with the use of data to drive efficiencies and better patient outcomes. As Machine Learning and Artificial Intelligence (AI) emerge as a driver for businesses to do more with less, healthcare can deliver better care with the same resources using data to drive out those efficiencies.

“We partner with industry in AI as we need as many talented and gifted people in this space as possible. said Dr Brent Richards. is the Medical Director of Innovation – Gold Coast Hospital and Health Service (GCHHS). “There is a lot that the industry can bring that healthcare specifically does not have in terms of hardware, software and talent.” Dr Richards played a key role in the project with IntelliHQ which is a partnership between Gold Coast Health, industry and universities to transform healthcare through AI, enhancing patient outcomes and improving quality of care, while maximising cost-effectiveness.

Oliver Rees, Chief Analytics Officer with Virgin’s Velocity Frequent Flyer program has commended TechConnect’s expertise and cites the direct benefits for member experience. “Velocity have always been a company that is passionate about using insights to understand and improve on their members’ experience. Velocity worked with TechConnect to build a platform that would allow Velocity to combine member insights in a single location to make it easier for members to then receive relevant program offers,” said Oliver Rees, Chief Analytics Officer – Velocity Frequent Flyer.

“In achieving this level of competency with AWS, TechConnect has demonstrated our ability to help customers solve their most challenging data problems within large scale production deployments. We proved that we have deep expertise in designing, implementing, and managing Data and Analytics applications on the AWS platform and have delivered solutions seamlessly in the AWS Cloud environment,” said Clinton Thomson, Director of TechConnect IT Solutions.

TechConnect is a fast-growing Australian company, headquartered in Queensland and serving clients around Australia and worldwide. TechConnect helps customers extract business value from data and it has plans to grow its team to 100+ people over the next three to five years, creating graduate employment and professional development opportunities in Queensland and throughout Australia. The company has offices in Brisbane and the Gold Coast and has a graduate pathways program for top students in the STEM fields.

CRN Fast50 Award

CRN Fast50 for 2019 Award

CRN Fast50 - Number 15TechConnect was listed in the CRN Fast50 for 2019, settling into the 15th position in our debut year. A day after receiving the Deloitte’s Technology Fast 50 awards we were again presented a great result. Read about the Deloitte’s Technology Fast 50 here if you missed it.

The CRN Fast50 award, now in its 11th year, recognises the fastest-growing companies in the Australian IT channel, based on year-on-year revenue growth. “The 2019 CRN Fast50 put up astounding numbers: they grew at least 15 times faster than Australia’s economy.” – Simon Sharwood, Editorial Director – CRN. CRN Fast50 Award

This was the first year that TechConnect has entered this award and are thrilled to be listed 15th, with a growth rate of 67%. Simon Sharwood also stated, “It’s a huge achievement to have made it into the CRN Fast50. Your company’s growth not only outpaced most others in our industry, it also vastly exceeds Australia’s current overall growth rate!”.

This is a great achievement for us, we would like to thank our customers and team, without them this would not have been possible. Team TechConnect would also like to extend a congratulations to all the other winners. TechConnect looks forward to expanding our growth and moving up the list in coming years.CRN Fast50 - Clinton Thomson

Deloittes Technology Fast50

Deloitte – Technology Fast 50 Australia 2019

Deloittes Fast50 AwardWe are extremely excited and honoured to have been listed as one of Deloitte’s Technology Fast 50 Australian companies. Ranking 43rd on the list with a growth rate of 161%, in our debut year. This is a huge achievement not only for the company but also for our team and the hard work they have put in to provide solutions for our customers.

“Now in its 18th year in Australia, the Deloitte Technology Fast 50 program ranks fast growing technology companies, public or private, based on percentage revenue growth over three years.” – DeloitteDeloittes Technology Fast50

“More than ever, this year showcases world-leading Australian business innovation and it is a tremendous achievement for any company to be named among the Deloitte Technology Fast 50,” Deloitte Private Partner and Technology Fast 50 Leader, Josh Tanchel.

With our significant growth over the years, our hard-working team and most importantly our valued customers, we were able to achieve this amazing outcome. TechConnect is getting ready for exponential growth, through expanding into new markets and deep data specialisations.

The awards night was held on Wednesday, 20th of November in Sydney at the Museum of Contemporary Art. Clinton, our Director, was there to accept the award on behalf of TechConnect. “It was a great night; Well done to all the other recipients and thank you to Deloitte’s for putting it all together.” Clinton Thomson, TechConnect IT Solutions.Deloitte Fast50 - Clinton Thomson

IntelliHQ - AWS - ECG Live Stream

IntelliHQ uses Machine Learning on AWS to create a web-based ECG live stream

The coming 4th Industrial Revolution – an exponential convergence of digital, physical, technological and biological advances – will transform industries over the course of the 21st century. No single innovation will lead this impending revolution, but one thing is clear: Artificial Intelligence (AI) will be at the forefront.

Healthcare is among the foremost industries ripe to be revolutionized by AI and the 4th Industrial Revolution. By enabling ground-breaking advances in healthcare digitisation, AI is expected to significantly contribute to medical advances and lead to marked improvements to healthcare delivery.


Embracing AI to revolutionize healthcare

A not-for-profit partnership between Gold Coast Health, industry and universities, IntelliHQ (Intelligent Health Queensland) is dedicated to promoting research, investment, and monetization of next-generation AI and machine learning technologies in the health sector. It aims to enhance patient outcomes, improve quality of care, create opportunities for skills, jobs and venture development, and encourage investment. By harnessing trusted AI, IntelliHQ aspires to become a globally recognised healthcare innovation and commercialization hub.

Building a global healthcare AI capability promises to deliver significant benefits, not only by relieving pressure on the medical system resulting from spiralling costs, but also by contributing to broader technological economic growth, global competitiveness, and skill creation.

But to realize its aspirations, IntelliHQ has to overcome several challenges and obstacles to AI adoption in healthcare. It needs to build community trust, and maintain secure access to patient data. As such, IntelliHQ needed a technology partner to enable it to achieve its goals.


IntelliHQ engages TechConnect

To enable its key initiatives, IntelliHQ worked with AWS Partner Network (APN) Advanced Consulting Partner, TechConnect to create a web-based ECG live stream with machine learning annotations as a proof-of-concept. This requires numerous cloud technologies for data security, storage, transformation, and means for deployment.

An ECG signal can be categorised into either a normal (healthy) signal or various types of abnormal (unhealthy) classifications, such as atrial fibrillation. Key health indicators are the standard deviation of the length of time between peaks and troughs in an ECG signal, and the ratio between low frequency vs high frequency signals.

It’s possible to generate annotations of these intervals and apply an algorithm to the resulting data to make classifications without machine learning. But given the ability for machine learning to make increasingly sophisticated classifications and predictions, applying these technologies to health data presents many advantages.


The Solution

To extract ICU data to the cloud, IntelliHQ used TechConnect’s Panacea toolset, a C# library and Windows service that connects to a GE Carescape Gateway High Speed Data Interface, subscribe to data feeds and push these data feeds to the cloud via AWS Kinesis Firehose, Amazon’s high speed data ingestion service.

As actual ICU data could not be streamed from a hospital to the cloud until all custodianship processes has been finalised, this proof-of-concept utilised a demonstration monitor to simulate a typical healthy heartbeat, several abnormal rhythms, and test with various combinations of leads connected.

Once subscribed to a data feed, the Windows service inspects each packet to identify source information, and then converts the ECG waveform and extra numeric data into a hexadecimal representation for transport to the cloud. The source information is parsed by an AWS Lambda function, which then writes the data into an Amazon S3 folder structure that uses folder names for a simple organisational scheme.

Parameters are extracted from the AWS Kinesis Firehose stream data by the Amazon Lambda function in a structure format that reflects the default naming convention for Hive partitions on Amazon S3. Many AWS and Hadoop compatible tools (like Amazon Athena) can use this partitioning scheme to efficiently select subsets of data when queried by these parameters.

Once the data was uploaded to Amazon S3 via Panacea in its proprietary raw hex format, they required conversion to a broadly supported file format using an Amazon EMR cluster to handle the parallel transform operations, for which Zeppelin was chosen. It exposes a web-accessible notebook interface to run code and display resulting visualisations, and each notebook consists of a sequence of cells which can utilise a different parser. This provided easy access to Spark, SQL and Scala for data analytics.

IntelliHQ then unpackaged the raw waveform and numerics data into a human readable tabular file format using a combination of PySpark and SparkSQL code. They chose the widely supported csv format, and created annotation files that labelled normal vs abnormal waveform periods. The next data preparation steps took place in Amazon Sagemaker, which gave access to serverless resources for training and deployment. After the addition of front end graphics, the resulting visualization displays in a simple UI. The code is then packaged with Amazon Elastic Container Service for deployment.

ECG Live Stream - IntelliHQ - AWS


The Benefits

The proof-of-concept process results in significant query time improvements and, for tools like Amazon Athena, substantial query cost savings. Furthermore, the folder structure is self documenting and unambiguous, allowing a highly decoupled architecture that can handle any future growth in data volume.

Also the data is non-public, and AWS Identity and Access Management (IAM) credentials establish who has access rights. Security tags and security groups enable IntelliHQ to allocate who has access to data at different stages of preparation. Because these cloud services come with baked-in security, IntelliHQ can ensure they have strong control over health data.


The Outcome

The web-based ECG live stream proof-of-concept showed that efficient, secure web services enable IntelliHQ to build effective, scalable cloud solutions. By embracing AI and machine learning, IntelliHQ will continue to advance the possibilities of commercialized healthcare innovation.


Download white paper here