Tech Talk Series: Solving the Challenges of developing a hybrid cloud integration platform

Tech Talk Series: Solving the Challenges of developing a hybrid cloud integration platform

At Jitterbit our mission is to simplify even the most complex connectivity challenges and to allow anyone to get connected in today’s digital world.   While we stress that you don’t need to be a developer to use Jitterbit’s products – behind every great software solution is a team of fantastic developers.   In the Jitterbit Tech Talk blog series, members of the Jitterbit development team give us insight to building an enterprise-class cloud platform and the challenges they had to solve around multi-tenancy, scalability, and security.

This week, Pankaj Arora, our Sr. Director of Technology looks at securely connecting on-premise and cloud apps and the unique challenges that hybrid IT infrastructures present when developing an enterprise-class cloud platform.

In today’s increasingly inter-connected world, connectivity needs have expanded.  Social media has connected people in new ways, the mobile era has ushered in “always on” connectivity to the digital world, and enterprises are adopting a wide range of disparate “best of breed” apps that need to be connected.  Whether on premise or in the cloud, connectivity makes business processes real-time, synchronizes data from different systems for faster decision-making, and ensures that anyone can access business critical information from anywhere.

Cloud-based integration services deployed on a multi-tenant, elastic cloud infrastructure takes the burden of end users by removing the need to depend on IT to deploy or manage software in house.   With that also comes the expectation that the cloud platform is reliable and always available.

This means that developing a cloud platform requires a plan to maintain uptime, elastic scalability, performance monitoring, flexible deployment options and more.  Solving the challenges of integration on a cloud platform presents an additional challenge:  very few companies have moved 100% to the cloud, which means in most cases there are systems to be connected that reside in on-premise networks.  This in reality meant developing a hybrid cloud integration platform that provided an architecture that used cloud and on-premise run-time agents to securely connect, transform, and move data no matter the location of the app or data source.

Simplifying this setup so that a non-technical user could easily design, deploy and manage these agents presented a number of architectural challenges.   In our initial planning stages we identified a number of requirements for building a robust and flexible hybrid architecture that would use the same guaranteed-delivery messaging layer and highly available, load-balanced (clustered) agents whether they were in the cloud or on-premise:

Location-independent Agent Nodes

An agent node providing connectivity should be able to run on any network. Since agent nodes will run in the customer’s network (on-premise) and Harmony platform (cloud), it should not matter where they are installed. Agents must only adhere to the instructions provided by the Harmony platform about what and when to run.

Self-dependent Agent Nodes

To provide true high-availability and load-balancing, agent nodes should be agnostic to the number of agent nodes joining or leaving a cluster of grouped agent nodes. Inter-dependence between agent nodes could negatively influence their lifecycle (e.g. if a single instance goes down the other dependent agents may stop responding and all integration processes may stop)

Compliance with Enterprise Security Guidelines

Agent nodes in any cluster should be designed to not only run in our Harmony platform but in our enterprise customers’ network with strict network policies. Enterprises don’t want to insecurely expose their internal systems to the world, so we must ensure that our agent nodes use only secure outbound connectivity with proxy support.

These challenges led us to build a messaging layer that would be used by both cloud and on-premise agent nodes to communicate with Harmony platform.  This layer does not need any separate logic to understand agent nodes deployed in different places, it treats them all the same and the agents all communicate with the messaging layer in the same way.

Agent nodes can work in a cluster called ‘agent groups’ and can be added or removed based on the load to be handled. This provides a true load-balanced environment with high availability being provided by adding more than one agent node in the cluster. Agents do not require any knowledge about the other agents in the cluster and can process messages to run integration processes independently. Addition or removal of another agent node does not change working of existing agent node/s in the cluster. The result is agents that are independent yet centrally managed. Harmony and messaging layer have in-built intelligence about clusters, agent node status and work-load routing as needed.

Clusters are available in the Harmony cloud platform or can be installed on-premise, on a virtual machine, or in a customer’s private cloud. Agents connect to the platform, process the instructions and behave in exact same manner independent of their location. The communication protocol that we use is always a secure outbound connection with support for strict proxy rules.

When we released the first version of Harmony a very large number of customers adopted the new platform.  Our free Salesforce Cloud Data Loader and enterprise integration solution run on the same platform, so right out of the gate we had tens of thousands of users, all running thousands to millions of integration processes at once.  And every deployment had some combination of all cloud, all on-premise and hybrid deployments.  Not only did we need to develop location-independent and self-dependent agents, but we had to horizontally scale our system to support large volumes right away.

This was a good problem to have as it forced us to think about massive scale and performance from Day 1.   It led us to expand our capabilities by adding a secure connection layer using the same technology embraced by Google and Facebook to open communication between a browser and a server. We customized this connection layer to enable secure outbound connections from agent nodes to Harmony.

Next time, we’ll talk about how we scaled our secure connection layer running on top of our messaging layer to support thousands of organizations moving millions of records.  In the mean time, try Harmony free for 30-days!