Posts in Serverless

AWS Data Analytics Design HOW To's Serverless Tutorials

Manually Migrate Data Between Redshift Clusters

You have been presented with a few pain points to solve around your company’s Redshift solution. The original Redshift cluster that was launched for the company’s analytics stack has become underpowered over time. Several groups wish to create incremental backups of certain tables to S3 in a format that can be plugged into data lake solutions, as well as other groups wishing to have select pieces of the main Redshift schema splintered to new department-specific clusters.

You’ve come up with a plan to utilize the UNLOAD and COPY commands to facilitate all of the above and need to test a proof of concept to ensure that all pain points above can be addressed in this manner.

Read More
Containers Kubernetes Serverless Technology

Containers or serverless? Consider this..

Every day in our technology world, we see a never-ending battle for the technology platform of choice. One among them is serverless vs. container that looks to be gaining steam. Both of them have their advantages, disadvantages, and proponents. Google, Cisco, and IBM are pushing the container approach, whereas Serverless is being pushed heavily by major public cloud providers like AWS, Azure & GCP (all these cloud operators also offer services for both native containers and Kubernetes).

As we saw in the growth of cloud, and now containers, this migration isn’t always seamless. In fact, we’ve seen both Microsoft (Azure Stack) and Amazon (Outposts) introduce products to work within our existing data center. Migrations haven’t returned the value that was promised because cost, security, regulatory needs, and complexity haven’t been as seamless as we all once imagined. The move to containers and serverless architectures will face some of the same dilemmas as cloud, but this is setting itself up to be 2019’s enterprise.

Unlike before, today’s industries wish to move to the latest and greatest technology quickly, usually because of significant advantages in price, time and simplicity once deployed. And, it’s simply more fun as an engineer to play and learn as the tech grows itself.

Now let’s examine some of the differences between the two latest and greatest:

New Apps:

Containers are useful when the application require multiple instances running parallelly. The hard parameters on the execution time also limit long-running and complicated processes from finishing, making large data crunching particularly challenging, whereas with a container the spawning of new containers to split workload is simpler and handled automatically.

Serverless, on the other hand lends itself to exciting possibilities for new applications, especially in the IoT, physical security and world of chatbots. The ability to use triggers to kick off various actions (without having any underlying infrastructure) and grow at scale allows for both a cost-effective and simplified management. Applications that are listening for triggers can execute code in an IoT environment as it changes, reducing the costs, and simplifying the software management for a company that may not have a giant DevOps staff.

Native Cloud Apps:

Migrating existing native cloud applications is basically a question about the nature of the application itself.  If you’ve purpose-built your applications with a singular public cloud vendor, then integrating and even migrating applications to serverless becomes very easy. All the major serverless platforms have built-in hooks to other native cloud services and allow for quick and seamless architecture changes. Specifically, applications that relied upon orchestration tools for quick scale up and down may find the simplicity that serverless provides to be very advantageous.

The argument for containers is like that of serverless: gain functionality from existing cloud services and improve orchestrated scaling. But the difference is in the maturity of containers versus serverless.  Container testing is much more advanced, and the number of people with the required skill set to efficiently architect the solution is far greater than with serverless. But most applications today are hybrid or multi-cloud/multi-architecture, and the differences here vary even greater.

Legacy Hybrid/On-Prem Apps:

The last ten years have shown the world the complexity with lift and shift, and the high costs associated with such a cloud strategy. Instead, we’ve seen the rise of Hybrid applications that run on multiple different clouds and may work with the existing on-premise solution providing some functionality.  Serverless can provide a faster path to the cloud; however, only if the time to architect the solution correctly has been applied. Legacy applications that perform on-demand quick checks and tasks can easily be ported towards a serverless solution, thus reducing the complexity of managing a larger, more complicated code base and physical footprint – and several these legacy mainframe tasks have already been ported successfully into serverless apps.

Containers offer a more straightforward fashion to migrating your legacy technologies, with far less architectural or vendor dependence.  The single biggest long-term hurdle for serverless growth is the vendor lock-in to one of the big 3 (Amazon, Microsoft & Google), as the technologies are not yet truly portable. Contrast that with Kubernetes running on top of various clouds to reduce the need to use a particular cloud vendor and adoption by companies like Cisco and IBM as part of their overall cloud strategy.  By contrast, the serverless approach is largely platform dependent, and thus being pushed heavily by the incumbent IaaS providers to keep you on their platform.

Summary:

In the short- term, it seems that serverless is still in its early stages and best suited to purpose-built applications and in specific domains. But it has fabulous upside and value as an alternative approach to handling large-scale complexity and heavy architecture costs. The recent announcement by Amazon at AWS hints they see a similar set of needs to complement Lambda and other serverless technologies in the future and the keen interest in serverless will only sprout up solutions to these problems over time.