Tuesday, 15 August, 2017 UTC


Summary

Recently, there’s been some buzz around the next great architectural shift in systems. There is a rising interest in the evolution of decentralized edge computing as a core part of that shift.
For over two years, npm has been using edge computing concepts to ensure that the developer experience for users of npm Enterprise, our private registry product, matches the experience of using the centralized, cloud-hosted version of the npm Registry.
Here’s why we’re doing that, and how:
The opportunity
Many enterprises have strict requirements that prevent them from using cloud-hosted products for critical parts of their infrastructure. This approach makes sense from a regulatory compliance perspective, but it makes life inconvenient for developers within those companies who wish to take advantage of open-source code from the npm Registry, or who wish to use npm to share and reuse their own code with their colleagues.
npm Enterprise allows developers at big companies to run a version of the npm Registry behind their firewall. Of course, it wouldn’t be enough for enterprise customers to simply deploy a fresh install of npm Enterprise with an empty registry. Much of npm’s value comes from the 500,000 packages available on the public registry, and being able to combine these packages with private code. Without access to these packages, developers would waste time reinventing a lot of wheels.
npm Enterprise lets companies mix public packages from the public registry with private code stored within their private registry without risks or complexity.
Enter edge node architecture
We designed npmE so that each npmE server is a private edge node to the public registry. Each npmE instance can replicate select parts or all of the npm Registry to offer this functionality to end users. It also provides additional local services that are only accessible to these users, based on a company’s unique requirements.
Our customers are able to configure their private registry as a full mirror of the public Registry to decrease latency, cut bandwidth costs, and offer npm Registry to end users who are restricted from accessing the public internet. Alternately, they may selectively mirror npm’s Registry using specific whitelists managed by the admin console or the npm command line.
When combined with npmE add-ons which enforce code quality, evaluate how packages and their dependencies are licensed, and scan for security vulnerabilities, this architecture gives companies total control of the public packages their developers may use.
At the same time, these developers may find, share, and re-use proprietary code by publishing to their private local registry. Private code stays private by never leaving the company’s own infrastructure.
End users don’t have to think about where each package is located; they can just pull from the npm Enterprise server. Behind the scenes, the server automatically determines the scope and proxy of the package pull.
But how?
There are two primary challenges of an edge node architecture:
  1. Enabling each enterprise IT admin to independently install, configure, deploy, manage, update and integrate their instance into their infrastructure.
  2. Maintaining the agility of the cloud hosted edition without getting bogged down with enterprise requirements.
For many years, deploying and maintaining a private instance of this kind of architecture would have been prohibitively difficult for everyone but the most advanced IT organizations. These sorts of enterprise software installations took several months to implement and involved manual processes of configuring servers, runtimes, and components. Every enterprise IT org would have taken responsibility for the ops role of their enterprise instance.
Fortunately, it’s now much easier to deploy and maintain private edge nodes thanks to technologies like containerization, orchestration and scheduling platforms.
Deployment and management are now baked into the design and development effort of modern applications. Generally, this creates reproducible and consistent cloud-native deployments, but it also is becoming the foundation of modern enterprise software deployments. This automation and inherent portability allows our customers to deploy into their own environments without deep knowledge of our architecture.
Of course, it isn’t quite as easy and magical as it all sounds. We initially built out our own containerized installation methods by packing all of our services into a single container. This approach still required npm Enterprise customers to be quite technical to complete the deployment. The system also lacked the tooling for managing versions, customers, backups, and updates.
Enter Replicated
After a few months of banging our heads against the wall, we decided that we were dedicating too many resources to deploying and managing of Enterprise instances. We switched over to a platform called Replicated to be our enterprise version management platform. Replicated’s platform provides workflows for integrating our existing CI/CD release pipeline with our enterprise release channels.
Similar to the way that npm packages are versioned for automatic updates, we use discrete versioned images for each of the services that make up npm Enterprise. We organize these into specific channels provided — “stable,” “beta”, and “unstable” — and when we promote a set of images to “stable,” Replicated automatically notifies our customers that an update to npm Enterprise is available and makes the update as simple as a single click. Our customers don’t have to manually update services, and we don’t have to manually push containers around in order to keep the edge nodes of the npm Registry on recent versions.
Beyond deployment and management, there also was the problem of developing enterprise-specific features such as change management processes, LDAP integration and an admin dashboard, which enterprise customers require but which fall outside our core product expertise. Many of these features are included (or at least made easier) in the Replicated platform and provide a consistent experience that enterprise IT admins are now familiar with.
These sorts of enterprise ready features are important to our Enterprise customers, but since they aren’t a core part of our value proposition, it has made a ton of sense for us to leverage a partner to power these as much as possible.
What’s next
The state of art in edge node architecture is still evolving, but it is gaining more traction in a variety of use cases. An increasing quantity of JS developers rely on npm, and as a result, an increasing number of enterprises will need npm Enterprise. For developers to be effective it’s imperative that they benefit from the global Registry of npm packages.
By partnering with Replicated to pioneer an architecture that delivers on that promise while reducing management overhead and satisfying security requirements, we can see an emerging future that embraces the distributed nature of the internet. To learn more about Replicated’s technology, visit their site.