Currently on a spree of open-sourcing new developer technology, LinkedIn this week announced it is open sourcing {py}gradl—a set of Gradle plug-ins the company created for building Python libraries and applications.
Gradle is a build automation tool that supports many programming languages. It coordinates the process of building software between multiple different code repositories and automates a number of important related tasks, like checking dependencies and warning programmers if something they have changed might break code written by someone else, explained Stephen Holsapple, a LinkedIn senior staff software engineer, in a blog post.
Written in Java and Groovy, Gradle provides strong support for Java Virtual Machine (JVM) languages and is the build system of choice for a variety of projects. Gradle is the build tool for Android Studio. Also, the Eclipse IDE provides first-class Gradle support via its Buildship plug-ins, which enable developers to set up, configure, and initiate Gradle builds from Eclipse.
Holsapple said Gradle core to LinkedIn’s multiproduct development strategy where the company employs a host of different languages to develop software.
“At LinkedIn, we have an extreme polyglot programming environment, supporting more than 42 programming languages,” he said. “One of the most important languages we wanted to integrate with Gradle was Python, the language that we use to build the vast majority of our internal developer tools.”
Holsapple noted that LinkedIn needed to get around the limitations of Python’s Setuptools package management system. For self-contained Python applications with a small set of external dependencies, Setuptools worked just fine, he said. But it could not hold up to the demands of LinkedIn’s fast-growing Python code base.
However, LinkedIn did not want to abandon Setuptools, but rather to extend it. That’s where Gradle came in. Holsapple’s teams enhanced the Python package management system with a set of Gradle plug-ins to address the Setuptools shortcomings.
“With a small amount of effort, we were able to wire in the Python package management ecosystem with LinkedIn’s rich Gradle build system, which we were already using with other languages to achieve continuous integration,” Holsapple said.
According to Holsapple, there were three primary areas where Setuptools came up short for LinkedIn: dependency management, interfacing with existing metadata systems and handling polyglot builds. Polyglot build are build projects where multiple programming languages are used. “Such as when we have native C/C++ extension alongside our Python code, or a frontend written in Ember.js, or a build system that has dependencies on non-Python tooling,” Holsapple said.
“As the size of our Python ecosystem grew, dependency management was the single most painful task that we dealt with on a day-to-day basis,” he said. However, “In using Gradle, we have adopted a powerful dependency and conflict resolution system, started emitting metadata about our dependencies, and begun powering entire suites of new tools that make our Python engineers more productive,” Holsapple noted.
Meanwhile, for polyglot programming, Gradle helps by having a plug-in architecture. For each language or technology stack, LinkedIn simply needs to apply the build plug-in for the underlying language or technology stack, the company said.
“Plug-ins can be developed that mix and match other plug-ins: to build a Python project that uses a native extension, one need only apply the C/C++ and Python plug-ins,” Holsapple said. “Afterwards, a first-class way of declaring a dependency between these stacks is possible, since Gradle uses a language-agnostic format to declare dependency information.”
Moreover, for the sake of familiarity for developers, LinkedIn ensured that a {py}gradle project looks nearly identical to a Python project that uses Setuptools.
Though just released to the community this week, LinkedIn’s {py}gradle code has been in production use for over a year, Holsapple said.
“We hope that using Gradle to build your Python code will be as simple as using Setuptools directly, but will provide you with more robust dependency management and a quality method to integrate against your existing metadata systems, in addition to allowing you to explore building products composed of multiple languages,” he noted.