PyTorch vs Keras

Over the last five years, the uptake and industry interest in two of the libraries in our recent machine learning frameworks overview suggests that they merit a more detailed comparison to each other: Facebook's PyTorch and the French-originated open-source library Keras, now backed by Google¹.

Although there is a great deal of ongoing absorption and consolidation in the machine learning research space, with frameworks rising, falling, merging and being usurped, the PyTorch vs Keras comparison is an interesting study for AI developers, in that it in fact represents the growing contention between TensorFlow and PyTorch — the former, with greater industry support in terms of links to manufacturing and deployment of both abstract systems and single-purpose products (such as CUDA-based GPU acceleration in the industrial and commercial spheres); the latter, more self-integrated, arguably easier to develop with, and to a certain extent better documented and less encumbered by technical debt.

Nonetheless, here we will consider the Keras API rather than TensorFlow itself, since Keras has evolved in recent years from its early ideals as a multi-backend API² into a committed stance as the 'accessible face' of TensorFlow, whereas PyTorch was conceived to provide both low-level computational resources and a relatively high level of user accessibility in a single product.

Keras, originated by AI researcher François Chollet¹⁸ in 2015, shortly before he joined Google as a deep learning researcher and engineer, was originally intended as a 'human-friendly' high-level API for complex but powerful machine learning frameworks.

Written in Python and available under an MIT license¹⁹ with Python, R and now JavaScript²⁰ interfaces, Keras supported a variety of machine learning frameworks including the now-discontinued Theano²¹ as well as PlaidML, a cross-platform Tensor compiler noted for opening up machine learning systems to non-NVIDIA GPUs.

However, as of June 2020, the latest version of Keras (2.3.0) has been announced as the last that would support secondary platforms beyond TensorFlow²², with users (including PlaidML users) advised to switch their code to the integrated tf.keras module in TensorFlow.

Keras supports recurrent and convolutional neural networks. It provides a highly abstracted API for the low-level functionality of TensorFlow, with the capability to create six types of core layers: input object, dense layer, activation layer, embedding layer, masking layer, and lambda layer²³. Further functionality is available with TensorFlow add-ons²⁴.

Keras' recent commitment to TensorFlow as of version 2.0 has also enabled resource optimization via mixed precision training on the NVIDIA GPU platform²⁵. In addition, the AutoKeras package redresses some long-term concerns about matching PyTorch's easy prototyping capabilities, as well as offering an open source alternative to Google's commercial AutoML offering.

As of early 2020, Keras claims a user base of 375,000 individual end users²⁶, as well as adoption by major names such as Netflix, Yelp, and Instacart. It further claims the number one position for mentions in scientific papers by large research bodies such as NASA and CERN (as opposed to the 'general' academic papers cited by PyTorch — see above).

Keras has arguably gained parity with PyTorch³² since its integration as the official high-level TensorFlow API, not least because its slightly easier learning curve³³ is no longer undermined by a dependence on static graphs, or some of its former limitations in accessing and exploiting GPU resources (such as poor support for distributed computing over multiple GPUs³⁴), and it can now easily implement automatic differentiation³⁵ and model and layer sub-classing³⁶.

With greater functional similarity between PyTorch and Keras, the superior abstraction of the Keras API is arguably one compelling reason to consider it as the central framework for a machine learning project — as long as one considers that where problems inevitably do emerge, bug fixes and up-to-date help resources can sometimes be harder to obtain than with PyTorch (see below).

However, this applies only if the proposed architecture is reasonably complex: it has also been argued³⁷ that in terms of API nomenclature, conventions and general usability when defining and instantiating models, there is by now little practical difference in user experience between the two frameworks.

When the time comes to debug, the C++ roots of Keras begin to show. Not only is the underlying code more difficult to navigate than in PyTorch, but it can be harder to identify the point in the dependency chain at which the code is causing problems.

While PyTorch can seem more arcane than Keras at first glance, depending on the complexity of the project, there are a number of high-level APIs that can provide a similar quality of abstraction and facility to Keras, where needed. These include fast.ai, Flare, and Ignite.

In contrast to PyTorch, Keras has undergone so many transformations in intent, outlook, and execution since its inception that versions, bug-fixes, and accuracy of available documentation have become interrelated issues over the course of time. Keras' roots in pure machine learning research have left its online resources in a state that isn't necessarily congruent with its subsequent success in commercial deployments or its general industry reach.

Though the project has a well-updated learning wiki⁴⁴, Keras-related documentation can prove lacking in practical solutions for some common problems, or an adequate number of code examples to cope with the popularity of the framework, or the breadth and scope of developers' issues.

Due to the specious nature of the average machine learning project and the need for its developers to maintain 'real-time' group hubs to address new issues and respond to bug reports, some of the best Keras developer information is found in discrete communities on Discord or other gated channels, hidden from search indexes, obscured by 'infinite scroll' chat interfaces, and missing from the static documentation that the developers do not have the time or inclination to write. This is a vicious circle, since more accessible 'fixed' resources would reduce the need for such cloisters.

At the time of writing, there are nearly 31,000 Keras-related questions on Stack Overflow⁴⁵, as opposed to less than 9,000 for PyTorch⁴⁶. Perhaps it Is for the individual to decide if this reflects on the popularity of Keras, or else on the fragmented nature of its support systems; or, perhaps, on the superior online resources for PyTorch developers.

Additionally there are over 3,000 open issues at the Keras GitHub at the time of writing⁴⁷. Though there are more than 5,000 unclosed issues at the PyTorch GitHub right now⁴⁸, that repository has closed over 11,000 issues to date, versus the 7,000 closed at the Keras project. The increased concentration of developer resources at PyTorch means a reduced number of long-term bugs compared to the slower productivity of the smaller Keras developer group.

Though PyTorch maintains an even more exhaustive instructional wiki⁴⁹ than Keras, it must be considered that PyTorch is also a lower-level and less-abstracted API, with a greater level of explicit complexity for the coder and a greater need for a higher volume of documentation.

One could argue that there is an inverse relationship between reach and documentation between the two frameworks: though the insurgent PyTorch is the relative newcomer for those who want to 'move fast and break things', its Facebook-led communities and investment in outreach means that PyTorch is generally much better (and usefully) documented than Keras*.

The fragmented history of Keras has even led its creator to archive the documentation of previous versions for those users who are currently committed to older iterations of the framework⁵⁰.

In terms of abstract internet wisdom, PyTorch is less hindered by the outdated 'authority' posts that can plague the Keras initiate, since PyTorch's development has been more consistent since its inception.

PyTorch maintains an active and helpful user forum⁵¹, while Keras defers to the Stack Overflow community.

*It should be noted that while PyTorch's documentation and help resources are currently in advance of those of Keras, it is a relative comparison — the breadth, accessibility and extent of PyTorch's resources still invite frequent complaints among end users⁵².