Let OpenAPI generate your Android network layer by leveraging Retrofit, Moshi, and Coroutines

The Ten Commandments (1923)

TLDR: By documenting the API with OpenAPI, we could automatically generate all the Moshi DTOs and the Retrofit Interfaces.

Where we were

Five years ago, the codebase of the Android Cookpad app did not even have different models to isolate the data mapping between layers: only massive objects casting their shadows through the entire stack.

When the resources allowed us to invest some time to improve this situation, we started by creating the DTOs that mapped the JSON responses from the server. With that, we set a clear separation between data boundaries and adjusted our representation to the actual data from the server. Nevertheless, the way we saw the data as clients was quite different from the view that the server had, leading to inefficiencies resulting from unreconciled views, namely indiscriminate nullability and abusive reusability of models that set the path to ambiguous abstractions.

Where we wanted to be: OpenAPI as a way to establish an SSOT for the server and the clients

A few months back, we decided to create clear contracts between the server and the clients as a cross-platform team effort. Both iOS and Android clients had declared nullability in their models as a defensive way to protect themselves from the lack of proper documentation of the consumed API. This capability to encode nullability as a semantic expression rather than a pathological fear, along with having a convenient way to document the API calls, was the main reason to finally decide to adopt a common tongue for all of us.

Henceforth, we decided to document the Cookpad API in a dedicated repository as an SSOT for Web, iOS, and Android engineers. Our tool of choice was the OpenAPI-Specification, as it has a community supporting it, and it seemed more mature.

OpenAPI helped us to document the API but also to redesign parts of it.

OpenAPI works by writing schemas, either on JSON or YAML files, that capture the structure of the data and the HTTP calls that expose them. Putting these schemas in place is meticulous work that requires the collaboration between the clients and the server, overlapping all those visions into one perspective that resembles some truth. Representative members of Web, iOS, and Android had extensive and productive discussions about how to document some specific pieces of data: should this attribute be marked as nullable? Should this endpoint be a PUT method instead of a PATCH? Should this model be reused and nested inside this other? Should this endpoint return an empty response?

The API started to make sense to us as we made collaborative improvements to shape the data structures and the endpoints used to expose them. We also discovered that a few endpoints were highly complex, subjecting the data structures of their responses to the supplied query parameters. This kind of endpoint, a harmful practice that can lead to obscure bugs, is almost impossible to document with the OpenAPI semantic constraints. The price you pay is adding lots of nullable properties to capture the plethora of data structures that endpoints can return.

The complexity boils down to reusing the same endpoint to fulfill different purposes, leading inevitably to the other painful practice when designing endpoints: polymorphism. This practice, which is helpful in other contexts, reveals itself here as a problem almost impossible to solve with the tools that OpenAPI offers. To make it compliant with the openapi-generator, and to be able to generate compilable sources, you will need to twist the OpenAPI semantics and add Regex-based Gradle tasks at the client-side to fine-tune the generated Retrofit endpoints and Moshi DTOs.

A common workflow for the server and the clients.

The way all engineers collaborate is through a common Github repository where we host all the OpenAPI schemas: when a feature needs to consume a new endpoint, or when a new property is deleted/updated, we change the docs accordingly to reflect this change, and the openapi-generator tool will re-generate at client-side the network layer with the latest changes. This workflow allows us to discuss the details of some specific endpoint beforehand, helping us shape a better interface for the clients considering the server specifications.

This practice encourages us to think more carefully about what changes could break the source compatibility and consider if old clients consuming previous API versions would support those new changes. We discover that we were pretty lenient in this regard and that more restrictive means were required to ensure backwards-compatible changes. We set a clear few rules about what we considered a breaking change and refined them based on an iterative process built of multiple situations. Once we identify a new change as a breaking change, it goes to the branch that represents the next API version. After that, clients need to prepare their codebase to accommodate those changes, starting a migration process that ensures the safety of the new API version adoption.

Each PR submitted to our OpenAPI repository requires passing multiple checks that guarantee compatibility with the existing serializers of the back-end implementation and the source compatibility for clients. These checks at both server-side and client-side make it easy to expose early the full impact of those changes by having a global picture of the new state of the API. Of course, these checks are still an ongoing effort that requires us to refine each new iteration with the code that brings us a unique situation. But the current state is mature enough to provide an effective workflow that seamlessly integrates the multiple collaborations between platforms.

The Android goodies: generating the whole network layer using the documented schemas and OpenAPI.

As if having documented all the data structures and endpoints of the API properly wasn’t enough reward, OpenAPI can create your entire network layer automatically as a bonus, including the DTOs and Retrofit interfaces. OpenAPI also offers various options for serialization (Moshi, Gson, or Jackson) and asynchronous code execution (RxJava and Coroutines), probably covering most of your needs. This mechanism saves us the cost of manually creating the DTOs and endpoints and provides an automated way of upgrading the whole network layer to new technology. For example, to migrate from RxJava to Coroutines, you only need to re-run the Gradle task with the appropriate settings changed, and a brand new network layer will pop up in front of your eyes.

Indeed, a Gradle Plugin built on top of the openapi-generator makes the client code generation from the defined schemas a breeze. Running the specific Gradle task supplying a proper output config makes it possible to generate the source for iOS and Android clients fully adjusted to your current implementation. However, on Android, we needed to create an extra Gradle task that we execute on top of the generated sources to fine-tune them. For instance, documenting the headers in the OpenAPI schemas will add each one of them as a parameter to each Retrofit method-endpoint created by the openapi-generator. Since we already had OkHttp interceptors to supply them, we needed to make a Gradle task relying on Regex to remove all those arguments and avoid duplication.

We also noticed that not all the code created by the openapi-generator was helpful, as it was also generating the code required code to consume the HTTP calls and sometimes malformed DTOs that made our builds break. We also addressed that by using Gradle to cut the edges and make the generated code work for our needs. This decoupling between Moshi adapters, DTOs, Retrofit endpoint definitions, and the infrastructure code was very handy, allowing us to easily cherry-pick what we need to keep and discard the rest.

A happy note

OpenAPI made our API docs up to date and fully accessible for all the parties involved. It also forced us to rethink and redesign suboptimal/complex endpoints and data structures and create a contract set in stone that needs multiple hands to engrave new changes. We also used the very same stone to erect the Android network layer from our app, automatically generating the Retrofit interfaces and DTOs.

And last but not least, this collaborative effort between Web, iOS, and Android allowed us to reconcile our points of view about the API and know each other better as linked platforms, potentially making the following collaborations even smoother.

A less happy note

While rewarding in the end, adopting OpenAPI didn’t come for free. If you are planning to commit to this, here are some notes that could serve as warnings signs or shortcuts to make things smoother:

• Some complex endpoints, such as login/sign-up, or any crucial feature of your app for that matter, require to be carefully examined before being documented, and all the underlying complexity to be exposed “at compile time” to avoid devastating pitfalls “at runtime”. The role of QA here is essential.
• Try to avoid polymorphic endpoints as much as possible. If that is not possible, consider using some automation (e.g. post-Gradle tasks) to make the code generated compilable and valuable for your context.
• Rewriting the mappers to adjust the existing DTOs to the newly generated ones is where the bulk of the work takes place. This “chore” is the most challenging part and requires intense focus on the data structures to avoid missing mapping properties that are encoded slightly differently.
• Starting from the easiest endpoints will speed up the learning curve, prepare you for more demanding endpoints whose complexity could overwhelm you, and address minor issues early.
• DTOs duplication: the old manually created one and the new shiny DTO automatically generated will need to coexist during the migration period. If you want to avoid missing recent changes in any of them it’s paramount to have a proper channel to make the whole Android team aware of any change to the data structures as long as the migration phase is ongoing.
• Whoever is going to tackle the migration requires a total commitment during several months. As an example, the Cookpad Android app took around three months to migrate its 143 endpoints.