RSS

GraphQL API News

These are the news items I've curated in my monitoring of the API space that have some relevance to the API deployment conversation and I wanted to include in my research. I'm using all of these links to better understand how the space is acquisitions their APIs, going beyond just monitoring and understand the details of each request and response.

GraphQL And REST Differences Explained With Burgers

GraphQL folks keep on with the GraphQL vs REST narratives, rather than a REST and / or GraphQL narrative lately with a recent burger meme/narrative. Continuing to demonstrate their narrow view of the landscape, and revealing the short lived power of an adversarial approach to community building. I get why people do this, because they feel they are being clever, and that the click response from the echo chamber re-enforces it, but ultimately it is something that won’t move the conversation forward, but it does get them kudos within their community–which is what many of them live for.

I’ll start with the usual disclaimer. I actually like GraphQL, and prescribe it as part of my API toolbox. However, rather than a REST vs GraphQL approach, I sell it as REST and GraphQL, depending on the developer audience we are trying to reach with our efforts. Whether or not you use GraphQL on your platform is completely based upon knowing your developers, and working with a group that understands the resources before offered–something the GraphQL community continues their failure to see. Also their adversarial marketing tactics has lost me several GraphQL projects in government because it comes off as being a trend, and not something that will be around very long.

With that said, I think this meme tells a great story about GraphQL, and demonstrates the illnesses of not the technology, but the ideology and beliefs of the community. I had a couple of thoughts after seeing the Tweet, and reviewing the replies:

1) I thought it was an anti-GraphQL meme at first. Demonstrating that you can build a horrible burger with some very well known ingredients. Spoofing on the burger emoji drama that has been going on in recent years.. I mean, is the lettuce the plate in the GraphQL burger?

2) Like GraphQL, the food choices demonstrate that GraphQL works well in very controlled environments. Where there are known ingredients, and your clients/customers/developers know the ingredients, and know what they want. Hell yeah GraphQL is a better choice in this environment. The problem is you are selling it as a a better solution than REST in general. I hate to tell you, but most of the business getting done in the world IS NOT FAST FOOD.

3) The meme demonstrates the whole fast food, limited world view of many technologists who work with known ingredients, and think everyone is just like them. This tool works for me, and everyone is just like me, so what I use is cool, and everyone should use the tools and the process that I do. A common perspective out of the white bread (bun?) world of technology.

4) Let’s take this GraphQL meme and begin applying it to an Ethiopian, Greek, or French menu. Let’s take it and apply to a BBQ, catering, or maybe home cooked family gathering. Try applying it when you get a food basket from your local community supported agriculture (CSA), where you have no idea the ingredients that are coming, and you’ll have to adjust based upon the season and whatever is available to you that week. Maybe do the same for a food shelter and pantry, does everyone get it their way?

5) There are some restaurants in New York I’d love to take you to, and have you ask for it your way. I’d love to see you get yelled out of the place when you think you know more than the chef, and you always should have things your way. Really, you know more than someone who has been cooking for years, and your fast food loving, unsophisticated tastes are going to dictate what gets served? Get outta here!!

6) I love API to restaurant menu analogies. I wrote one to support the Oracle v Google copyright case, which the Google lawyer referenced in the latest round. There are many ways you can use restaurants and food to make API comparisons, and educate people about the potential of APIs. I’m sorry though, this one just wasn’t sophisticated enough to really bring home the potential of APIs, and it was more about reflecting this same unsophisticated approach of people marketing and telling stories around GraphQL.

I’ll say it again, and again, and again. I’m not anti-GraphQL. I’m against ya’ll saying it is a replacement for REST. Stop it. It’s dumb. It shows your lack of awareness of the larger API world. It shows you live in tech isolation, where you think everyone wants it your way. Most developers I know do not have a clue as to what they want. They don’t understand the existing schema being used, and need menus, and hand-crafted buffets. Sure, there are development groups who know exactly what they need, and have a full grasp on the schema and resource models being used, but this isn’t EVERYONE!! Stop it. I get GraphQL, but I’m getting tired of coming across new APIs I don’t understand at all, and being expected to just know what I want. I love GraphQL for Github because I KNOW GITHUB. I don’t love GraphQL for the OpenStates API, because I have no clue what the schema and model is for their API–please do the extra work to document your resources, and provide me intelligent, well-crafted paths to get at your valuable data.

Instead of bashing REST, how about thinking more about REST as a starter. Having feedback loops in place to get to your audience. Sure, if it is all internal development, in service of a known group of React developers, go for it–use GraphQL! However, if it is a public API, start with REST, establish feedback loops, and get to know your audience. If enough developers are requesting a query language (GraphQL isn’t the only show in town), and it makes sense in your roadmap, then offer GraphQL alongside REST, but not instead of REST. GraphQL works in a known known, and sometimes a known unknown environment, but not in an unknown unknown environment. The community needs to wake up and realize this. Stop selling it as a replacement to REST, and realize it is just another tool in the API toolbox. Y’all are just hurting your cause, and running some people off with this regular REST v GraphQL storytelling. In the end, you are just showing your lack of knowledge and respect for the web–just like Facebook does.

P.S. Anyone who has their feelings hurt by this post, needs to get out more. Maybe change jobs, move to a new city and industry. You need to see and experience more than you have currently.


GraphQL, Just Get Out Of My Way And Give Me What I Want

One of the arguments I hear for why API providers should be employing GraphQL is that they should just get out of developers way, and let them build their own queries so that they can just get exactly the data that they want. As an application developer we know what we want from your API, do not have us make many different calls, to multiple endpoints–just give us one API and let us ask for exactly what we need. It is an eloquent, logical argument when you operate and live within a “known bubble”, and you know exactly what you want. Now, ask yourself, will every API consumer know what they want? Maybe in some scenarios, every API consumer will know what they want, but in most situations, developers will not have a clue what they want, need, or what an API does.

This is where the GraphQL as a replacement for REST argument begins breaking down. In a narrowly defined bubble, where every developers knows the schema, knows GraphQL, and knows they want–GraphQL can make a lot of sense. In the autodidact alpha developer startup world this argument makes a lot of sense, and gets a lot of traction. However, not everyone lives in this world, and in this real world, API design can become very important. Helping people understand what is possible, learn the schema behind an API, and become more familiar with an API, until they have a better understanding of what they might want. I’m not saying GraphQL doesn’t have a place when you have a significant portion of your audience knowing what they want, I’m just saying that you shouldn’t leave everyone else behind.

To further turn this argument on its head, as a developer, if I know what I want, why make me build a query at all? Just give me a single URL with what I want! Don’t make me do the heavy lifting, and work to craft a query, just give me a single URL with minimal authentication, and let me get what I need in one click. Sure, it will take some time before you can craft enough URLs to meet everyone’s needs, but it will be worth it for those who you can. You can always craft new paths based upon requests, and yes, you can also augment your web APIs with a GraphQL endpoint for those neediest, most demanding developers who love building queries, and know exactly what they want. I guess my point is that there are a lot of definitions of knowing what you want, and how APIs can satisfy that, and not everyone will be in the same mindset as you are.

Now I have to bury in this last paragraph the fact that I’m not anti-GraphQL. I am anti-GraphQL being sold as a replacement for simpler, resource centered web APIs (aka REST). So if you are going to come at me with the Y U HATE GraphQL response–don’t. I’m just trying to show GraphQL believers that they are leaving some people behind with a GraphQL only approach, unless you are 100% sure that ALL your API consumers know GraphQL, know your schema, and now what they want. GraphQL is an important tool in the API toolbox, but it isn’t the one size fits all tool it is often sold as. After much contemplation, and working on the ground within enterprise groups, I am trying to put GraphQL to work in more use cases where it makes sense, and before I can do this I have to push back much of the misinformation that has been peddled about it, and undo the damage I’me seeing on the ground.


My GraphQL Thoughts After Almost Two Years

It has been almost a year and half since I first wrote my post questioning that GraphQL folks didn’t want to do the hard work of API design, which I also clarified that I was keeping my mind open regarding the approach to delivering APIs. I’ve covered several GraphQL implementations since then, as well as my post on waiting the GraphQL assault out-to which I received a stupid amount of, “dude you just don’t get it!”, and “why you gotta be so mean?” responses.

GraphQL Is A Tool In My Toolbox
I’ll start with my official stance on GraphQL. It is a tool in my API toolbox. When evaluating projects, and making recommendations regarding what technology to use, it exists alongside REST, Hypermedia, gRPC, Server-Sent Events (SSE), Websockets, Kafka, and other tools. I’m actively discussing the options with my clients, helping them understand the pros and cons of each approach, and working to help them define their client use cases, and when it makes sense I’m recommending a query language layer like GraphQL. When there are a large number of data points, and a knowledgeable, known group of developers being targeted for building web, mobile, and other client applications, it can make sense. I’m finding GraphQL to be a great option to augment a full stack of web APIs, and in many cases a streaming API, and other event-driven architectural considerations.

GraphQL Does Not Replace REST
Despite what many of the pundits and startups would like to be able to convince everyone of, GraphQL will not replace REST. Sorry, it doesn’t reach as wide of an audience as REST does, and it still keeps APIs in the database, and data literate club. It does make some very complex things simpler, but it also makes some very simple things more complex. I’ve reviewed almost 5 brand new GraphQL APIs lately where the on-boarding time was in the 1-2 hour range, rather than minutes with many other more RESTful APIs. GraphQL augments RESTful approaches, and in some cases it can replace them, but it will not entirely do away with the simplicity of REST. Sorry, I know you don’t want to understand the benefits REST brings to the table, and desperately want there to be a one size fits all solution, but it just isn’t the reality on the ground. Insisting that GraphQL will replace REST does everyone more harm than good, hurts the overall API community, and will impede GraphQL adoption–regardless of what you want to believe.

The GraphQL Should Embrace The Web
One of my biggest concerns around the future of GraphQL is its lack of a focus on the web. If you want to see the adoption you envision, it has to be bigger than Facebook, Apollo, and the handful of cornerstone implementations like Github, Pinterest, and others. If you want to convert others into being believers, then propose the query language to become part of the web, and improve web caching so that it rises to the level of the promises being made about GraphQL. I know underneath the Facebook umbrella of GraphQL and React it seems like everything revolves around you, but there really is a bigger world out there. There are more mountains to conquer, and Facebook won’t be around forever. To many of the folks I’m trying to educate about GraphQL, they can’t help but see the shadow of Facebook, and their lack of respect for the web. GraphQL may move fast, and break some things, but it won’t have the longevity all y’all believe so passionately in, if you don’t change your tune. I’ve been doing this 30 years now, and seen a lot of technology come and go. My recommendation is to embrace the web, bake GraphQL into what is already happening, and we’ll all be better off for it.

GraphQL Does Not Do What OpenAPI Does
I hear a lot of pushback on my OpenAPI work, telling me that GraphQL does everything that OpenAPI does, and more! No, no it doesn’t. By saying that you are declaring that you don’t have a clue what OpenAPI does, and that you are just pushing a trend, and have very little interest in the viability of the APIs I’m building, or my client’s needs. There is an overlap in what GraphQL delivers, and what OpenAPI does, but they both have their advantages, and honestly OpenAPI has a lot more tooling, adoption, and enjoys a diverse community of support. I know in your bubble that GraphQL and React dominates the landscape, but on my landscape it is just a blip, and there are other tools in the toolbox to consider. Along with the web, the GraphQL should be embracing the OpenAPI community, understanding the overlap, and reaching out to the OAI to help define the GraphQL layer for the specification. Both communities would benefit from this work, rather than trying to replace something you don’t actually understand.

GraphQL Dogma Continues To Get In The Way
I am writing this post because I had another GraphQL believer mess up my chances for it being in the roadmap, because of their overzealous beliefs, and allowing dogma to drive their contribution to the conversation. This is the 3rd time I’ve had this happen on high profile projects, and while the GraphQL conversation hasn’t been ruled out, it has been slowed, pushed back, and taken off the table, due to pushing the topic in unnecessary ways. The conversation unfortunately wasn’t about why GraphQL was a good option, the conversations was dominated by GraphQL being THE SOLUTION, and how it was going to fix everything web service and API that came before it. This combined with inadequate questions regarding IP concerns conveyed, and GraphQL being a “Facebook solution”, set back the conversations. In each of these situations I stepped in to help regulate the conversations, and answer questions, but the damage was already done, and leadership was turned off to the concept of GraphQL because of overzealous, dogmatic beliefs in this relevant technology. Which brings me back to why I’m pushing back on GraphQL, not because I do not get the value it brings, but because the rhetoric around how it is being sold and pushed.

No, I Do Get What GraphQL Does
This post will result in the usual number of Tweets, DMs, and emails telling me I just don’t get GraphQL. I do. I’ve installed and played with it, and hacked against several implementations. I get it. It makes sense. I see why folks feel like it is useful. The database guy in me gets why it is a viable solution. It has a place in the API toolbox. However, the rhetoric around it is preventing me from putting it to use in some relevant projects. You don’t need to convince me of its usefulness. I see it. I’m also not interested in convincing y’all of the merits of REST, Hypermedia, gRPC, or the other tools in the toolbox. I’m interested in applying the tools as part of my work, and the tone around GraphQL over the last year or more hasn’t been helping the cause. So, please don’t tell me I don’t get what GraphQL does, I do. I don’t think y’all get the big picture of the API space, and how to help ensure GraphQL is in it for the long haul, and achieving the big picture adoption y’all envision.

Let’s Tone Down The Rhetoric
I’m writing about this because the GraphQL rhetoric got in my way recently. I’m still waiting out the GraphQL assault. I’m not looking to get on the bandwagon, or argue with the GraphQL cult, I’m just looking to deliver on the projects I’m working on. I don’t need to be schooled on the benefits of GraphQL, and what the future will hold. I’m just looking to get business done, and support the needs of my clients. I know that the whole aggressive trends stuff works in Silicon Valley, and the startup community. But in government, and some of the financial, insurance, and healthcare spaces I’ve been putting GraphQL on the table, the aggressive rhetoric isn’t working. It is working against the adoption of the query language. Honestly it isn’t just the rhetoric, I don’t feel the community is doing enough to satisfy the wider IP, and Facebook connection to help make the sale. Anyways, I’m just looking to vent, and push back on the rhetoric aound GraphQL replacing REST. After a year and a half I’m convinced of the utility of GraphQL, however the wider tooling, and messaging around it still hasn’t won me over.


API Is Not Just REST

This is one of my talks from APIDays Paris 2018. Here is the abstract: The modern API toolbox includes a variety of standards and methodologies, which centers around REST, but also includes Hypermedia, GraphQL, real time streaming, event-driven architecture , and gRPC. API design has pushed beyond just basic access to data, and also can be about querying complex data structures, providing experience rich APIs, real-time data streams with Kafka and other standards, as well as also leveraging the latest algorithms and providing access to machine learning models. The biggest mistake any company, organization, or government agency can do is limit their API toolbox to be just about REST. Learn about a robust and modern API toolbox from the API Evangelist, Kin Lane.

Diverse Toolbox After eight years of evangelizing APIs, when I participate in many API conversations, some people still assume I’m exclusively talking about REST as the API Evangelist–when in reality I am simply talking about APIs that leverage the web. Sure, REST is a dominant design pattern I shine a light on, and has enjoyed a significant amount of the spotlight over the last decade, but in reality on the ground at companies, organizations, institutions, and government agencies of all shapes and sizes, I find a much more robust API toolbox is required to get the job done. REST is just one tool in my robust and diverse toolbox, and I wanted to share with you what I am using in 2018.

The toolbox I’m referring tool isn’t just about what is needed to equip an API architect to build out the perfect vision of the future. This is a toolbox that is equipped to get us from present day into the future, acknowledging all of the technical debt that exists within most organizations which many are looking to evolve as part of their larger digital transformation. My toolbox is increasingly pushing the boundaries of what I’ve historically defined as an API, and I’m hoping that my experiences will also push the boundaries of what you define as an API, making you ready for what you will encounter on the ground within organizations you are delivering APis within.

Application Programming Interface API is an acronym standing for application programming interface. I do not limit the scope of application in the context to just be about web or mobile application. I don’t even limit it to the growing number of device-based applications I’m seeing emerge. For me, application is about applying the digital resources made available via an programmatic interface. I’m looking to take the data, content, media, and algorithms being made available via APIs and apply them anywhere they are needed on the web, within mobile and device applications, or on the desktop, via spreadsheets, digital signage, or anywhere else that is relevant, and sensible in 2018.

API does not mean REST. I’m really unsure how it got this dogmatic association, nor do I care. It is an unproductive legacy of the API sector, and one I’m looking to move beyond. Application programming interfaces aren’t the solution to every digital problem we face. They are about understanding a variety of protocols, messaging formats, and trying to understand the best path forward depending on your application of the digital resources you are making accessible. My API toolbox reflects this view of the API landscape, and is something that has significantly evolved over the last decade of my career, and is something that will continue to evolve, and be defined by what I am seeing on the ground within the companies, organizations, institutions, and government agencies I am working with.

SOAP I have been working with databases since 1987, so I fully experienced the web services evolution of our industry. During the early years of the web, there was a significant amount of investment into thinking about how we exchanged data across many industries, as well as within individual companies when it came to building out the infrastructure to deliver upon this vision. The web was new, but we did the hard work to understand how we could make data interoperability in a machine readable way, with an emphasis on the messages we were exchanging. Looking back I wish we had spent more time thinking about how we were using the web as a transport, as well as the influence of industry and investment interests, but maybe it wasn’t possible as the web was still so new.

While web services provided a good foundation for delivering application programming interfaces, it may have underinvested in its usage of the web as a transport, and became a victim of the commercial success of the web. The need to deliver web applications more efficiently, and a desire to hastily use the low cost web as a transport quickly bastardized and cannibalized web services, into a variety of experiments and approaches that would get the job done with a lot less overhead and friction. Introducing efficiencies along the way, but also fragmenting our enterprise toolbox in a way which we are still putting back together.

XML & JSON RPC One of the more fractious aspects of the web API evolution has been the pushback when API providers call their XML or JSON remote procedure call (RPC) APIs, RESTful, RESTish, or other mixing of philosophy and ideology, which has proven to be a dogma stimulating event. RESTafarians prefer that API providers properly define their approach, while many RPC providers could care less about labels, and are looking to just get the job done. Making XML and JSON RPC a very viable approach to doing APIs, something that still persists almost 20 years later.

Amazon Web Services, Flickr, Slack, and other RPC APIs are doing just fine when it comes to getting the job done, despite the frustration, ranting, and shaming by the RESTafarians. It isn’t an ideal approach to delivering programmatic interfaces using the web, but it reflects its web roots, and gets the job done with low cost web infrastructure. RPC leaves a lot of room for improvement, but is a tool that has to remain in the toolbox. Not because I am designing new RPC APIs, but there is no doubt that at some point I will have to be integrating with an RPC API to do what you need to get done in my regular work.

REST at Center Roy Fielding’s dissertation on representational state transfer, often referred to as simple REST, is an amazing piece of work. It makes a lot of sense, and I feel is one of the most thorough looks at how to use the web for making data, content, media, and algorithms accessible in a machine readable way. I get why so many folks feel it is the RIGHT WAY to do things, and one of the reasons it is the default approach for many API designers and architects–myself included. However, REST is a philosophy, and much like microservices, provides us with a framework to think about how we put our API toolbox to work, but isn’t something that should blind us from the other tools we have within our reach.

REST is where I begin most conversations about APIs, but it doesn’t entirely encompass what I mean when every time I use the phrase API. I feel REST has given me an excellent base for thinking about how I deliver APIs, but will slow my effectiveness if I leave my REST blinders on, and let dogma control the scope of my toolbox. REST has shown me the importance of the web when talking about APIs, and will continue to drive how I deliver APIs for many years. It has shown me how to structure, standardize, and simplify how I do APis, and help my applications reach as wide as possible audience, using commonly understood infrastructure.

Negotiating CSV As the API Evangelist, I work with a lot of government, and business users. One thing I’ve learned working with this group is the power of using comma separated values (CSV) as a media type. I know that us developers and database folks enjoy a lot more structure in our lives, but I have found that allowing for the negotiation of CSV responses from APIs, can move mountains when it comes to helping onboard business users, and decision makers to the potential of APIs–even if the data format doesn’t represent the full potential of an API. CSV responses is the low bar I set for my APIs, making them accessible to a very wide business audience.

CSV as a data format represents an anchor for the lowest common denominator for API access. As a developer, it won’t be the data format I personally will negotiate, but as a business user, it very well could mean the difference between using an API or not. Allowing me to take API responses and work with them in my native environment, the Excel spreadsheet, or Google Sheets environment. As I am designing my APIs, I’m always thinking about how I can make my resources available to the masses, and enabling the negotiation of CSV responses whenever possible, helps me achieve my wier objectives.

Negotiating XML I remember making the transition from XML to JSON in 2009. At first I was uncomfortable with the data format, and resisted using it over my more proven XML. However, I quickly saw the potential for the scrappy format while developing JavaScript applications, and when developing mobile applications. While JSON is my preferred, and default format for API design, I am still using XML on a regular basis while working with legacy APIs, as well as allowing for XML to be negotiated by the APIs I’m developing for wider consumption beyond the startup community. Some developers are just more comfortable using XML over JSON, and who knows, maybe by extending an XML olive branch, I might help developers begin to evolve in how they consume APIs.

Similar to CSV, XML represents support for a wider audience. JSON has definitely shifted the landscape, but there are still many developers out there who haven’t made the shift. Whether we are consumers of their APIs, or providing APIs that target these developers, XML needs to be on the radar. Our toolbox needs to still allow for us to provide, consume, validate, and transform XML. If you aren’t working with XML at all in your job, consider yourself privileged, but also know that you exist within a siloed world of development, and you don’t receive much exposure to many systems that are the backbone of government and business.

Negotiating JSON I think about my career evolution, and the different data formats I’ve used in 30 years. It helps me see JSON as the default reality, not the default solution. It is what is working now, and reflects not just the technology, but also the business and politics of doing APIs in a mobile era, where JavaScript is widely used for delivering responsive solutions via multiple digital channels. JSON speaks to a wide number of developers, but we can’t forget that it is mostly comprised of developers who have entered the sector in the last decade.

JSON is the default media type I use for any API I’m developing today. No matter what my backend data source is. However, it is just one of several data formats I will potentially open up for negotiation. I feel like plain JSON is lazy, and whenever possible I should be thinking about a wider audience by providing CSV and XML representations, but I should also be getting more structured and standardized in how I handle the requests and responses for my API. While I want my APIs to reach as wide as possible audience, I also want them to deliver rich results that best represents the data, content, media, algorithms, and other digital resources I’m serving up.

Hypermedia Media Types Taking the affordances present when humans engage with the web via browsers for granted is one of the most common mistakes I make as an API design, developer, and architect. This is a shortcoming I am regularly trying to make up for by getting more sophisticated in my usage of existing media types, and allowing for consumers to negotiate exactly the content they are looking for, and achieve a heightened experience consuming any API that I deliver. Hypermedia media types provide a wealth of ways to deliver consistent experiences, that help be deliver many of the affordances we expect as we make use of data, content, media, and algorithms via the web.

Using media types like Hal, Siren, JSON API, Collection+JSON, and JSON-LD are allowing me to deliver a much more robust API experience, to a variety of API clients. Hypermedia reflects where I want to be when it comes to API design and architecture that leverages the web, but it is a reflection I have to often think deeply about as I still work to reach out to a wide audience, forcing me to make it one of several types of experience my consumers can negotiate. While I wish everyone saw the benefits, sometimes I need to make sure CSV, XML, and simpler JSON are also on the menu, ensuring I don’t leave anyone behind as I work to bridge where we are with where I’d like to go.

API Query Layers Knowing my API consumers is an important aspect of how I use my API toolbox. Depending on who I’m targeting with my APIs, I will make different decisions regarding the design pattern(s) I put to work. While I prefer investing resources into the design of my APIs, and crafting the URLs, requests, and responses my consumers will receive, in some situations my consumers might also need much more control over crafting the responses they are getting back. This is when I look to existing API query languages like Falcor or GraphQL to give my API consumers more of a voice in what their API responses will look like.

API query layers are never a replacement for a more RESTful, or hypermedia approaches to delivering web APIs, but they can provide a very robust way to hand over control to consumers. API design is important for providers to understand, and define the resources they are making available, but a query language can be very powerful when it comes to making very complex data and content resources available via a single API URL. Of course, as with each tool present in this API toolbox, there are trade offs with deciding to use an API query language, but in some situations it can make the development of clients much more efficient and agile, depending on who your audience is, and the resources you are looking to make available.

Webhooks In my world APIs are rarely a one way street. My APIs don’t just allow API consumers to poll for data, content, and updates. I’m looking to define and respond to events, allowing data, and content to be pushed to consumers. I’m increasingly using Webhooks as a way to help my clients make their APIs a two-way street, and limit the amount of resources it takes to make digital assets available via APIs. Working with them to define the meaningful events that occur across the platform, and allow API consumers to subscribe to these events via Webhooks. Opening the door for API providers to deliver a more event-driven approach to doing APIs.

Webhooks are the 101 level of event-driven API architecture for API providers. It is where you get started trying to understand the meaningful events that are occurring via any platform. Webhooks are how I am helping API providers understand what is possible, but also how I’m training API consumers in a variety of API communities about how they can deliver better experiences with their applications. I see webhooks alongside API design and management, as a way to help API providers and consumers better understand how API resources are being used, developing a wider awareness around which resources actually matter, and which ones do not.

Websub In 2018, I am investing more time in putting Websub, formerly known as the word which none of us could actually pronounce, PubSubHubbub. This approach to making content available by subscription as things change has finally matured into a standard, and reflects the evolution of how we deliver APIs in my opinion. I am using Websub to help me understand not just the event-driven nature of the APIs I’m delivering, but also that intersection of how we make API infrastructure more efficient and precise in doing what it does. Helping us develop meaningful subscriptions to data and content, that adds another dimension to the API design and even query conversation.

Websub represents the many ways we can orchestrate our API implementations using a variety of content types, push and pull mechanisms, all leveraging web as the transport. I’m intrigued by the distributed aspect of API implementations using Websub, and the discovery that is built into the approach. The remaining pieces are pretty standard API stuff using GETs, POSTs, and content negotiation to get the job done. While not an approach I will be using by default, for specific use cases, delivering data and content to known consumers, I am beginning to put Websub to work alongside API query languages, and other event-driven architectural approaches. Now that Websub has matured as a standard, I’m even more interested in leveraging it as part of my diverse API toolbox.th

Server Sent Events (SSE) I consider webhooks to be the gateway drug for API event-driven architecture. Making API integrations a two street, while also making them more efficient, and potentially real time. After webhooks, the next tool in my toolbox for making API consumption more efficient and real time are server-sent events (SSE). Server-sent events (SSE) is a technology where a browser receives automatic updates from a server via a sustained HTTP connection, which has been standardized as part of HTML5 by the W3C. The approach is primarily used to established a sustained connection between a server, and the browser, but can just as easily be used server to server.

Server-sent events (SSE) delivers one-way streaming APIs which can be used to send regular, and sustained updates, which can be more efficient than regular polling of an API. SSE is an efficient way to begin going beyond the basics of client-server request and response model and pushing the boundaries of what APIs can do. I am using SSE to make APIs much more real time, while also getting more precise with the delivery of data and content, leverage other standards like JSON Patch to only provide what has changed, rather than sending the same data out over the pipes again, making API communication much more efficient.

Websockets Shifting things further into real time, websockets is what I’m using to deliver two-way API streams that require data be both sent and received, providing full-duplex communication channels over a single TCP connection. WebSocket is a different TCP protocol from HTTP, but is designed to work over HTTP ports 80 and 443 as well as to support HTTP proxies and intermediaries, making it compatible with the HTTP protocol. To further achieve compatibility, the WebSocket handshake uses the HTTP Upgrade header to change from the HTTP protocol to the WebSocket protocol, pushing the boundaries of APIs beyond HTTP in a very seamless way.

SSE is all about the one-way efficiency, and websockets is about two-way efficiency. I prefer keeping things within the realm of HTTP with SSE, unless I absolutely need the two-way, full-duplex communication channel. As you’ll see, I’m fine with pushing the definition of API out of the HTTP realm, but I’d prefer to keep things within bounds, as I feel it is best to embrace HTTP when doing business on the web. I can accomplish a number of objectives for data, content, media, and algorithmic access using the HTTP tools in my toolbox, leaving me to be pretty selective when I push things out of this context.

gRPC Using HTTP/2 While I am forced to use Websockets for some existing integrations such as with Twitter, and other legacy implementations, it isn’t my choice for next generation projects. I’m opting to keep things within the HTTP realm, and embracing the next evolution of the protocol, and follow Google’s lead with gRPC. As with other RPC approaches, gRPC is based around the idea of defining a service, specifying the methods that can be called remotely with their parameters and return types. gRPC embraces HTTP/2 as its next generation transport protocol, and while also employing Protocol Buffers, Google’s open source mechanism for the serialization of structured data.

At Google, I am seeing Protocol Buffers used in parallel with OpenAPI for defining JSON APIs, providing two speed APIs using HTTP/1.1 and HTTP/2. I am also seeing Protocol Buffers used with HTTP/1.1 as a transport, making it something I have had to integrated with alongside SOAP, and other web APIs. While I am integrating with APIs that use Protocol Buffers, I am most interested in the usage of HTTP/2 as a transport for APIs, and I am investing more time learning about the next generation headers in use, and the variety of approaches in which HTTP/2 is used as a transport for traditional APIs, as well as multi-directional, streaming APIs.

Apache Kafka Another shift I could not ignore across the API landscape in 2017 was the growth in adoption of Kafka as a distributed streaming API platform. Kafka focuses on enabling providers to read and write streams of data like a messaging system, and develop applications that react to events in real-time, and store data safely in a distributed, replicated, fault-tolerant cluster. Kafka was originally developed at LinkedIn, but is now an Apache open source product that is in use across a number of very interesting companies, many of which have been sharing their stories of how efficient it is for developing internal data pipelines. I’ve been studying Kafka throughout 2017, and I have added it to my toolbox, despite it pushing the boundaries of my definition of what is an API beyond the HTTP realm.

Kafka has moved out of the realm of HTTP, using a binary protocol over TCP, defining all APIs as request response message pairs, using its own messaging format. Each client initiates a socket connection and then writes a sequence of request messages and reads back the corresponding response message–no handshake is required on connection or disconnection. TCP is much more efficient over HTTP because it allows you to maintain persistent connections used for many requests. Taking streaming APIs to new levels, providing a super fast set of open source tools you can use internally to deliver the big data pipeline you need to get the job done. My mission is to understand how these pipelines are changing the landscape and which tools in my toolbox can help augment Kafka and deliver the last mile of connectivity to partners, and public applications.

Message Queuing Telemetry Transport (MQTT) Continuing to round off my API toolbox in a way that pushes the definition of APIs beyond HTTP, and helping me understand how APIs are being used to drive Internet-connected devices, I’ve added Message Queuing Telemetry Transport (MQTT), an ISO standard for implementing publish-subscribe-based messaging protocol to my toolbox. The protocol works on top of the TCP/IP protocol, and is designed for connections with remote locations where a light footprint” is required because compute, storage, or network capacity is limited. Making MQTT optimal for considering when you are connecting devices to the Internet, and unsure of the reliability of your connection.

Both Kafka, and MQTT have shown me in the last couple of years, the limitations of HTTP when it comes to the high and low volume aspects of moving data around using networks. I don’t see this as a threat to APIs that leverage HTTP as a transport, I just see them as additional tools in my toolbox, for projects that meet these requirements. This isn’t a failure of HTTP, this is simply a limitation, and when I’m working on API projects involving internet connected devices I’m going to weight the pros and cons of using simple HTTP APIs, alongside using MQTT, and being a little more considerate about the messages I’m sending back and forth between devices and the cloud over the network I have in place. MQTT reflects my robust and diverse API toolbox, as one that gives me a wide variety of tools I’m familiar with and can use in different environments.

Mastering My Usage Of Headers One thing I’ve learned over the years while building my API toolbox is the importance of headers, and they are something that have regularly been not just about HTTP headers, but the more general usage of network networks. I have to admit that I understood the role of headers in the API conversation, but had not fully understood the scope of their importance when it comes to taking control over how your APIs operate within a distributed environment. Knowing which headers are required to consume APIs is essential to delivering stable integrations, and providing clear guidance on headers from a provider standpoint is essential to APIs operating as expected on the open web.

Content negotiation was the header doorway I walked through that demonstrated the importance of HTTP headers when it comes to deliver the meaningful API experiences. Being able to negotiate CSV, XML, and JSON message formats, as well as being able to engage with my digital resources in a deeper way using hypermedia media types. My headers mastery is allowing me to better orchestrate an event-driven experience via webhooks, and long running HTTP connections via Server-Sent Events. They are also taking me into the next generation of connectivity using HTTP/2, making them a critical aspect of my API toolbox. Historically, headers have often been hidden in the background of my API work, but increasingly they are front and center, and essential to me getting the results I’m looking for.

Standardizing My Messaging I have to admit I had taken the strength of message formats present in my web service days for granted. While I still think they are bloated and too complex, I feel like we threw out a lot of benefits when we made the switch to more RESTful APIs. Overall I think the benefits of the evolution were positive, and media types provide us with some strong ways to standardize the messages we pass back and forth. I’m fine operating in a chaotic world of message formats and schema that are developed in the moment, but I’m a big fan of all roads leading to standardization and reuse of meaningful formats, so that we can try to speak with each other via APIs in more common formats.

I do not feel that there is one message format to rule them all, or that even one for each industry. I think innovation at the message layer is important, but I also feel like we should be leveraging JSON Schema to help tame things whenever possible, and standardize as media types. Whenever possible, reuse existing standards from day one is preferred, but I get that this isn’t always the reality, and in many cases we are handed the equivalent of a filing cabinet filled with handwritten notes. In my world, there will always be a mixed of known and unknown message formats, something that I will always work to tame, as well as be increasingly apply machine learning models to help me identify, evolve, and make sense of–standardizing things in any way I possibly can.

Knowing (Potential) Clients I am developing APis for a wide variety of clients. Some are designed for web applications, others are mobile applications, and some are devices. They could be spreadsheets, widgets, documents, and machine learning models. The tables could be flipped, and the APIs exist on device, and the cloud becomes the client. Sometimes the clients are known, other times they are unknown, and I am looking to attract new types of clients I never envisioned. I am always working to understand what types of clients I am looking to serve with my APIs, but the most important aspect of this process is understanding when there will be unknown clients.

When I have a tightly controlled group of target clients, my world is much easier. When I do not know who will be developing against an API, and I am looking to encourage wider participation, this is when my toolbox comes into action. This is when I keep the bar as low as possible regarding the design of my APIs, the protocols I use, and the types of data formats and messages I use. When I do not know my API consumers and the clients they will developing, I invest more in API design, and keep my default requests and responses as simple as possible. Then I also allow for the negotiation of more complex, higher speed, more control aspects of my APIs by consumers who are in the know, targeting more specific client scenarios.

Using The Right Tools For The Job API is not REST. It is one tool in my toolbox. API deployment and integration is about having the right tool for the job. It is a waste of my time to demand that everyone understand one way of doing APIs, or my way of doing APIs. Sure, I wish people would study and learn about common API patterns, but in reality, on the ground in companies, organizations, institutions, and government agencies, this is not the state of things. Of course, I’ll spend time educating and training folks wherever I can, but my role is always more about delivering APIs, and integrating with existing APIs, and my API toolbox reflects this reality. I do not shame API providers for their lack of knowledge and available resources, I roll up my sleeves, put my API toolbox on the table, and get to work improving any situation that I can.

My API toolbox is crafted for the world we have, as well as the world I’d like to see. I rarely get what I want on the ground deploying and integrating with APIs. I don’t let this stop me. I just keep refining my awareness and knowledge by watching, studying, and learning from what others are doing. I often find that when someone is in the business of shutting down a particular approach, or being dogmatic about a single approach, it is usually because they aren’t on the ground working with average businesses, organizations, and government agencies–they enjoy a pretty isolated, privileged existence. My toolbox is almost always open, constantly evolving, and perpetually being refined based upon the reality I experience on the ground, learning from people doing the hard work to keep critical services up and running, not simply dreaming about what should be.


My Evolving Definition Of A Robust And Diverse API Toolbox

It is always telling when folks assume I mean REST when I say API. While the web dominates my definition of API, and REST is definitely a leading architectural style, these assumptions always define the people who bring them to the table, more than they ever do me. I’m in the business of studying how people are applying programmatic interfaces using the web. To reflect my research I’ve been evolving a diagram of my toolbox that I’ve been publishing as part of workshops, presentations, and some talks I’m preparing for 2018. It reflects what I’m seeing as the evolving API toolbox that I’m seeing companies working with, and a diversity in which I’m encouraging others to think about more, as we choose to ignore the polarizing forces in the API sector.

To set the tone for any API conversation I am participating in, I prefer to introduce the concept of the API toolbox including more tools than just REST, acknowledging that their are a growing number of tools in our API infrastructure toolbox which can be applied to different APIs, to solve a variety of problems and challenges we face. Also we need to be more honest about the fact that there are many legacy solutions still in use across large organizations, even as we consider adopting the latest in leading edge approaches to API deployment in newer projects.

  • HTTP - Leverage the web, and the HTTP standard across ALL API efforts.
  • SOAP - Acknowledging there are still a number of SOAP services in use.
  • RPC - Understand how and why RPC APIs still might be viable in production.
  • REST - Making REST, and a resource-centered approach the focus of the operations.
  • Microservices - Emphasis on independently deployable and module API services.
  • Verbs - Knowing, and putting to use HTTP verbs across API implementations.
  • Content-Type - Understanding the negotiation between XML, JSON, and other types.
  • Hypermedia - Considering how hypermedia design, and content types play a role.
  • GraphQL - Thinking about GraphQL when it comes to data intensive API projects.
  • HTTP/2 - Understanding and embracing the evolution of the HTTP standard.
  • gRPC - Considering two-speed APIs, and using gRPC for higher volume API implementations.
  • Webhooks - Seeing APIs as a two-way street, and pushing data to APIs as well as receiving.
  • Server-Sent Events (SSE) - Leveraging HTTP push technology to make things real time.
  • Websockets - Opening up two streams that allow for bi-directional API interactions.
  • PubSubHubbub - Considering a distributed publish-subscribe approach to API interactions.
  • Apache - Being aware of the Apache stack which includes Spark, Kafka, and other real time data solutions.

While HTTP and REST are definitely the focal point of many API conversations I am in, SOAP and RPC are legacy realities we must accept are still getting the job done in many environments, and we shouldn’t be shaming the folks who own this infrastructure. At the same time I’m helping folks unwind this legacy infrastructure, I also find myself participating in discussions around event-driven architecture, streaming, and HTTP/2 which represent where API architecture is headed. I’m needing a toolbox that reflects this spectrum of API tooling, as well as where we’ve been, and find ourselves still supporting in 2018.

I’m still evaluating the Apache stack, as well as GraphQL and gRPC, to better understand how they fit in my definition. This work, as well as part new partnership with Streamdata.io is pushing me to re-evaluate exactly what is real time and streaming using webhooks and server-sent events, alongside a more event-driven approach I am seeing emerge within many leading organizers. People love to say that APIs are done. I wish I could show how silly this way of thinking makes y’all look. The idea that using the web to exchange data, content, and algorithms in a machine readable formats is going anywhere is laughable. My objective is to keep tracking on the tools people are using to get this job done, and help folks ensure their toolbox is as robust and diverse as possible, not traffic in silly dogmatic fantasies about API trends and religions.


My Evolving Definition Of A Robust And Diverse API Toolbox

It is always telling when folks assume I mean REST when I say API. While the web dominates in my definition of API, and REST is definitely a leading architectural style, these assumptions always define the people who bring them to the table, more than they ever do me. I’m in the business of studying how people are applying programmatic interfaces using the web. To reflect my research I’ve been evolving a diagram of my toolbox that I’ve been publishing as part of workshops, presentations, and some talks I’m preparing for 2018. It reflects what I’m seeing as the evolving API toolbox that I’m seeing companies working with, and a diversity in which I’m encouraging others to think about more, as we choose to ignore the polarizing forces in the API sector.

To set the tone for any API conversation I am participating in I prefer to introduce the concept of the API toolbox including more tools than just REST, acknowledging that their are are a growing number of tools in our API infrastructure toolbox which can be applied to different APIs, to solve a variety of problems and challenges we face. Also we need to be more honest about the fact that there are many legacy solutions still in use across large organizations, even as we consider adopting the latest in leading edge approaches to API deployment in newer projects.

  • HTTP - Leverage the web, and the HTTP standard across ALL API efforts.
  • SOAP - Acknowledging there are still a number of SOAP services in use.
  • RPC - Understand how and why RPC APIs still might be viable in production.
  • REST - Making REST, and a resource-centered approach the focus of the operations.
  • Microservices - Emphasis on independently deployable and module API services.
  • Verbs - Knowing, and putting to use HTTP verbs across API implementations.
  • Content-Type - Understanding the negotiation between XML, JSON, and other types.
  • Hypermedia - Considering how hypermedia design, and content types play a role.
  • GraphQL - Thinking about GraphQL when it comes to data intensive API projects.
  • HTTP/2 - Understanding and embracing the evolution of the HTTP standard.
  • gRPC - Considering two-speed APIs, and using gRPC for higher volume API implementations.
  • Webhooks - Seeing APIs as a two-way street, and pushing data to APIs as well as receiving.
  • Server-Sent Events (SSE) - Leveraging HTTP push technology to make things real time.
  • Websockets - Opening up two streams that allow for bi-directional API interactions.
  • PubSubHubbub - Considering a distributed publish-subscribe approach to API interactions.
  • Apache - Being aware of the Apache stack which includes Spark, Kafka, and other real time data solutions.

While HTTP and REST are definitely the focal point of many API conversations I am in, SOAP and RPC are legacy realities we must accept are still getting the job done in many environments, and we shouldn’t be shaming the folks who own this infrastructure. At the same time I’m helping folks unwind this legacy infrastructure, I also find myself participating in discussions around event-driven architecture, streaming, and HTTP/2 which represent where API architecture is headed. I’m needing a toolbox that reflects this spectrum of API tooling, as well as where we’ve been, and find ourselves still supporting in 2018.

I’m still evaluating the Apache stack, as well as GraphQL and gRPC, to better understand how they fit in. This work, as well as part new partnership with Streamdata.io is pushing me to re-evaluate exactly what is real time and streaming using webhooks and server-sent events, alongside a more event-driven approach I am seeing emerge within many leading organizers. People love to say that APIs are done. I wish I could show how silly this way of thinking makes y’all look. The idea that using the web to exchange data, content, and algorithms in a machine readable formats is going anywhere is laughable. My objective is to keep tracking on the tools people are using to get this job done, and help folks ensure their toolbox is as robust and diverse as possible, not traffic in silly dogmatic fantasies about API trends and religions.


Always Being Prepared For An Api Future That May Never Come

404: Not Found


Just Waiting The GraphQL Assault Out

I was reading a story on GraphQL this weekend which I won’t be linking to or citing because that is what they want, and they do not deserve the attention, that was just (yet) another hating on REST post. As I’ve mentioned before, the GraphQL’s primary strength seems to be they have endless waves of bros who love to write blog posts hating on REST, and web APIs. This particular post shows it’s absurdity by stating that HTTP is just a bad idea, wait…uh what? Yeah, you know that thing we use for the entire web, apparently it’s just not a good idea when it comes to exchanging data. Ok, buddy.

When it comes to GraphQL, I’m still watching, learning, and will continue evaluating it as a tool in my API toolbox, but when it comes to the argument of GraphQL vs. Web APIs I will just be waiting out the current assault as I did with all the other haters. The link data haters ran out of steam. The hypermedia haters ran out of steam. The GraphQL haters will also run out steam. All of these technologies are viable tools in our API toolbox, but NONE of them are THE solution. These assaults on “what came before” is just a very tired tactic in the toolbox of startups–you hire young men, give them some cash (which doesn’t last for long), get them all wound up, and let them loose talking trash on the space, selling your warez.

GraphQL has many uses. It is not a replacement for web APIs. It is just one tool in our toolbox. If you are following the advice of any of these web API haters you will wake up in a couple of years with a significant amount of technical debt, and probably also be very busy chasing the next wave of technology be pushed by vendors. My advice is that all API providers learn about the web, gain several years of experience developing web APIs, learn about linked data, hypermedia, GraphQL, and even gRPC if you have some high performance, high volume needs. Don’t spend much time listening to the haters, as they really don’t deserve your attention. Eventually they will go away, find another job, and technological kool-aid to drink.

In my opinion, there is (almost) always a grain of usefulness with each wave of technology that comes along. The trick is cutting through the bullshit, tuning out the haters, and understanding what is real and what is not real when it comes to the vendor noise. You should not be adopting every trend that comes along, but you should be tuning into the conversation and learning. After you do this long enough you will begin to see the patterns and tricks used by folks trying to push their warez. Hating on whatever came before is just one of these tricks. This is why startups hire young, energetic, an usually male voices to lead this charge, as they have no sense of history, and truly believe what they are pushing. Your job as a technologist is to develop the experience necessary to know what is real, and what is not, and keep a cool head as the volume gets turned up on each technological assault.


Revisiting GraphQL As Part Of My API Toolbox

I’ve been reading and curating information on GraphQL as part of my regular research and monitoring of the API space for some time now. As part of this work, I wanted to take a moment and revisit my earlier thoughts about GraphQL, and see where I currently stand. Honestly, not much has changed for me, to move me in one direction or another regarding the popular approach to providing API access to data and content resources.

I still stand by my cautionary advice for GraphQL evangelist regarding not taking such an adversarial stance when it comes to the API approach, and I feel that GraphQL is a good addition to any API architect looking to have a robust and diverse API toolbox. Even with the regular drumbeat from GraphQL evangelists, and significant adoption like the Github GraphQL API I am not convinced it is the solution for all APIs and is a replacement for simple RESTful web API design.

My current position is that the loudest advocates for GraphQL aren’t looking at the holistic benefits of REST, and too wrapped in ideology, which is setting them up for similar challenges that linked data, hypermedia, and even early RESTafarian practitioners have faced. I think GraphQL excels when you have a well educated, known and savvy audience, who are focused on developed web and mobile applications–especially the latest breed of single page applications (SPA). I feel like in this environment GraphQL is going to rock it, and help API providers reduce friction for their consumers.

This is why I’m offering advice to GraphQL evangelists to turn down the anti-REST, and complete replacement/alternative for REST–it ain’t helping your cause and will backfire for you. You are better to off educating folks about the positive, and being honest about the negatives. I will keep studying GraphQL, understanding the impact it is making, and keeping an eye on important implementations. However, when it comes to writing about GraphQL you are going to see me continuing to hold back, just like I did when it came to hypermedia and linked data because I prefer not to be in the middle of ideological battles in the API space. I prefer showcasing the useful tools and approaches that are making a significant impact across a diverse range of API providers–not just echoing what is coming out of a handful of big providers, amplified by vendors and growth hackers looking for conversions.


If you think there is a link I should have listed here feel free to tweet it at me, or submit as a Github issue. Even though I do this full time, I'm still a one person show, and I miss quite a bit, and depend on my network to help me know what is going on.