In this blog post, CTO Richard Develyn unpicks the confusion surrounding the significance of the various levels of application interfacing. He outlines how APIs and data-stream format are not as complex an issue as many people feel them to be. He argues that it is, rather, sufficiency and process in interfacing that are of much greater importance when ensuring that applications can successfully communicate with each other.
We probably spend more time worrying about the things that we shouldn’t and not worrying about the things that we should with regards to interfacing than we do about anything else.
Interfacing is a many-layered thing. Some of the layers are hard and some are easy. Unfortunately, all of the layers are called “interfacing” and that’s where the confusion appears, because we want to make sure that we have a clear answer in our heads about whether interfacing is easy or hard and the fact is that it’s both easy and hard depending on how you’re looking at it.
We therefore need to prise apart these layers in order to get a better understanding of it, in particular with respect to the challenges that we face here at CloudTrade when we try to connect our analytical capture service to other applications and systems.
This is the least important part of the interfacing challenge and yet it is often the one that gets the most talked about. Although API stands for Application Programming Interface, in actuality it is interfacing at the simplest possible level in inter-application communication. APIs provide the mechanism for applications A and B to exchange data, but that’s as far as it goes – i.e. APIs make no guarantee on whether A and B will actually speak the same language or be able to make sense of what each of them is doing.
The reason APIs get a lot of prevalence is because there are a great many As and Bs in this world and therefore many APIs for connecting them. Sometimes these APIs are standardised, such as SFTP, HTTPS. Sometimes they’re not, which can give rise to interesting API wars, where the manufacturer of application A will publish an API for connecting with it, as will the manufacturer of application B; the APIs will not be compatible and neither side will want to shift to accommodate the other. Outside of such chest-beating, however, companies such as ours will always try to work with any API, and will happily implement a bespoke API as long as it is documented and the customer will commit to help with the testing. It is typically 1 – 2 weeks of development work for a completely new API.
The bottom line is – don’t worry about APIs. If we don’t support your API but we’re definitely going to be doing business together then we’ll sort that problem out even if it involves a small amount of new development.
If the API sorts out the wire that connects us, “format” is where we agree on what the data that we exchange actually means. Typically, this is done with XML file / data-stream formats, though all sorts of other formats are possible too. XML stands for Extensible Markup Language, and what it basically says with the word “markup” is that it defines how meaning is attributed to data. For example, in the XML sequence:
the data item “Richard” is given the meaning “name”. If I extend the example like this:
Then the meaning of “Richard” is further qualified as being a customer’s first name.
XML file formats tend to be more standardised, though variations in the use of these standards, as well as extensions to the standards and the presence of completely bespoke file formats also do exist. Generally speaking, however, people don’t tend to worry so much about getting this sort of interfacing sorted out as much as they do about APIs; I guess it feels easier, though it can actually be just as awkward.
At the end of the day, however, the bottom line for format interfacing is the same as with APIs. If we’re not already using a common interface, we might need to do some additional configuration work at our end to sort that out, but it’s all quite possible.
This is where things can get a bit thorny. If we can’t provide you with everything that you need, or vice versa, then we’re not going to be able to work together.
This can be so “drop dead” that if we have any sort of doubt whatsoever then we need to bottom it out before we go any further. The difficulty is that sometimes these problems of insufficiency are down in the detail, the most classic one for us being if your finance system cannot process invoices without line-level, purchase-order line numbers, and your suppliers don’t provide them in their invoices, and we have no other place that we can get them from.
Sometimes the problem is not so “drop dead”, but might imply that there is some significant work on our part that is needed for us to do in order to get that missing information, or possibly even on your part to adjust your target system so that it doesn’t need that information any more. Either way, this interfacing layer is an important one, so the bottom line here is that we need to make sure early on that we don’t have an unresolvable issue with sufficiency.
Is our application doing what your application needs it to do? It must be, otherwise we probably wouldn’t be talking to each other in the first place. You would have thought, anyway.
In the Accounts Payable and Accounts Receivable application space, where we have been operating for many years, this isn’t really a question in anybody’s minds, but in some of the new areas that we are moving into, such as Fraud Detection and RPA data feeds, some sort of sanity check on expectations should take place.
For example, if we detect fraud, are you expecting us to call the police? Probably not, but the bottom line for this one is that we need to make sure that we have a common understanding of the functionality that we are providing and you are expecting.
Finally, the one that seems to cause us more problems than any other.
Applications have Use Cases – a term borrowed from Software Analysis and Design but now used to catalogue all of the different ways that an application can be used. When applications join together, these Use Cases have to interface with each other in order to satisfy the over-all Use Cases of the application suite. If you’re not careful to join these Use Cases together properly then anomalies and holes can occur into which transactions can fall never to be seen again.
This requires careful analysis of how each Use Case is going to progress through the whole system. The problematic ones tend to be the ones where an error occurs somewhere in the process. If it happens in the “feeder” application, which is typically ours here at CloudTrade, then the “receiver” application must be able to receive this error from us (however it wants) and do something sensible with it. If the error occurs in the “receiver” application, perhaps as a result of something that we did but had no idea was wrong when we fed the transaction through, then again the “receiver” application has to do something sensible with that. The two things that cannot happen are for the “feeder” application to be left holding the baby with nowhere to put it, or for the “receiver” application to drop the baby into some black hole which is either literally a black hole or one that nobody ever bothers looking into.
The bottom line for process interfacing is that this has to be sorted out before we do business, otherwise we’re likely to set something up which will ultimately fail.
We probably wouldn’t even be discussing working together if we didn’t have our application-level interfacing needs sorted out, but it’s worth just touching base if we’re not in the world of Accounts Payable or Accounts Receivable. After that we should take the time to ensure we have no fundamental problems with either sufficiency-level or process-level interfacing before we even begin to get worried about APIs or data formats. As for the latter two, although we’re very unlikely to develop bespoke API or data formats “on spec”, because there’s just so many of them, they aren’t a barrier to business as long as we agree development costs, and we can generally develop or modify what we have during the time that we take to deploy a new system.
Interfacing is a many-layered problem. The important thing as we consider working together is to make sure that we worry about process and sufficiency, rather than API or data format, as the former are the ones that can cause real issues, whereas the latter are ones we can sort out with a little bit of time and effort.