Machine Learning and Hybrid Solutions

‘Project Grandalf’, Machine Learning and Hybrid Solutions

Reading Time: 4 minutes
Machine Learning and Hybrid Solutions - Project Grandalf
‘Project Grandalf’, Machine Learning and Hybrid Solutions

Making the right choice when looking at the technological solutions to solve your business problems has always been an issue. Too much commitment, too greater system change, perhaps too greater a cost – but what if you could find the best of both worlds? Richard Develyn, Chief Technical Officer, discusses the new and exciting developments in the CloudTrade solution suite to bring about the best of both worlds when it comes to document content recognition – Project Grandalf.

Implementing a Machine Learning solution in your organisation is a bit of a risky undertaking: you’re never entirely sure how long you’re going to have to wait before you get to see anything useful come out of it, or even whether something useful is ever going to come out of it at all.

In the meantime, however, you have to find a way of delivering your core business in an efficient and dependable way, which these days means using IT: not the strange, new, neural-network style IT being hyped about so much now, but rather the traditional IT which has been holding the world together for the last 50 years or so.

Machine Learning, exciting though it is, is still a very long way away from being able to take over from IT in total, and although there is a class of problem where traditional IT struggles, such as those requiring human assistance, which is where Machine Learning can usefully participate, most IT solutions are still delivered using the traditional IT “programming” way.

Hybrid solutions, however, can combine all of these approaches in order to get the best out of all worlds, as long as they’re created with care. Human judgement is slow to make and Machine Learning slow to learn, so processing still needs to go through the traditional IT route as much as possible if IT-speed levels of automation are going to be achieved. Humans and Neural Networks, however, can configure the system to make it run more accurately or efficiently, without trying to take over the job completely.

Project Grandalf, as it is affectionately known by the CloudTrade Development Team, is CloudTrade’s hybrid solution to the problem of document content recognition.

Extracting properly identified data from human readable documents is a complicated problem to solve. CloudTrade’s flagship product, Gramatica, does so by implementing a rules engine which allows rules to be written specifically for each document format to be processed. It is extraordinarily powerful, and Gramatica can deal with any requirement and complexity as long as it has the right rules written to do so.

Some documents, however, are not sufficiently complicated, or processed in sufficient quantities, to justify the rules-writing effort that Gramatica requires.

This is where ‘Grandalf’ comes in.

Grandalf’s Machine Learning Engine, which runs every night on a comprehensive data sample set, powers its knowledge database of data extraction algorithms. This collection of algorithms is then applied to every new document which arrives at the service, with an operator asked to clarify which algorithm has produced the right answers.

The operator’s responses are persisted in a database so that documents subsequently received from the same sender are automatically addressed by the right algorithm without further need for operator intervention. Should there be a variation in a document so that the right answers cannot be found, then the knowledge-base and operator process can be re-invoked so that the alternatives can be accurately handled.

It is this combination of Machine Learning, traditional IT and human intervention which provides the key benefit that Grandalf brings against the competition. It also illustrates the advantage that hybrid systems have against the more common “one technology only” approach.

There has always been a tendency in the marketplace to look for silver bullets. Silver bullets are easy to sell (i.e. “you have this problem, you use this silver bullet; you have that problem, you use that one”). Even if you were shooting at werewolves, however, you would be silly to make your bullets entirely out of silver – just put enough silver in them to make them toxic to the creature you’re shooting at then build the rest from good old fashioned lead-antimony and steel (that’s probably how they did it – back in the day).

We, at CloudTrade, don’t believe in silver bullets (or werewolves). We believe in solutions which are crafted from the best that the different technologies relative to the problem can offer, especially when they are made to work in harmony. As a result we are firmly convinced that Grandalf’s hybrid solution is the best way to approach the problem of document content recognition, beyond those documents which are so complicated that they require specific rules to be written to understand them (i.e. Gramatica). It is this hybrid combination of approaches that allows Grandalf to hit the sweet spot that the solution demands: Machine Learning and human assistance supporting traditional, deterministic, IT.

Grandalf is characterised by the following features:

  • It learns from one example only
    • Grandalf has a huge knowledge base of data capture rules which are applied to every document, with an operator then asked to help via a simple question and answer form
  • It’s 100% accurate
    • Once chosen, rules don’t exercise judgement or refer back to some Machine Learning database to get possible values and confidence levels; Grandalf’s rules are completely deterministic
  • It’s fast
    • Once an operator has helped Grandalf determine which rules should be used, documents fly through it at the speed of IT
  • It handles document variations
    • Grandalf returns to an operator if the rules for a given document fail to find a value, re-running its knowledge base to offer more alternatives
  • It continually learns and improves
    • CloudTrade’s selected data set feeds into the Machine Learning algorithm which every night updates Grandalf’s rules knowledge base by adding further data capture possibilities
  • It’s expandable
    • Grandalf can easily be expanded to cater for additional customer capture requirements

Project Grandalf, is set to be released and available to CloudTrade customers, under its official name, in January 2021. If you’d like to know more about how CloudTrade can help your business automate its data and documents, irrespective of volume or document type, please arrange a short meeting with us here.

Tour de France 2020

Bonjour de CloudTrade – en présentant la première Française de l’équipe, Rose Massie

Reading Time: 5 minutes
Tour de France

Read this blog post in English at the bottom of the page.

Comme CloudTrade continue d’élargir sa clientèle dans le monde entier, nous avons décidé d’agrandir notre équipe et avons fait notre première embauche en France. Parallèlement, au lancement de notre site web en français, nous cherchons à faire connaître les produits et services de CloudTrade à un public plus large en France et dans les régions francophones. Dans son premier billet de blog pour CloudTrade, nous présentons Rose Massie qui a eu l’extraordinaire privilège de voir le Tour de France dans sa ville la semaine dernière.

Allez CloudTrade!

Bonjour à tous de Charente-Maritime. Moi, je m’appelle Rose et je suis ravie d’avoir été nommée la nouvelle Directrice de Marketing en France. CloudTrade a déjà une forte présence en France et à partir de maintenant nous offrirons plus de contenu pour nos francophones, y compris le lancement prochain de notre site web en langue française – restez à l’écoute pour le lancement. Je suis extrêmement fière de faire partie de cette entreprise dynamique et je me réjouis de pouvoir aider davantage d’entreprises en France à automatiser leur saisie de données. Même dans la circonstance actuelle qui représente un défi pour nous tous, CloudTrade garde son optimisme et a de plus en plus de clients qui ont besoin de ses services. C’était avec cet esprit de confiance que je me suis trouvée avec le PDG de CloudTrade, David Cocks, à Saint-Palais-sur-Mer en Charente Maritime pour quelques jours cet été. Il est de retour en Angleterre et, bien sûr il est maintenant obligé de travailler de chez lui à cause des nouvelles restrictions. Toutefois, David a eu la chance ultime de voir le Tour de France passer dans ma ville, à cinq minutes de la maison !

C’était un honneur énorme pour CloudTrade de soutenir le Tour de France et, le Jour J arrivé, moi j’ai choisi la robe la plus jaune dans mon armoire. David portait une casquette bleue CloudTrade pour diffuser la présence de l’entreprise d’une manière colorée. Nous avons regardé le peloton de tout près dans l’Avenue de la République à Saint-Palais-sur-Mer avec les fans de cyclisme palaisiens et d’autres touristes de la ville, pour la plupart français – tout le monde, sauf les coureurs, bien masqués comme il faut.

Le tracé de cette dixième étape du Tour est allé de l’Ile d’Oléron à l’Ile de Ré via mon petit coin des Charentes. Il y avait une grande caravane publicitaire qui a pris une avance de 1 h 40 sur le peloton. Alors, il fallait attendre avec patience en attendant l’événement lui-même. Heureusement, la foule n’était pas dense et nous pouvions voir les coureurs en gros plan. C’était vraiment comme un rêve. Le peloton est arrivé en fanfare et avec un bourdonnement palpitant. Le ciel était d’un bleu-azure et la grande chaleur était modérée par un vent léger de l’ouest agréable. J’ai fait une vidéo de vingt-deux secondes, puis les cyclistes ont disparu à toute allure, direction Ile de Ré. Ça a été vingt-deux seconds de splendeur – et, hourra, CloudTrade était là où s’est trouvé l’événement du jour.

 Veuillez consulter le site web de CloudTrade, qui sera bientôt lancé en langue française avec une multitude de blogs et plus d’informations sur nos produits. Vous pouvez me trouver sur LinkedIn et je vous invite à me contacter si vous souhaiter en savoir plus sur nos produits et services.

Rose Massie

Après avoir étudié le français et l’allemand à l’université, Rose a travaillé dans le domaine de l’éducation et de la traduction pendant de nombreuses années avant de se lancer dans le monde des affaires et du marketing.

Elle partage son temps entre Saint-Palais-sur-Mer et le Royaume-Uni. Elle est mariée, a quatre enfants adultes, et passe ses week-ends à marcher, jardiner et lire.

English translation

Bonjour from CloudTrade – Introducing the first French team member Rose Massie

As CloudTrade continues to expand its customer base worldwide, we have decided to grow our team and made our first hire in France. Alongside the launch of our French language website, we look to bring the products and services of CloudTrade to a wider audience in France and the French-speaking regions. In her first blog post for CloudTrade, we introduce Rose Massie, who had the extraordinary privilege of seeing the Tour de France in her home town, just last week.

Allez CloudTrade!

Bonjour to everyone from Charente-Maritime. My name is Rose, and I am delighted to have been appointed the Marketing Manager in France. CloudTrade already has a strong presence in France and from now on we will offer more content for our French speakers, including the upcoming launch of our French language website – stay tuned for the launch. I am extremely proud to be part of this dynamic company and look forward to helping more businesses in France to automate their data capture.   

Even in the current challenging situation, CloudTrade is optimistic and moving forward with growth plans. It was in this confident frame of mind that I spent a few days this summer with CloudTrade’s CEO, David Cocks, who joined me in Saint-Palais-sur-Mer in Charente Maritime on the west coast of France. Having returned to England he must of course now work in isolation for two weeks due to the new restrictions. For David this is definitely a price worth paying for as he had the great good fortune to see the Tour de France pass through my town – only five minutes’ walk from the house.

It was a great honour for CloudTrade to support the Tour de France! On the day I picked a yellow dress from the wardrobe and David sported a blue CloudTrade cap to add a little colourful publicity to the day. We were able to watch the peloton from right up close, at the very edge of the local Avenue de la République, standing with local cycling fans and some other (mostly French) tourists, not too close together and all of us appropriately masked.

The route of this 10th stage of the Tour went from Ile d’Oléron to Ile de Ré via my neck of the woods in Charente-Maritime. There was a raucous one-and-a-half-hour procession of advertising floats that noisily prepared the ground for the arrival of the cyclists, so we had to exercise some patience while waiting for the event itself. However, we were delighted with our wonderful roadside position and when the competitors came into view it was like entering into a fabulous dream. The peloton was given a spectacular welcome and there was a thrilling and intense whirring of very fine wheels. The sky was azure blue and the unusual September heat was moderated by a welcome westerly breeze. I busied myself making a 22-second video and then they were gone, at unbelievable speed, off to Ile de Ré. It had been a magnificent 22 seconds – and, hurrah, CloudTrade had been present at the action!

Please look out for the CloudTrade website, launching in the French language soon, with a host of blogs and more information about our products and services. You can find me on LinkedIn and I invite you to contact me as I would very much appreciate discussing your company’s data capture and document automation requirements.

About Rose Massie –

Having specialised in French and German at University, Rose worked within education and translation for many years before moving into business and software sales.

She divides her time between Saint-Palais-sur-Mer and the UK, is married and has four grown-up children. She spends her weekends walking, gardening and reading.

Video of the Tour de France –

The Tour de France en Charente-Maritime

Did you hear the long tale about the long tail?

Reading Time: 4 minutes
Giraffe
A tail up to 2.4m is pretty long

National Geographic tells us that the Giraffe has the longest tail of any land mammal – a Giraffes tail can measure up to 2.4m (apparently).

I asked my children what other animals have a long tail (it should be noted that during the months of lockdown we’ve been asking more and more abstract questions, so this seemed quite normal to them!). Responses included: the ring tailed lemur, monkeys (no particular sub-species offered), rodents and our (long suffering) dog. However, the longest tail when it comes to the proportion of an animal’s body in relation to its tail length is the Asian grass lizard, according to National Geographic. Although the tail length is only 25cm’s long, this is over three times its body length.

I’ve been fortunate to have travelled the world quite a bit for work and fun and I’ve had opportunity to see giraffes, lizards and monkeys up close. I’ve also had opportunity to see rodents up close (very close!) recently during a house renovation.

Each of these animals seems quite comfortable with its body and its long tail. In fact, our dog can entertain herself for hours chasing her tail (but that’s another blog altogether).

The only animal that seems to suffer from a long tail is the Homosapien.

So what do we mean by the ‘long tail’?

Simply put, long tail documents in the business world are the low volume of documents from a high volume of senders. This is typically seen in the Accounts Payable and Finance departments, but the premise can also be applied to any type of business transaction requiring documentation.

Having worked with ERP systems for 20 years, I’ve seen the challenges, pitfalls and benefits of getting to grips with the data trail. So, sharing some of these wise old years, I’ve pulled together my thoughts of the different solutions available to address the challenges.

Optical Character Recognition (OCR) – Recognising the limitations

Optical Character Recognition (OCR) was a game changer to the business world. With it’s earliest inventions originating in 1870, it was developed and in widespread application use since the 1960’s, it has helped streamline business processes and, to a point, support automation. Incorporating OCR tech into business process was good addition, however its limitations always meant that other solutions were needed to support the process. The misreading characters, changing document structures and the manual intervention, needed to ensure high levels of data accuracy, have pushed organisations to look for more sophisticated technologies to help automation.

Also, when you think about it, bar coding, loading scanners and correcting mistakes doesn’t really support the digital transformation organisations are looking for or need. It seems like OCR tech needs more manual processes to solve a manual process – odd huh?

Electronic Data Interchange (EDI) – Interchangeable but inflexible

Many organisations turned to Electronic Data Interchange (EDI). This is a much more reliable method of capturing data accurately and at speed. Large files transmit data in an agreed format and allow seamless integration between sender and receiver. Ok, sounds great! But the challenge here is that it needs both sides to commit to a technical and operational strategy and often requires a high financial commitment (relative to the value of the document processed) to set up and maintain. So, EDI is ideal for the highest volume senders, but for the long tail? Most definitely not.

Purchase Order Flip (PO) – PO Flip or PO Flop?

The emergence of portals and offering the supplier the ‘opportunity’ to do PO Flip to create an invoice seems like a perfect option to reduce the Accounts Payable long tail, well in theory. The major challenge here is that suppliers don’t want to re-key information or have to manage multiple portals to raise their invoices, ok this idea might just be a PO Flop.

Although these technologies have helped reduce the long tail slightly, they do not provide the coverage needed. Long tail? Still a problem.

The new tech on the block?

Emerging technologies like Robotic Process Automation (RPA) and Neural networks are technologies that I believe will offer some assistance in this area in the future. Although not new tech, their increasing involvement in the document automation domain has been noted.  

More organisations are exploring RPA with great effect in areas such as sharing data (critical at the moment for the response to Covid-19) and accelerating tasks such as the onboarding of staff. However, for processing documents the projects are falling short of expectations. Many are proving costly or just unreliable as the systems are still reliant on dated technology at the source (OCR) to capture the data on entry. Using RPA to process inbound documents is arguably a problem for the entire supply chain, not just the long tail, and many projects still require manual correction of data. It’s still a problem either way.

Natural Language Processing (NLP) is the technology on which CloudTrade’s service is based. This is a proven subset, having been around for many years, of Artificial Intelligence that enables our service to understand the logic and meaning of a document. Once you understand this the data is available with 100% accuracy regardless of ‘problems’ like data moving on the page.

Additional benefits such as no change in process for the supply chain, deployment within weeks and with no manual processing, mean its clear why there is such a high demand for the service, now more than ever.

Did you know?

The tufted ground squirrel (nicknamed the ‘vampire squirrel’) has a tail that is 130% of its body volume. This is to confuse its predators.

The longtail seems to cause confusion to a lot of solution and service providers, but there is a way to manage it. Does CloudTrade solve the challenge of the long tail? Well simply put, yes. Our core solution, Universal Capture, does process the long tail documents and automate much of the processing with great accuracy. Perhaps not quite as well as the short tail (these can be truly automated with perfect accuracy), but since many of our clients have elected to close their post rooms, get rid of their scanners for documents such as invoices and use our solution instead, I think it’s the closest the world has seen yet.

David Cocks out cycling

CEO David Cocks – ‘CloudTrade continues on the path to growth success’

Reading Time: 4 minutes
Lockdown has got the family and me out cycling!

Despite challenging economic circumstances and the impact on businesses globally, CloudTrade has maintained support levels to meet customer demand and is still set to meet growth targets for FY21. David Cocks, CloudTrade CEO, discusses the challenges, successes and unexpected benefits to come from the global pandemic and subsequent lockdown.

David Cocks, CloudTrade CEO –

The Team

It has now been four months since the UK went into lockdown and we temporarily moved out of our London and Newcastle offices into the atypical situation of complete home working. Fortunately, due to our cloud set-up, all teams were able to quickly settle into home working, with very imaginative work set ups, including bedrooms, kitchens, garden sheds and, for Michael Thomson (Head of Engineering) his under-the-stairs cupboard.

Our team remaining healthy, physically and mentally, has been of the utmost priority and I’m pleased to say that, thank goodness, the team and their families have largely avoided the virus and remained fit and well.

For our customers and partners, the experience of CloudTrade day-to-day has remained unchanged with ‘business-as-usual’ being the overwhelming phrase. Although, our online calls have become a bit more lively and included backgrounds to provoke a conversation (“nice wallpaper” is one I’ve heard) or perhaps the interruption of a pet/child/spouse – which I’d like to think has in fact added variety and the human element to business conversations.

I’m very proud of how our teams have transitioned so smoothly and coped well with the upheaval and unpredictability of the lockdown. As the saying goes, a business is only as good as its people, and that couldn’t be truer than at CloudTrade.     

Transactional volumes – we’re breaking records!

With the well reported economic slump seen during the early spring, it was only natural that CloudTrade saw a speedy decline in volume of business documents processed each day, as trading in general declined. However, these volumes quickly bounced back with May seeing a steady increase and June back to pre-lockdown levels. The dynamic, adaptable, and sustainable nature of CloudTrade’s data capture and extraction software means it responds readily to changes in volume without compromising on the speed of processing or quality of the data capture. The CloudTrade service continues to deliver the data you trust whatever document or how many are processed.

Big news for CloudTrade – July is a record breaker! From our preliminary reports July 2020 is set to be a record month for CloudTrade in terms of the volume of documents process and the number of new customers going live. This is an exciting step in the growth plans for CloudTrade. We are still on course to hit our business growth targets despite a global pandemic and when many businesses in our sector have struggled. This is testament to the hard work of our teams, the fantastic customers we work with and a great product.

The future looks green and blue

In spite of the challenges seen in the last few months, the Development and Operations teams have successfully continued the rollout out of our new, Azure-hosted, auto-scalable-on-demand, containerised architecture. Although processing times have always been speedy, this new rollout has reduced processing times in some cases by over 90%. The inherent fault tolerance and resilience in the design of the containerised architecture now boosts our service when running 365 days a year 24/7 and guarantees processing times even during the absolute peak loads.

Lockdown has also given us opportunities to refocus on our product development. Very excitingly we are developing the latest version of our machine learning algorithms to create ever more sophisticated capture heuristics from large data sets of historical documents, increasing the knowledge learnt from documents therefore enabling speedy and more sophisticated auto-rules writing. We will be demonstrating the beta release with a new user interface around Christmas 2020 – one to watch out for.

Lockdown priorities changed

And finally, as we start to cautiously move back into our offices and make these COVID safe, it is a chance to look back and reflect on what lockdown has meant for our customers, CloudTrade and ourselves. For me, overwhelmingly the lockdown has re-prioritised the need for automated business document processing without the ties to manual processing for many businesses. In times of crisis, the phrase ‘all hands on deck’ springs to mind and everyone rallies to keep a company afloat. What you don’t need is valuable team members being tied to manual processing of invoices etc. or relying on needing a physical OCR scanner in the office (which is closed). We’ve seen some customers appreciate the need for our tech and we’ve then onboarded them quickly (sometimes in two weeks), to support them during the pandemic.

Furthermore, it is a chance to reflect personally on what I’ve experienced during lockdown. Without the commute, my bike rides have certainly been more frequent. I thought you’d all enjoy the picture of my family and me on a recent bike ride around the West Sussex countryside – a beautiful place to enjoy, which I’d thoroughly recommend!

Interested in learning more?

Look out for my upcoming report on the impact of COVID on different industries, looking at the impact on trade in general and the estimated recovery.

David Cocks, CloudTrade CEO and cycling enthusiast

Logistics Technology

That’s enough – the logistics industry needs to change.

Reading Time: 5 minutes

I know, its contentious, but I’ve said it. Logistics as an industry needs to change. I’ve met far too many businesses in this industry that are still using the processes and systems that were put in 20 years ago, that should have been replaced by something less manual. It’s not just the workers or revenue that suffers from repetitive and clunky fulfillment but the customer expects better, especially in an age where you can order something today, and track it all the way to your front door by tomorrow.

I don’t think it’s the big bucks

The reason for the lack of change? Well there’s a few, but it’s certainly not the lack of spend in the market. In 2018 US, companies spent $1.5 trillion on logistical expenses! That’s 8% of the entire US GDP*. In my opinion, the main reason for the lack of change is the concern on implementing modern systems, integrating them with back office processes and the disruption to the supply chain. It’s not like a business can stop for a few days to change systems and then deal with the consequences for up to months on end (I’ve seen this too). But what if there was a solution that could automate your processes to meet and exceed customer expectations, with no change to your processes, no interruption to your supply chain, no corrections needed downstream and with a cost of a few cents a document?

Here’s how you can automate and accelerate your freight invoice processing – a relatively simple but dramatic improvement to processes, using lite touch technology, without business interruption.

Freight invoice processing can be tedious and challenging, and often exacerbated by the dependence on OCR systems or manual keying data from an invoice. The struggle is real – its a labor intensive process and is prone to high error rates when capturing data. Everyone accepts this, recognizes that automating these processes is the key to improve processing times, maintaining customer satisfaction and to removing resource heavy processes from employees daily tasks.

Accurate, efficient and saving your time and money

If my comments in the paragraph above resonate then you should consider Universal Capture from CloudTrade.

We’ve expanded our technology stack and now provide a one-stop solution that automates the capture process for all types of freight invoices – digital files and image files and high or low volumes, with optimized accuracy.

It’s the only solution on the market, with a comprehensive rules engine, that can be tailored to your business’s exact requirements and it meshes perfectly with any ERP or TMS.

Send an email – is that it?

CloudTrade Universal Capture is the leading choice when it comes to capturing data as it removes the entry and engagement barriers of entry to use the solution – we guarantee high supplier adoption. All your customers need to do is send their Freight Invoice to a specific email address which CloudTrade sets up, then in the background we work our magic (well it’s science really, but that’s not quite as exciting), by capturing the data and validating it against your bespoke requirements using our world class rules engine. Then augmenting the data as required and submitting it into your systems automatically. If there is something amiss, its flagged before it enters the system, ensuring your business gets the right data every time 24 hours per day, 365 days per year – often processed and returned to you in minutes.

We know that accuracy is so important to our freight logistics customers because of the cost of rectifying mistakes downstream through the payment or audit process. Delivery failures, cost of returns and correcting volume or load board order errors is costly, and these days, unnecessary expense when the data is easy to capture.

All invoices are welcome!

The struggle for most businesses is trying to apply a ‘one-size fits all solution’ to their system or processes, even though invoices are received in different document types and volumes per supplier. With Universal capture, all invoice types and other documents are welcome, and the solution can be configured to handle a wide range of file types and sizes; when it comes to the processing of the data, we manage single page and multi-page document sets containing both data and images and separate these into single document sets to minimize human intervention and maximize data accuracy.

Universal Capture offers one inbound channel for all types of documents and can validate, sort and process those documents synchronously for organizations to upload into their TMS. Other types of documents are welcome too, from invoices to orders to carrier receipts to the large PDF invoice sets sent out by the larger carriers (we regularly process a 1500-page document into separate shipments for one of our customers).

But how is that actually possible?

So, we’ll let you in to our secret box of tricks. To get the job done, Universal Capture works by being versatile when processing data from different files. With high supplier invoice volumes, CloudTrade can guarantee 100% data accuracy by setting up a standard set of logic rules for data capture for that supplier. The data capture then follows the same rules for each invoice transmitted through CloudTrade, these rules are managed by CloudTrade support to maintain completely accurate data capture, even if the invoice design were to change.

Image files, while less commonly used, are still an important part of data capture and something that Universal Capture can process. Despite image files not containing a data layer (so characters are interpreted rather than extracted), CloudTrade can still offer accuracy levels of around 90%, as we use the same rules technology to check what we have captured – at the time of capture. If we can’t validate the captured data against your requirements, we will send it to our intervention portal where you can fix the problem and resubmit it for processing. Just to be clear some manual correction may be required with image files, but we can help with that to. We can set automated messaging encouraging the suppliers to submit digital files to help improve the accuracy of their data capture and we have found that most will do this if they can get their invoice paid faster!

Finally, for suppliers with low volumes of inbound invoices and documents, CloudTrade Universal Capture can use its learned experience from other invoice types to extract a generic set of the data. Once again, this will eliminate most of the human intervention that is typically required when it comes to processing documents.

Document automation during COVID

We know that these are uncertain times for businesses, but one thing is clear, the trucks will keep on rolling and goods still need to be moved quickly and reliably. In the midst of the current pandemic, human intervention in these processes has, for some organizations, been a challenge and in some cases, it was just not possible with a home-based workforce. The good news is that we were able to help. Our technology is delivered as SaaS, accessed via the web and our portal is accessible from all well-known browsers. We have been able to offer an enhanced service to our clients for their home based workers that guaranteed high capture rates and access to our intervention screens for any data capture or mapping issues – now if only you had known that 3 months ago!

With Universal Capture from CloudTrade, you could use just one solution for all inbound freight and other documents, with implementation in as little as two weeks. Within 15 mins, you can find out how we can help your business and we’ll prove our solution works.

Go on, book in 15 mins with us, it could be the first step to an automation reality.

The blog post was originally published on our Logistics specialist website, cloudtradelogistics.com

*Flock Freight (2018) For the Love of Logistics

RPA Technology

Robots don’t make mistakes – but data does!

Reading Time: 3 minutes
RPA Technology
RPA bots, don’t make mistakes if the instructions are correct.

There has been a huge amount written about the benefits of Robotic Process Automation (RPA) and probably as many column inches dedicated to the challenges and pitfalls. In this article and our upcoming webinar, we explore the role that data plays in all RPA projects and the impact that bad data has on the robots and the desired business outcome.

Whatever industry you work in, or in whichever interest you may have, you will almost certainly have come across a story about how “data” is changing the face of our world, particularly “big data”. You may have heard this term as part of a study helping to cure a disease, boost a company’s revenue, improve customer service, make a building more efficient or be responsible for those targeted ads we keep seeing.

But we don’t mean THAT “data”!

Despite what term is commonly used, data is simply another word for information. But in computing and business, data refers to information that is machine-readable as opposed to human-readable.

In business, we receive masses of data in human readable form such as contracts, invoices, orders  or HR records etc. These documents need to be converted to a machine-readable form so that technology, like RPA, can be used to automate the process end-to-end.

The challenge is to firstly have the creator of the document produce it in a digital format that is also human readable, so that further downstream this can be read, data extracted and passed to a robotic process for downstream automation. Data extraction can be achieved at 100% accuracy if produced in a digital format (if the format contains a text layer).

Images causing havoc

But, where the sender chooses to create an image file, you must rely on Optical Character Recognition (OCR) to convert the text to a machine-readable format. The problem with OCR is that as the receiver has no control over the image quality or how data is presented, the net result is you can never guarantee accuracy and it’s these data errors that cause havoc with the RPA process.

Ensuring the best data for your robots

To make sure your bots do not go awry, the first challenge is getting the sender to create a digital document. To do this, we need to remove any barriers, ensure there is no cost or resource requirement and ideally no process change for the sender. The second challenge is to remove paper or image files that require OCR.

Bad data, big problems

Let’s consider the consequences of bad data for a minute. The impact of misreading a measurement or value could mean an engine part is manufactured to the incorrect size or an order gets processed with the wrong amount, a -10 becomes 100 and so on. Data without context delivers a second layer of complexity, as ‘ea’ could be read as ‘each box’ and not ‘each unit’ etc. There is a clear and obvious need to not only read data accurately but also to understand the context of a data element.

Now consider these challenges at scale and the impact of such errors on ‘big data’ as more of the world’s business processes become digital and move online, the need to process data at scale accurately has never been more important.

RPA for business process automation

In the world of shared services, we have looked to deploy RPA in areas such as invoice and order processing to increase automation and drive efficiencies. Through the implementation of innovative technologies, such as RPA, the human task is rapidly moving from the mundane and repetitive to those of quality control and cognitive value creation. The theory is great, but the reality is that unless the right technology and business process is deployed to convert human readable documents to that of a machine readable format, the data for the RPA bots will always contain errors. You can read more about RPA integration and CloudTrade here.

Technology for data perfection

There is a solution to read digital documents and process that data into a format a machine can read to give bots the right tools for the job.

We’re running a webinar focusing on this integration for RPA, sign up is available here and will address how this proven approach works for RPA , provide a live demonstration of delivering 100% accurate data, and how to automate business processes that will eliminate human intervention.

Microservices

Our journey from Monolith to Microservices

Reading Time: 5 minutes

Richard Hooper, Head of Systems, explains how CloudTrade upgraded its software environment to cope with increased demand and some of the problems solved along the way.

Just over a year ago at CloudTrade, we made the jump and decided that containers (using Kubernetes) were the answers to all our application issues. In this article I will examine why we have chosen to jump on the container band wagon, which could be termed as the ‘latest tech craze’, as well as how we solved some of the issues along the way, but firstly, a little about me.

About me

I’m Richard Hooper, Head of Systems and a Microsoft MVP in Azure. I started with CloudTrade back in March 2018 as a Systems Architect. As CloudTrade grew so did my responsibilities, and now I manage a team that look after the internal servers as well as the desktop, Azure estate, and the whole production estate.

My passion lies in all thing’s technology based and specially Microsoft Azure. In my spare time I blog about Azure at Https://pixelrobots.co.uk and can be found hosting the North East Azure User Group.

Was a container system the right thing to do?

It’s a question I ask myself often. With the rate of change in the cloud world you kind of have to keep questioning and evaluating, as a new technology comes out almost monthly, well it seems to anyway. Every time I ask myself, I always come to the conclusion of, yes. However, as we became more familiar with microservices and what we need from our application, I know we made the right choice.

Why microservices?

The application that powers CloudTrade’s unique data acquisition technology, Gramatica, started life as a sort of Desktop application. It needed the user to be logged in and wrote a lot of files onto the server or desktop. One good thing is that when the application was first created, it was created with steps and each step had a sort of handover using files. When I found out about this, it was a relief as it should make the move to microservices easier.

Why change then, I hear you ask! Well for a start the management of the server and application became difficult, especially if you wanted to do any kind of automated patches and, I certainly did not want to keep patching servers out of hours. But the main driving force for the move was scalability – the dream for a software business.

With the way the application was created, and all the file access, at the time, scaling was a right pain! First you had to run more copies of the application per user if there were enough free resources on the server or spin up a new server and migrate the user and application to it. Sometimes we would also hit disk issues, capacity and IOPS.

With the move to Kubernetes, an open-source container-orchestration system, and more specifically Azure Kubernetes Services (AKS) this headache has gone away. Our AKS cluster utilises something called Virtual Machine Scale Sets (VMSS) which allows for the cluster to auto scale it’s nodes when resources are becoming constrained, all done automatically. Another great feature with Kubernetes is the way it can automatically scale your deployments (a deployment is a collection of pods, a pod is a wrapper for containers in Kubernetes). How awesome is that?! But all this awesomeness still came with issues, issues that we had to get over to make this journey a true success.

Oh no, not issues!

Yes, with any journey you are always going to have hurdles along the way and this one is no different. One of our main issues, is that part of our new microservices application needs to be run in Windows containers. This was the problem we tried to fix first – some may say that was a mistake as Kubernetes did not support Windows containers at the time, but Docker did!

To get round this issue, we are currently running the microservice on Windows server 2019 in a VMSS using a custom hardened image. We currently run 6 containers per node, 1 for configuration and 5 for actual processing.

Scaling became a bit of an issue as we moved more onto this new microservice. As we are now using RabbitMQ instead of the file system, we came up with a brilliant solution of using an Azure Logic App to query the RabbitMQ cluster, which is running inside our AKS cluster, every 15 minutes. It checks the queue size and how many containers are consuming the queue and will then either scale up or down the VMSS nodes. Unfortunately, we had to choose 15 minutes for the check as the nodes can take a while to come up.

We are currently rewriting this application to run in Linux, so my tip is if you can get away with not running Windows containers then do it!

As we are using RabbitMQ, to scale our microservices that run inside the AKS cluster, we were unable to utilise any of the basic container autoscaling that comes with Kubernetes. After some research we came across Keda, which is an open source project by Microsoft and Red Hat. Keda extends the basic container autoscaling and allows us to scale based on RabbitMQ queue size and quicker than the logic app approach we used above. We were quite lucky that Keda went GA just in time for us to release the second phase of containers.  

What’s next?

We are continuing our journey with the next phases being worked on. We hope to get the release into production by the second half of this year. Once each step has been finished, we will end up with what we are calling a skeleton of our old application which will still be running on the servers. There will need to be some time spent to remove these to complete our journey as we are envisioning that there will be no need for any servers apart from the AKS nodes.

We will also continue with another journey. This one is to utilise tools like GitHub Actions and Azure DevOps which will help to automatically build and release each microservice to our test and then production AKS cluster. This will enable us to fully embrace the ‘DevOps mentality’ by not only improving internal processes, but also improving the application.

Feel free to reach out if you would like to discuss any of the above – thanks for reading!

CloudTrade specialises in converting documents (with 100% accuracy)

so humans can read them.

Learn more about CloudTrade and our technology here.

It may not be rocket science, but it can be complex

Reading Time: 3 minutes

Reading documents may not be rocket science, but computers struggle to do what humans find simple. Is technology finally able to read documents in the same way as humans?

CloudTrade are in the business of extracting and interpreting information out of documents which have been written can be understood not by people, not computers.

This is probably one of the most frustrating problems in the history of IT.

Reading stuff out of documents feels easy to us, as people. Nowadays anything to do with people communicating to other people feels easy, and we ultimately think that since computers are cleverer than we are (in many ways), that if a person finds a task easy, then a computer should find this no trouble at all.

The problem is: we tend to forget just how clever people are. Even if you struggle with long division, that brain of yours which controls everything from getting out of bed in the morning, to washing, driving to work, eating lunch, watching TV and so on, leaves the most powerful computers in the world floundering at the starting pole like electronic tortoises.

Communicating with other people, in speech or in writing, falls into that category of stuff that your brain is very good at but computers struggle to do. People get a lot of practice at it. No computer in the world could have read what you’ve read so far and have any idea of what I’m talking about, but you’ve understood me completely (well I hope so!).

CloudTrade aren’t in the world of building robots, of course, not even robotic tortoises. Neither are we trying to write a full natural language processor which could understand everything that a human being might want to say to it. These sorts of achievements are truly well within the realms of science fiction. However, what we have built at CloudTrade, is a natural language processing engine which can understand those documents which we have programmed it to understand. This is much more sophisticated than the approaches which are otherwise prevalent in the marketplace.

For example, just hoping that a particular bit of information on a document (for instance, a VAT number) might always be found on the same place on a page, just isn’t going to work. Neither will the idea that you might be able to go hunting for some unique piece of text and then look in some predetermined distance and direction to find what you’re after. These sorts of techniques work occasionally, but most of the time pages jiggle around and the chances of being able to find something which is not only guaranteed to be unique, but also always in the same location relative to what you’re looking for is tiny.

We frequently get people coming to us after they’ve tried these sorts of solutions and then given up in frustration and I sympathise with this scenario. Often, they thought that the problem they had was an easy one so they bought into an easy solution, more often or not wrapped up with some sort of neural network element, which then proves unhelpful. They’ve then discovered that this easy solution didn’t work and that they had to spend all of their time filling in for its mistakes, or being told that they had “yet another” special case which would require costly scripting or programming.

CloudTrade are simply not like this.

Ok, I know anyone can make that sort of claim, but I like to think that we put our money where our mouth is by offering our solution as a full service, rather than as a software licence where you may be left to find out for yourself whether the solution works effectively or not. We configure it to fit your requirements and when it’s up and running we correct its mistakes and maintain it for as long as you stay with us. Furthermore, we’ll charge you the same price for every document we handle, no matter how awkward or complicated it may be.

We’re the only company prepared to do this because we know, ultimately, that we’ve built the right solution. It may not be rocket science, but it’s actually pretty clever, and it turns out that you need to be pretty clever if you want to solve this problem.

CloudTrade specialises in converting documents (with 100% accuracy)

so humans can read them.

Learn more about CloudTrade and our technology here.

Self isolation? No problem – keep your business running 24/7

Reading Time: 4 minutes
Remote Working

With the world slowly putting itself into self-isolation, never has it been more evident that digitisation of your business practices enables increased flexibility in where your team works and helping to keep it ‘open as usual’. Here, David Cocks, CloudTrade CEO, discusses how the move towards digitisation and automation can not only help keep your business running during times of crisis such as these, but can also benefit your business in the long run.

In the modern world, customers have come to expect services 24/7. Companies like Amazon have set the bar very high when it feels like that drone delivering the next brown parcel is already overhead before you have even clicked “proceed to checkout”. However, not every company has the luxury of scale to ensure they always have staff ready to complete the onsite business processes, whatever the peaks in demand.

Some companies now incorporate flexible working strategies to encourage home working where it fits with the business needs, but many tasks remain dependent on the physical presence of staff within the business office. This often includes the need for access to paper documents or the use of on-premise technology, be that computers, printers or even scanners.

In these difficult times, we are also forced to accept the real possibility of large numbers of staff not being able to travel to the physical business office through actual illness or community health protection necessities. The more we modernise our business so our staff can work from anywhere, the more we are resilient to the threats of massive disruption.

The state-of-the-art way to ensure your business meets scalability and business continuity demands is to automate fully the critical processes. Persisting with manual, repetitive tasks in the spirit of “it ain’t broke, why fix it” will not work when faced with a global crisis that is emptying streets, offices and leaves the commuter train ghost-like. Also, your customers will not tolerate whatever reasons  as to why you’re unable to deliver the goods and services promised, especially if these business critical – they will look to others who have identified the risks of reliance on outdate manual processes and have taken actions to mitigate these issues.

What do I mean by ‘automating the critical processes’?

Firstly, look at your internal processes and understand why things are done manually. This is often because of external factors, notably data not available in a form or quality that is consistent with the requirements for straight through processing (STP).

Just think of a sales order arriving by email. The customer service team have to open the email, save the attachment, scan the PDF, correct the OCR (scanned data) and finally upload the data into the fulfilment system. One person can process maybe 10 orders an hour – max! If staff are not available or not on-site, then there are no orders in your system, unhappy customers and loss of revenue. All you really need is the correct data in a machine-readable form without the reliance of scanning equipment, and you can go straight through to fulfilment. Your order is shipped in minutes, and not days.

Second is to use the right technology, designed to automate and deliver touch-free processing. I don’t mean systems designed to make a manual process more efficient, where you still need elusive staff to complete routine tasks. Systems that are designed to achieve guaranteed high levels of automation and accuracy are essential for the true, scalable, high-volume straight-through process. A system that can guarantee quality must be deterministic. That is, it must be clear how it works, why it works and (probably just as important) when it fails, why it has failed. If your business systems are non-deterministic, the output cannot be predicted and you don’t know how it works, then you can’t drive for full automation. Perhaps you already have an invoice scanning/OCR service – think how annoying it is when sometimes it gets the data correct, and other times it corrupts or misses information that appears clearly on the original document. The result is you need to check manually each document. It is only when the mechanisms of automation are transparent that you can achieve continuous improvement. A system that is best endeavours and (maybe) improves in an obscure way can never deliver the straight-through processes you should strive for.

So, business leaders think automation and think STP. Modernising your business is not vanity, or even just a way of improving margins, it is a necessity to survival.

I wouldn’t want you to be the next Kodak, Blockbuster, or more recently, Thomas Cook.

CloudTrade specialises in automating traditionally manual business processes

such as invoice processing.

Want to know more about electronic invoicing and the different methods available?

CloudTrade – a woman’s world?

Reading Time: 3 minutes
Gender Equality in Technology

Well, not quite yet… But we’re getting there! Here Amee Patel, Operations Manager, discusses some of the challenges of being a woman in a male-dominated industry and how CloudTrade has changed our practices to encourage female applicants for job openings.

Being a woman in a typically male-dominated environment, like tech, can be a tough gig, especially if you are the first female to join the technical team, as I was at CloudTrade a few years ago. However, I can safely say that I have survived (and now thrive!) in an environment with roles mainly filled by men and am continuing to show women that roles in tech are not scary places, and they can fit in here!

I began my career in 2013 on an IT helpdesk – where I was affectionately known as “Helpdesk Girl”, (the name wasn’t quite that nice, but you get the idea) which accurately summed up my duties. Two years after carrying this mantle, the glitz and glamour of commuting into London became too much of a temptation and I started to apply to join tech companies in the big city. When I interviewed at CloudTrade, one of the first questions I was asked was: “You will be the only woman in the company. Is that okay?”. I was taken aback, – I come from a technical background and I spent three years studying a male dominated field at university! Of course it was okay, I thought, and ultimately it had to be okay!

Yet during my first week at CloudTrade, the imposter syndrome set in. What am I doing here?! This is far too technical for me! The men in the team are much better than me… I will never be successful here, and so on… But I stuck at it and I worked hard. I made it my business to become a master of my trade. I was supported by management and I felt like I had found somewhere that I could succeed and was not made to feel inferior to my male colleagues.

As CloudTrade is a small tech company that continues to grow, we often recruit to fill new positions. Six months into my employment, CloudTrade employed its second woman to fill a marketing position. Eighteen months after that, we employed our third into a technical role. Within another six months, I was lucky enough to move into a management role within our Operations team, and within that time we recruited another three women. CloudTrade went from no women to six, but this took almost two years – recruiting women into tech roles is just not that easy, and whenever we tried to recruit, we saw a huge imbalance in the gender of those applying.

This disparity of genders did not surprise me. Being a woman in a tech company is daunting. Imposter syndrome is real. Feeling like you need to work hard to prove your credibility and gain recognition is extremely common, and it doesn’t feel very fair. These sentiments are echoed with facts. Women are less likely than men to study STEM subjects, and even less likely to pursue careers in tech.

While CloudTrade actively tried to recruit women to balance the gender divide, the wider sector also saw more global initiatives to support women, which were gaining momentum. We recognised this wasn’t just happening at CloudTrade, it was happening everywhere, and was being brought to the forefront of people’s attention. Here at CloudTrade, to help redress the balance we reviewed our recruitment process across the various departments – the adverts we were producing, the perception of the company from potential employees, and the profile of the people we were targeting. This piece of work saw a change in the candidates we were getting. Suddenly, we were getting applications from women – talented, qualified, ambitious women!

Today, as we approach International Women’s Day, I write this blog post as one of twelve women in a company of forty-four. I am fortunate enough to work closely with these women, who all bring something different to the table in their various roles. It is never easy as a woman to walk into a tech company and not feel a sense of “I don’t belong here”, which is why I feel so proud of CloudTrade’s journey over the last five years, and so grateful for the personal and professional growth I, as a woman, am offered here.