By Petra Beck, Senior Analyst, Software Practice at Infosource
Later this year, Infosource will deliver an updated quantitative assessment for the Capture and Intelligent Document Processing (IDP) market. But in the meantime, I have taken a qualitative look at the major trends expected to shape IDP market dynamics in 2024 and beyond. Generative artificial intelligence (AI) is the mega-trend behind most of the other trends we observe in IDP.
Here are my 10 predictions for the year to come:
1. ADVANCED AI TECHNOLOGY WILL ACCELERATE IDP DEMAND IN 2024 AND BEYOND
Increasing operating costs and a shortage of knowledge workers, along with increasing expectations for fast and seamless B2C and B2B transactions, is driving demand for process automation solutions. Advanced Capture and IDP solutions are a critical component as they allow enterprises to automate the handling of business inputs.
Their value (return on investment) increases with integration into downstream systems, ideally allowing the end-to-end automation of an entire business process. The increased demand is being addressed by IDP solutions that are easier to deploy and scale. In addition to cloud offerings and low-code architectures, higher adoption of advanced AI technologies makes IDP solutions increasingly attractive.
2024 will therefore mark accelerated market demand as advanced AI technologies like large language models (LLMs) become more mature, vendors complete their testing and piloting, and end customers show willingness to upgrade or embrace the latest IDP solution updates. We forecast higher demand for Capture & IDP solutions, based on these factors.
2. ADVANCED AI TECHNOLOGY WILL ACCELERATE IDP DEMAND IN 2024 AND BEYOND
AI is a polarizing market dynamic. Some consumers and knowledge workers are concerned about the potential risks and the black box nature of AI based solutions. Others acknowledge the benefits related to the reduction of manual tasks and enhanced customer and employee experience and are excited about the step change in innovation.
There is consensus about the importance of legislation related to AI, as demonstrated in the G7 leaders’ agreement on Guiding Principles and a Code of Conduct on AI. However, the approach varies significantly across different parts of the world. In geographies like Southeast Asia, support for innovation is the priority as visible in the framework for AI governance recently agreed by the ASEAN member states.
Other nations start with a directive for their federal government agencies like the Executive Order of the US president. There are also material differences in the implementation and enforcement of the different legislations ranging from recommendations to mandates associated with high fines like in the EU AI Act, also recently finalized.
All approaches foster valuable exchanges between politicians, software developers and organizations, which increase transparency and allow business users to understand the trade-off in risks and value. Legislators will need to race to keep up with rapidly emerging AI developments. AI legislation will have a positive impact on demand for AI based solutions and development of advanced AI tech in IDP solutions.
3. Sustainability is becoming an important dimension in purchase decisions
Climate change is one of the major macro trends with a far-reaching impact on the Capture and IDP market. The carbon footprint is a major decision criterion in businesses and organizations, particularly in Western Europe. When purchasing office equipment, energy consumption is a key factor. The public sector in particular is interested in refurbished machines.
When an organization automates or optimizes a business process, the environmental impact is increasingly one of the considerations. There are a multitude of aspects to be considered in information-intensive processes, ranging from the transportation and digitization of paper-based inputs to the computing power involved in the processing and the storage of documents and data.
Government organizations need to provide guidance on the environmental impact of the different approaches for Information Capture and Management. Vendors need to be ready to respond to questions from their clients and prospects when assessing solutions and strategies, not only when they involve the use of LLMs.
I am presenting on the topic at the 2024 AIIM conference.
4. Generative AI fosters the convergence of IDP and CEM
Providing consistently excellent customer experience has never been more important. But hurdles are increasing on the supplier side: a shortage of staff, increased competition, and pressure to protect margins and profitability. If meeting increasing and changing customer expectations has been an uphill battle in the past, the hill is becoming steeper yet.
In this context, the discussion about the convergence between solutions focused on Intelligent Information Capture and those designed with focus on Customer Experience Management (CEM) has been reignited. The idea has been talked about for decades, but it has remained largely conceptual rather than practical.
Some parameters are changing in addition to the challenges described earlier. The expectations for customized and personalized offers and customer service are critical when selecting business partners in the B2C market. There is an increasing demand for end-to-end automation in information intensive processes requiring communication and feedback loops to the end customer.
Generative AI solutions offer capabilities to enhance IDP solutions to meet these requirements. When integrated with IDP solutions and CEM solutions, generative AI can build bridges between the two market segments. In 2024, we will finally see the convergence come to life, a trend that will intensify in the coming years.
5. Traditional Capture Software vendors focused on data extraction will become obsolete
In our Infosource Capture & IDP market assessment, we track the full spectrum of vendors with Capture and IDP capabilities. Those range from vendors who offer software with rudimentary capabilities for creating metadata to capture solutions that are fully integrated into an end-to-end process automation solution.
As part of my market forecast effort, I have illustrated the evolution of Capture and IDP solutions in a multi-step visual. After the first step from completely manual to partially automated, this becomes a continuum. Vendors continue to expand their capabilities as they launch new releases or platforms. And those with modern software architectures can easily add services from other vendors.
Vendors who are focused on Basic Capture have been slower to augment their software packages, or they have focused on optimizing their scan and index capabilities, including output options. Basic Capture vendors risk obsolescence as automation and AI capabilities become front and center in IDP purchasing decisions.
6. The second generation of LLMs will be use case-focused
The introduction of LLMs has been a milestone in the development of AI. For information intensive processes, they bring impressive capabilities for streamlining time-consuming tasks like document classification, summarization and translation. But they have limited proficiency in handling processes that are vertical or use case specific and involve domain specific terminology or legal requirements.
IDP vendors, by contrast, have built proprietary models that are optimized for specific use cases and come equipped with machine learning capabilities. Over the past year, many have augmented their models with LLM integrations to expand their own capabilities – with results differing based on the use case and specific requirements.
In light of the known limitations of LLMs for more demanding business use cases, domain specific LLMs have started to emerge. Verticals with specific terminology and requirements are Healthcare and Legal. Here examples are Google’s MedLM offer and a variety of LLMs focused on the legal segment including Harvey.ai that is built on OpenAI’s technology.
I expect the next wave of LLM developments to offer vertical or domain specific models as well as to address the need to augment the general LLMs with use case-specific and organization-specific knowledge bases. This will be a double-edged sword for IDP vendors, as they will be faced with a decision about whether to augment or replace some of their own technology accomplishments.
7. Gen AI specialists emerge as a new breed of outsourcing provider
Service bureaus and outsourcing providers play a vital role in the IDP market as an alternative to in-house IDP operations. They range from traditional scan bureaus that hand over scanned images and meta data to end customers to outsourcing providers who take over entire business processes for a business or public organization.
We expect the market demand to expand from basic capture services to the outsourcing of process elements or entire business processes. This will change the mix of service providers significantly in the next few years. Service bureaus that only provide basic capture services will increasingly struggle and may become acquisition targets if their customer base or assets are attractive for other vendors.
Global business services (GBS) and business process outsourcing (BPO) providers that offer IDP based services and take on entire business processes will support end-customers’ desire for end-to-end automation. These BPO and GBS service providers are interested in advanced AI technologies to support their automation offers.
They have a window of opportunity to support businesses and public organizations that see the value of generative AI solutions but lack the knowledge and expertise to deploy them. This also opens an opportunity for gen AI startups to partner with outsourcing providers or establish their own service operations.
8. IDP, RPA and BPM combine to provide end-to-end automation
We consider the North Star for the IDP market to be an integrated solution for the end-to-end automation of a business process. By end-to-end processing, we mean the fully automated processing of a business transaction from the time it is triggered by an external business input, for example, an offer followed by a purchase order through to the receipt of an invoice and the final payment.
IDP vendors are increasingly augmenting their solutions with capabilities to automate process steps triggered by the extracted data like validation and augmentation. They are using proprietary technologies or are integrating robotic process automation (RPA) or adaptive business process management (BPM) solution elements from their own portfolio or a partner offering.
RPA vendors have also realized the need to process unstructured business inputs, which they are covering with their own IDP development or partnerships with Capture or IDP vendors. We see a growing demand for these solutions with increasing efforts from vendors to move up the maturity ladder.
9. With Generative AI, search and retrieval is becoming a new game
Generative AI can significantly enhance the search and retrieval functionality of a database or even a Document Management System. Its understanding of the query context will should enhance the accuracy of the search results. Natural language prompts, meanwhile, make it easier for users to classify documents, extract key data, obtain deeper insights or summarize documents.
The tedious task of finding important business information through basic searches has become quicker and easier. But not all documents are neatly stored in databases or content management systems. Many unstructured documents are buried in file shares or repositories without relevant metadata. I refer to these files as “dark data.”
They may contain important business information, which cannot be retrieved; they also cannot be used for analytics and may even become a liability if they contain data governed by legislation. Generative AI tools will help to address these challenges and bring some light into the dark. We will see an impact in these use cases in 2024.
10. The digital divide continues to widen
My final prediction for the IDP market in 2024 is that the digital divide will continue to widen. This divide refers to the unequal access to digital technology and its resulting impact on parts of the population. Lack of internet access and basic digital skills are just some of the factors contributing to this issue, particularly in rural areas.
Closing this gap presents opportunities to increase the living standards of disadvantaged countries through solutions like telemedicine and remote education. Organizations like United Nations and initiatives such as the Affordable Connectivity Program in the US are striving to close the digital divide with universal connectivity.
However, the acceleration of digitalization and automation efforts boosted by AI technology may cause the digital divide to widen again. Nations, industries, and regions that are already advanced will accelerate their digital investments. Others, who lack the infrastructure or basic skills and struggle with limited budgets, are much slower in their innovation efforts and will hence fall further behind.
