“We really expected a lot more with all the promise from the software—not sure where we failed.”
I attended a round-table where multiple executives were discussing their technology deployments and none were too pleased. In a few cases, these software deployments were underutilized, while in others the journey to the promised land, seemed always a few months and upgrades away. At a bank, years after deployment, efficiency gains were marginal and user adoption limited. When I saw the executive’s exasperation, I could visualize a bunch of users feeling quite helpless and their CFO’s furrowed brow.
But it isn’t all doom and gloom! What separates the happy few from the frustrated majority? Some of them (thankfully, my customers included!) were quite content with their solution deployments and spoke about a framework comprising a clear understanding of technology capability, process alignment and structured user training.
If we are able to take a broader perspective- from the wheel to the web, technologies that thrive are those built with a sense of empathy for users, with a seamless integration into their work life.
Navigating Technological Transitions
As we look back through history, it is quite fascinating to see how various forms of technology have shaped societies and economies. The introduction of gunpowder marked a pivotal shift in military strategy, transforming brute-force battles into exercises of precision and expertise. Similarly, the rise of steam power transitioned craftsmanship from family-centric workshops to the efficiency of mass production, as articulated in Adam Smith’s principles of economic efficiency.
These historical shifts underscore the importance of planning in harnessing the full potential of new technologies. These shifts would certainly have entailed multiple iterations of trial-and-error to determine the optimal model. When considering the choice of innovation areas, essential risk management controls, and training for adapting to the new approach, it is crucial to carefully evaluate each aspect. As we move this way of thinking to modern technology and specifically Artificial Intelligence, the same elements around identifying the right use cases and determining applicability remain as relevant. The only difference would be that by automating cognitive tasks and by ‘learning to learn’, this could more significantly redefine the landscape of professional services.
Recent research in the legal domain provides very valuable data around how this innovation impacts complex domains requiring higher order thinking– in terms of determining optimal areas of application, benefits to different user segments and having a plan for seamless integration.
This research by Choi, Monahan, and Schwarcz of the University of Minnesota Law School, provides a comprehensive examination of AI’s integration into the legal analysis field. Their study meticulously explores how AI tools are being adopted in legal practices, shedding light on the nuanced ways these technologies are reshaping professional tasks.
Overview of Research findings in Legal Analysis
The research conducted the first randomized controlled trial to study the effect of AI assistance on human legal analysis. Sixty students from the University of Minnesota Law School were selected and imparted training on using an AI tool to help them perform the allotted tasks. The selected tool chosen was GPT4- but presume the result would not change significantly if the tool selected were different. The students were assigned realistic legal tasks at random — both with and without the assistance of AI.
The four assigned tasks are common tasks handled by fresh legal professionals.
- Drafting a complaint,
- Contract drafting,
- Employee handbook,
- Client memo
The efforts taken by the students were tracked and the output graded.
Key findings
Here are the key outcomes.Outcomes- Quality improvements (subjective grade scores)| Tasks | Average Quality (without AI) | Average Quality (with AI) | |
| 1 | Drafting a complaint | 3.14 | 3.31 |
| 2 | Contract drafting | 3 | 3.24 |
| 3 | Employee handbook | 3.20 | 3.26 |
| 4 | Client memo | 2.92 | 2.85 |
| Tasks | Average Time (without AI) | Average Time (with AI) | Effort Savings | |
| 1 | Drafting a complaint | 160.7 | 122 | 24.1% |
| 2 | Contract drafting | 69.7 | 47.6 | 32.1% |
| 3 | Employee handbook | 37.2 | 29.4 | 21.1% |
| 4 | Client memo | 244.4 | 215.6 | 11.8% |
- As seen, access to AI caused little average improvement on the quality of output in lawyering tasks but a substantial increase in speed of completion.
- On Quality, it was noticed that the improvements were primarily for the poorer performers- allowing for a compression in the performance curve. Participants who were performing poorly, without support from AI benefited the most in terms of quality- while individuals with good performance without AI, didn’t see material benefits.
- Improvement in speed was largely consistent among participants. Again, certain tasks delivered greater performance improvements.
- Participants indicated that their ability to use AI improved over the course of the experiment. They had a reasonably accurate view of areas where AI benefited them, vis-à-vis other tasks.
- Most importantly, participants had positive view of AI and would prefer to use them across user profiles.
Key Takeaways
I was thinking around this research with the context of various customer situations demanding benefits inclusive of turnaround time, agility, quality, and productivity. This reinforced the following views for me:
- The benefits of AI are not uniform across all tasks. Each task might see a different kind of value generated across these themes- and this could also vary based on the expertise of the agent. To build meaningful adoption and develop a business case, it is important to identify the use cases, and determine how the processes and tools are adapted to achieve the best benefits.
A key element in emerging AI domains, is to carefully calibrate areas where we automate, and those where we augment with a human-in-loop layer to ensure process control. As an example, for a leading insurance player, we initially started with 100% manual check to build confidence. We then gradually shifted from human-validation to automation in batches, and finally reached a stage where manual checks were done only for exceptions. - AI could operate in two modes.
- Level the playing field by providing a very valuable ‘expert assistance’ service, offering greater advantages to less experienced professionals than to seasoned experts.
- Enable power users to innovate on the capability, to raise productivity.
This requires a better understanding of the specific task, and careful design of the process to achieve either or both objectives. Don’t get me wrong- while artificial intelligence can be quite potent, it does need traditional time-tested human intelligence to create magic.
- Adoption needs empathy with users and successful implementations need to ensure that required change management and training is conducted. It is important to address fears and uncertainties during this change- to ensure that the transition is smooth, and there is transparency on objectives. As an example, for a leading financial service institution, we ensured that business users were leading product usage, not only in identifying use cases, but also the configuration and deployment. This helped achieve greater buy-in and improved innovation.
- The overall process will need to be looked at holistically, to impact end-objectives and not just shift the bottleneck further down the process. It is quite tempting to deploying AI to resolve problems deemed ‘easy-wins’ – but value is not realized. This was likely to be a problem with some of the executives at the round table, and I do hope that they have been able to get their issues resolved.
Leveraging AI for Enhanced Organizational Performance
Unleashing the AI beast (for good, of course!) is a combination of creating a deeper awareness of the technology coupled with a new mindset of curiosity and experimentation. It calls for a combination of strategic planning to align business objectives with process & technology, and then moving with a sense of urgency. Individuals and enterprises that navigate this journey are best positioned to harness the power of AI and build up unique business advantages. These are indeed exciting times!
