According to recent estimates, generative AI is expected to become a $1.3 trillion market by 2032 as more and more companies are starting to embrace AI and custom LLM software development. However, there are certain technical challenges that create significant obstacles of AI/LLM implementation. Building fast, robust, and powerful AI-driven apps is a complex task, especially if you lack prior experience.
In this article, we will focus on common challenges in AI adoption, discuss the technical side of the question, and provide tips on how to overcome these problems to build tailored AI-powered solutions.
Common AI Adoption Challenges
We will primarily focus on the wrapper approach, which implies layering AI features on top of existing systems instead of deeply integrating AI into the core. In such cases, most AI products and features are built as wrappers over existing models, such as ChatGPT, called by the app through the OpenAI API. Its incredible simplicity is the most attractive feature about such an approach, making it very popular among companies aiming for AI transformation. You simply explain your problem and the desired solution in natural language and get the result: natural language in, natural language out. But this approach has several drawbacks. Here’s why you should consider different strategies and ways of implementing them efficiently.
const response = await getCompletionFromGPT(prompt)
Lack of differentiation
It may be challenging to differentiate a product in the rapidly evolving domain of AI-powered software. For example, if one person creates a QA tool with an uploaded PDF document, many others will soon do the same. Eventually, even OpenAI might integrate that feature directly into their chat (as they have already done). Such products rely on simple techniques using existing models that anyone can replicate quickly. If your product’s unique value proposition hinges on advanced AI technology that can be easily copied, you’re in a risky position.
High costs
Large language models (LLMs) are versatile but costly. They are designed to handle a wide range of tasks, but this versatility makes them large and complex, increasing operational costs. Let’s estimate: Suppose users upload 10 documents per day, each with 10 pages (500 words per page on average), and the summary is 1 page. Using GPT-4 32k models to summarize this content would cost about $143.64 per user per month. This includes $119.70 for processing input tokens and $23.94 for generating output tokens, with token prices at $0.06 per 1,000 input tokens and $0.12 per 1,000 output tokens. Most cases don’t require a model trained on the entire Internet, as such a solution is, typically, inefficient and costly.
Performance issues
LLMs are mostly slow in comparison to regular algorithms. The point is that they require massive computational resources to process and generate text, involving billions of parameters and complex transformer-based architectures.
While slower model performance might be acceptable for some applications, like chat where responses are read word by word, it’s problematic for automated processes where the full output is needed before the next step. Getting a response from an LLM may take several minutes, which is not viable for many applications.
Limited customization
LLMs offer limited customization. Fine-tuning can help, but it’s often insufficient, costly, and time-consuming. For instance, fine-tuning a model that proposes treatment plans for patients based on data might result in slow, expensive, and poor-quality outcomes.
The Solution – Build Your Own Tool Chain
If you face the issues mentioned above, you will likely need a different approach. Instead of relying only on pre-trained models, build your own tool chain by combining a fine-tuned LLM with other technologies and a custom-trained model. This isn’t as hard as it might sound – moderately experienced developers can now train their own models.
Benefits of a custom tool chain:
- Specialized models built for specific tasks are faster and more reliable
- Custom models tailored to your use cases are cheaper to run
- Unique technology makes it harder for competitors to copy your product
Most advanced AI products use a similar approach, breaking down solutions into many small models, each capable of doing something specific. One model outlines the contours of an image, another recognizes objects, a third classifies items, and a fourth estimates values, among other tasks. These small models are integrated with custom code to create a comprehensive solution. Essentially, any smart AI model is a chain of small ones, each performing specialized tasks that contribute to the overall functionality.
For example, self-driving cars do not use one giant super model that takes all input and provides a solution. Instead, they use a tool chain of specialized models rather than one giant AI brain. These models handle tasks like computer vision, predictive decision-making, and natural language processing, combined with standard code and logic.
A Practical Example
To illustrate the modular approach in a different context, consider the task of automated document processing. Suppose we want to build a system that can extract relevant information from documents (e.g., each doc might contain various information: invoices, contracts, receipts).
Step-by-step breakdown:
- Input classification. A model to determine the type of document/chunk. Based on the classification, the input is routed to different processing modules.
- Specific solvers:
- Type A input (e.g., invoices): Regular solvers handle straightforward tasks like reading text using OCR (Optical Character Recognition), formulas, etc.
- Type B input (e.g., contracts): AI-based solvers for more complex tasks, such as understanding legal language and extracting key clauses.
- Type C input (e.g., receipts): Third-party service solvers for specialized tasks like currency conversion and tax calculation.
- Aggregation. The outputs from these specialized solvers are aggregated, ensuring all necessary information is collected.
- LLM Integration. Finally, an LLM can be used to summarize and polish the aggregated data, providing a coherent and comprehensive response.
- Output. The system outputs the processed and refined information to the user, your code, or some service.
This modular approach, as depicted in the flowchart, ensures that each component of the problem is handled by the most appropriate and efficient method. It combines regular programming, specialized AI models, and third-party services to deliver a robust, fast, and cost-efficient solution. Furthermore, while constructing such an app, you can still utilize third-party AI tools. However, in this methodology, these tools do less processing as they can be customized to handle distinct tasks. Therefore, they are not only faster but also more cost-effective compared to handling the entire workload.
How to Get Started
Start with a non-AI solution
Begin by exploring the problem space using normal programming practices. Identify areas where specialized models are needed. Avoid the temptation to solve everything with one supermodel, which is complex and inefficient.
Test feasibility with AI
Use general-purpose LLMs and 3rd party services to test the feasibility of your solution. If it works, it is a great sign. But this solution is likely to be a short-term choice. You will need to continue its development once you start significant scaling.
Develop layer by layer
Break down the problem into manageable pieces. For instance, try to solve problems with standard algorithms. Only when we hit the limits of normal coding did we introduce AI models for some tasks like object detection.
Leverage existing tools
Use tools like Azure AI Vision to train models for common tasks. These services have been on the market for many years and are quite easy to adopt.
Continuous improvement
Owning your models allows for constant improvement. When new data isn’t processed well, user feedback helps you refine the models daily, ensuring you remain competitive and meet high standards and market trends. This iterative process allows for continual enhancement of the model’s performance. By constantly evaluating and adjusting, you can fine-tune your models to better meet the needs of your application
Conclusions
Generative AI models offer great opportunities for software development. However, the traditional wrapper approach to such models has numerous solid drawbacks, such as the lack of differentiation, high costs, performance issues, and limited customization opportunities. To avoid these issues, we recommend you to build your own AI tool chain.
To build such a chain, serving as a foundation to a successful AI product, minimize the use of AI at the early stages. Identify specific problems that normal coding can’t solve well, then use AI models selectively. This approach results in fast, reliable, and cost-effective solutions. By owning your models, you maintain control over the solution and unlock the path to its continuous improvement, ensuring your product remains unique and valuable.
The post Adopting AI into Software Products: Common Challenges and Solutions to Them appeared first on Datafloq.
