More and more employees are using AI tools in an attempt to increase the efficiency and productivity of their work. But is that really the case?
Almost all AI tool providers claim that using their tool helps to increase efficiency. However, only very few tools also offer direct options for measuring success.
In this article, I will show you how you can measure the success of tools such as ChatGPT, Gemini or SwissGPT and which KPIs play a role in this.
Which employees use the AI tools?
The first question you should ask yourself is which employees in your organization are already using the AI tools ChatGPT, SwissGPT and the like. In some cases, organizations have team licenses, but a closer examination reveals that many employees do not use the tools in their day-to-day operations.
Conversely, it is also possible that certain AI tools are actually prohibited in your company, but a closer analysis of the browser data reveals that the tools are being used secretly.
Finally, you can use this key figure to get a feel for which employees in your organization are particularly keen to use AI tools. You will find that usage varies according to age and level of education. But the function and the department also play an important role. There are departments where the added value of AI is significantly higher than in other departments.
What are the benefits of this KPI?
If you have allowed the use of AI tools in your company and your employees have access to ChatGPT, SwissGPT or Gemini, but do not use them or only use them to a very limited extent, there may be a lack of suitable training. Even if the use of ChatGPT seems very intuitive to many, not all employees are aware of how they should use these tools and, above all, for which applications and requests they should ask the AI for support.
Every new tool that is introduced in the company requires sufficient employee training. The use of ChatGPT also requires training. Have a look at the page of my ChatGPT Starter Workshop.
If your analysis reveals that employees are using AI tools even though they are actually prohibited, this is a sign that your employees are willing to use AI tools and you should immediately look for ways to provide your colleagues with suitable tools. The Swiss start-up AlpineAI, for example, offers very secure AI solutions.
How often are the AI tools used?
The next important KPI is the frequency of use. Here you can find out whether the use of AI tools has already become routine in your company or is still a rarity. Usage will vary depending on the department.
What are the benefits of this KPI?
When evaluating the values for the frequency of AI tool use, it should be noted that this will never be the same in all departments. There are departments and areas of responsibility where the use of AI tools is more important, while the human factor still plays a much greater role in other departments.
It is important to understand whether the employees who can expect the greatest increases in efficiency through the use of AI tools are actually utilizing this potential. At the same time, you should generally see an increase in the use of AI tools among most employees. If you have already offered training courses on AI tools, the usage figures should increase significantly again.
For which requests are the AI tools used?
When asked about the type of request, you can differentiate between various characteristics. The following distinctions are relevant:
1. use cases
This is about the specific requests for which the AI tools are used. For example, they are used to create translations, summarize texts or generate social media posts.
2. uniqueness of the requests
The issue here is whether the AI tools are mainly used for repetitive activities and therefore the same requests are increasingly made, or whether the intentions always vary and the requests have to be formulated anew each time.
If the requests are repetitive, it may make sense to work with custom GPTs instead of the general use of ChatGPT. more on this in my article on the possibilities with custom GPTs. If you use tools such as SwissGPT or Gemini, you can create prompt templates. For example, I have created a prompt template for Generative AI Business Cases.
3. share of the safety risk
The data protection risk also varies depending on which employees use the AI tools and the purposes for which they are used. In most cases, the use of AI tools in marketing works without sensitive or personal data. However, when AI tools are used in HR management, customer service or internal communication, the security risk increases. The higher the risk of jeopardizing data protection, the more important it is to provide employees with good information and training. At the same time, secure AI tools such as SwissGPT can completely prevent sensitive data from reaching third parties.
How often are the prompts optimized?
Prompts are the inputs that we enter into an AI tool so that the tool performs a specific task for us. The list of prompt guidelines suggests that entering the right prompt is not always easy. Users often enter one prompt per request in different variations or optimize the original prompt until the result meets their expectations.
You should measure how often a prompt needs to be improved before the result can be used further.
What are the benefits of this KPI?
If, when measuring the number of prompt optimizations, you find that the majority of employees need significantly more than one prompt iteration, I recommend targeted prompt workshops. It is often just a few tips that increase the quality of the individual prompts.
How useful is the output really?
The key figure for the prompt iterations also includes the evaluation of the output. The use of these AI tools is only justified if the users of SwissGPT, ChatGPT and Gemini actually use the results.
What are the benefits of this KPI?
If you notice at this point that most of the results of the AI tools cannot be used any further, there are two reasons for this:
- The output is not good enough. In this case, your team will need further training.
- The output was created without a business case. More and more users of AI tools are making requests and producing outputs that have no concrete connection to the business. In this case too, further training on AI applications is essential.
How do I measure the success of AI tools?
Now you know which KPIs you should measure, but the question arises as to how these figures can be measured. The well-known tool, ChatGPT, for example, does not offer direct dashboards with these KPIs. You therefore only have the option of asking your employees in the form of team surveys to measure success.
The Swiss AI startup AlpineAI recognized this gap very early on and offers its customers a variety of dashboards and analysis options. Employees with admin rights can track the above-mentioned KPIs directly via a dashboard in real time.
Conclusion: Measuring the success of AI tools
In this article you have learned the following 3 things:
1. the most important KPIs that you should consider when using AI tools are the following:
- Which employees use the AI tools?
- How often are the AI tools used?
- For which requests are the AI tools used?
- How often are the prompts optimized?
- How useful is the output really?
- How do I measure the success of AI tools?
2. the KPIs cannot always be measured straight away. In some cases, additional evaluations and employee surveys are necessary.
3 KPIs alone are not enough. Measures are also needed to help ensure that the KPIs are constantly improved.
Fancy more?
I would be happy to help you measure and optimize your KPIs. Just write to me.
I look forward to your inquiries.
