AI is on everyone’s mind but what happens when we use our own and custom content like documents on different LLMs (Large Langauge Models)?
Let’s have a short look at different LLMs like llama2, falcon and gpt. Within this case we do not just want to use the pre-trained data from the models but also process our own content. Here, we just use a public available price list (T-Mobile One Fact Sheet) in a PDF format and try to gather information by prompt requests. We will now try to gather some information from especially this section which provides information for additional lines and costs per month (file here).
Now, different LLMs are requested to answer the following questions:
"How much is the monthly fee for 4 lines?"
llama2-70b-chat:
Based on the information provided in the context, the monthly fee for 4 lines would be ﹩140. This is calculated by adding the monthly price for 3-8 lines (﹩50) to the monthly price for 1 line (﹩70), resulting in a total monthly fee of ﹩120. With AutoPay enabled, the monthly fee would be ﹩115.
llama2-13b-chat:
Based on the information provided in the document, the monthly fee for 4 lines on T-Mobile ONE with AutoPay enabled would be: 1 line: ﹩70 + ﹩20/line x 3 = ﹩210 So, the monthly fee for 4 lines would be $ 210.
llama2-7b-chat:
According to the information provided in the document, the monthly fee for 4 lines on T-Mobile ONE is: ﹩70 + (﹩50 x 3) = ﹩190 So, the monthly fee for 4 lines on T-Mobile ONE is ﹩190.
falcon-40b-v2:
The monthly fee for 4 lines would be ﹩70 + ﹩50 + ﹩20 + ﹩30 = ﹩180. However, if you do not have AutoPay enabled, the monthly fee would be ﹩5 more per line, so the total would be ﹩185.
falcon-7b-v3:
The monthly fee for 4 lines of T-Mobile ONE is ﹩140. This includes the ﹩70 per month for the first line, plus ﹩50 per month for each additional line. The ﹩30 per month discount for each line with AutoPay is not included in this calculation.
gpt-3.5-turbo:
According to the information provided, the monthly fee for 4 lines on T-Mobile ONE is ﹩70 + ﹩20/line. Therefore, the total monthly fee for 4 lines would be ﹩70 + (﹩20 x 3) = ﹩130.
Summary:
Already with the trained data it became apparent that a cautious and above all testing handling of the results is elementary. Unfortunately, the use with own documents and use as an assistant is also only possible to a very limited extent here, so that all results must be carefully checked again. Likewise, it is clearly visible that almost every LLM gives a completely different result. Not only the quality differs, but also the elaboration of the results in terms of speed within the data processing. While a simple price list already has such problems, the use of much longer and more complex documents is even more prone to errors.