I have an admittedly-odd hobby of reading old (1970s, 1980s) computer magazines. As I go along, I'll often see something that piques my curiosity, and so I'll do an Internet search for more tidbits about the item. Well, about a third of the time, the LLM box will confidently assert that the thing I just read about did not ever exist. Even when there's a directly-relevant search result a handful of items down.
Similarly, I tried, a few months ago, to have several LLMs perform a quite simple task -- take a list of the countries that are parties to the Rome Statute of the International Criminal Court, take a list of countries by population, and give me the combined population of the countries that are members of the ICC. Every single time, the LLM crapped out. Even given explicit instructions on how to complete the task, a model would at most get a dozen countries in, and then quit in favor of some method of estimation. With results that were all over the map (as it were).
(Incidentally, Wolfram Alpha, which is not an LLM, will give you accurate-for-2023 stats for the member countries in the case of any number of organizations, including the ICC. Manually adding Ukraine and subtracting the Philippines is easy enough, and indicates that every LLM estimate was off by at least 10%, and usually more.)
I accordingly am of the opinion that the thing LLMs are most suited for is enabling students to cheat on their homework. Facile regurgitation of shallow knowledge, an inability or unwillingness to work, and an obligation to answer instead of saying "I don't know" -- LLMs do a great job of emulating a sixteen-year-old just trying to skate through a required class.
Similarly, I tried, a few months ago, to have several LLMs perform a quite simple task -- take a list of the countries that are parties to the Rome Statute of the International Criminal Court, take a list of countries by population, and give me the combined population of the countries that are members of the ICC. Every single time, the LLM crapped out. Even given explicit instructions on how to complete the task, a model would at most get a dozen countries in, and then quit in favor of some method of estimation. With results that were all over the map (as it were).
(Incidentally, Wolfram Alpha, which is not an LLM, will give you accurate-for-2023 stats for the member countries in the case of any number of organizations, including the ICC. Manually adding Ukraine and subtracting the Philippines is easy enough, and indicates that every LLM estimate was off by at least 10%, and usually more.)
I accordingly am of the opinion that the thing LLMs are most suited for is enabling students to cheat on their homework. Facile regurgitation of shallow knowledge, an inability or unwillingness to work, and an obligation to answer instead of saying "I don't know" -- LLMs do a great job of emulating a sixteen-year-old just trying to skate through a required class.
