Jul 6, 2024
The GenAI Disappointment Curve
date
Jul 7, 2024
slug
generative-disappointment
status
Published
tags
type
Post
There is a common trend in user feedback across GenAI models: a perceived decline in output quality weeks/months after the model’s release. I’ve also seen this anecdotally. Initially, we are blown away by the results, but eventually experience disappointment (even frustration) with the tools’ capabilities. [1]
I can’t speak for all models, where degradation might have actually taken place, but the ones I have been a part of were left untouched. This can only mean that the perception of quality, not the quality itself, was reduced.
While the novelty effect is a well-known phenomenon, I haven't seen any discussions of it in the context of GenAI. Generative outputs lend themselves well to the phenomenon because:
- Output quality is subjective and not perfectly measurable.
- Outputs are unpredictable, so they may need multiple attempts and are therefore prone to negativity bias.
This results in the familiar product launch arc:
- Models are released alongside perfect demos, videos, and reviews.
- Users are delighted by how realistic the outputs are, especially compared to non-AI tools, often far beyond the actual quality.
- After several weeks, users expect the same level of astonishment with each generation, which the model cannot deliver.
- The hype and novelty fade at a rate proportional to the hype compared to the actual quality of the outputs. In some cases, the frustration caused by this difference can reduce the users' perceived quality to below the actual quality, leading to complaints and cancellations.
- A new model is released with marked improvements over the last one, and the cycle continues.
While these cycles are to be expected, there are several practical approaches to flatten the curve:
- Be upfront about what models can and cannot do.
- Release fresh demos, use cases, and capabilities even after launch.
- Allow users to edit and adjust outputs natively.
- Use standardized testing to prove consistent quality.
Interestingly, AGI may be the ultimate solution, where the speed of continual improvements surpasses our proclivity to novelty. But we’ll cross that bridge when we get there.