Read on Twitter
Folks who work on predictive models as part of your product/team roadmap: How do choose your (quantitative) goals for model improvement?
- Do you goal on absolute or relative performance?
- Relative to your baseline or to previous model?
- How do you choose the goal threshold?
- Do you goal on absolute or relative performance?
- Relative to your baseline or to previous model?
- How do you choose the goal threshold?
If you're kind enough to answer, I'd also be interested to know if the models you're setting goals for perform a classification, regression, forecasting, or some other learning task.
And to put this in somewhat plainer language, it often comes up during planning that people will say:
We need to set a specific goal for this model for Q3, and then ask, "How much of a better model should be the goal, and in what particular way do we define 'better'?"
We need to set a specific goal for this model for Q3, and then ask, "How much of a better model should be the goal, and in what particular way do we define 'better'?"
Getting the feeling this is going to be one of those problems where none of us really knows what we're doing here and we're all just making it up as we go along.
