Read on Twitter
Today, I uploaded my Masters Thesis Seminar to YouTube for easy access (). However, given that my topic is the interpretation of SPC outlooks by broadcasters and the general public, I thought I would be remiss if I didn't share on here. Thread time!
So for starters, I'd been a busy few weeks on twitter discussing the SPC outlook! Lots of concern about the words in the outlook not lining up with the risk that they are supposed to convey. Some concerns, some humor, but wide anecdotal agreement that something's up.
However, there were two tweets that really captured my attention by leading to some good questions - questions that I've been investigating for my masters' thesis!
Beyond the anecdotal evidence, the outlook is a key part of the early messaging of severe weather events, priming people and getting them ready for the potential disruption storms can bring. They also haven't been studied much from an interpretation/value standpoint.
So, my research questions: how do broadcast meteorologists, who act as the link between NWS and the public, use and gain value from the SPC convective outlook? How does the public interpret the outlook? Do anecdotal and broadcaster concerns match data on the public?
First, let's look at how the outlook came about. Though tornado alerts were first attempted in 1884, and successfully applied in 1948 at Tinker AFB, the outlook and SPC as we know it really started to come together in the 1990s.
In 2014, we had a BIG change - the addition of two new categories, Marginal and Enhanced. These were added in response to feedback from emergency managers, who felt that there were "slight risk" days and "BAD slight risk" days. The other categories remained totally unchanged.
Before we move on, I want to talk about what makes a forecast like the SPC outlook good. Murphy (1993) describes three pillars of forecast goodness - Consistency, Accuracy, and Value.
Consistent forecasts represent the true forecast that the forecaster has internally. Accurate forecasts match the atmospheric outcome they forecast. However, forecasts have no inherent value - value is generated by the decisions of users of that forecast information.
Thus, to study the value of a forecast, we study its users. I used two data collection methods to study broadcaster outlook use - CTAs and Focus Groups.
These methods collect qualitative data - not something easily analyzed by statistics. I instead used Thematic Coding, where my brain is the instrument - taking careful notes as I seek out patterns in the data, then mapping those patterns, or themes, to draw conclusions.
My analysis of the CTA interviews (thus far) has resulted in the development of the following thematic map, which I will break down by supporting each subtheme.
First - broadcasters report using the outlook as a way to communicate the risk of severe weather, with some translating it into a bar graph displaying threat levels, and others showing the outlook as a way to display the region at risk.
Broadcasters also use the outlook as a "sanity check" - does the outlook match my thoughts? Did anything change overnight while I slept? The outlook helps broadcasters maintain situational awareness for severe weather events.
Onto the outlooks impacts on behavior - first, broadcasters reported increasing contact with news directors and management for higher risks, as you can see for these two broadcasters across the three risk levels we discussed with them.
Next, broadcasters reported getting into work earlier for higher risk days. I built this table to display how early each broadcaster interviewed got into work, compared to a no risk day - on average 0.78 hours early on a slight day, and 2.02 hours early for a higher risk!
Finally for behavior, anxiety. Broadcasters mentioned feeling anxious, stressed, and overwhelmed when high or moderate risk days were forecast. Some even reported taking sleeping pills to make sure they were rested for the coming storm. Its a feeling I can relate to.
However, broadcasters also had some concerns about the outlook. Several mentioned not really experiencing high risk days - that slight risk days were their "big days."
Additionally, several broadcasters reported having issues translating the outlook into a form their viewers could use, with MET18 putting it best - "KISS principle is important when you only have 2 min." Spanish translations were also an issue ( @LatinWx is working this problem!)
That wraps up the CTA findings. I'm still working through the data analysis on my focus group data, but I do have some initial details. Broadcasters reported using the outlook as a forecast baseline, and ramping up staffing and messaging in advance of higher risk days.
They also *really* did not like the outlook terms, ESPECIALLY slight and moderate. Big concerns there - especially for areas, like where MET1 is from, where high risks are very uncommon and slight risks are the big days.
So to sum up the broadcaster investigation - findings are consistent thus far. Outlooks aren't an end-all for the forecast process, they have behavioral *and* emotional impacts, and broadcasters really really don't like "slight".
Now we get to look at the investigation with the general public - how do they interpret the outlook? I generated two survey questions, asking participants to rank the words and colors from least to greatest risk, to find out!
(got about 15 tweets worth of data out of these two)
(got about 15 tweets worth of data out of these two)
I then included these questions in the @OURISK 2019 Severe Weather and Society Survey. 3006 respondents in a demographically representative sample. Oh. A statistician's dream. Love working where I do.
So lets jump into it! First off, looking at respondent's preferred ranking for each word... is a bit tough. Slight and marginal are swapped, as are enhanced and moderate. High is DEFINITELY high though.
Same for colors now. Things a bit better off here - moderate and high get swapped, and there's a lot of uncertainty about where magenta should be. Red is DEFINITELY the highest risk color though, and green is DEFINITELY the lowest.
I also broke down participants by how they ordered the risk words and colors - as you can see, the words are a lot messier than the colors. Correct word order is the 4th largest group, while correct color order is the 2nd largest.
