Survey Says…
The Public Service Employee Survey could be better.
I don't mean the reported results. Last week, the release of the 2024 PSES data set generated the usual articles about the hardships of working as a Canadian bureaucrat, and the worst departments in which to do so.
No, I'm talking about the survey itself. It has its problems.
Let's start with the fact that the Survey isn't actually a survey. Surveys are administered to a small subset of a population, and use carefully calibrated sampling to make sure their results are representative of the whole.
The PSES doesn't use samples. Instead, it puts the questionnaire to its entire population, all federal public servants. This means it's an employee census, not a survey. And census data is sensitive to response rate. The fewer responses a census gets, the less reliable its results will be.
You might remember, about a decade ago, the Chief Statistician of Canada quit when the government of the day moved from a mandatory National Census form to a voluntary National Household Survey.
The Chief had reason to be concerned. The non-response rate for the old Census had been 6.5%. For the NHS, that ballooned to 26%.
This forced Statistics Canada to try to figure out how much of the NHS data could be salvaged as usable. Their conclusion was to toss out results from areas with a response rate of 50% or less. They arrived at that number by looking at scientific studies which "showed that with a global non-response rate of 50% or more, the bias was so large that the estimates were not of sufficiently high quality."
Back to the most recent Public Service Survey. Its global non-response rate was 49.5%.
Ok, technically more than half of employees filled out the PSES, so... that's good I guess? But yeah, it's not great either. And in some departments, the rate was quite a bit lower. At Justice, my old department, just 44.7% filled it out.
So that's the first problem. The non-response bias inherent in the PSES renders large swaths of its results statistically "not of sufficiently high quality", in the words of StatsCan, to be reliable.
The other one has to do with the PSES itself. It's long. Reeeeeally long.
The form has a daunting 118 questions, but some of these are multi-parted, meaning the real tally is higher. If you tried to tackle it in good faith, taking your time to read, understand, and think about each response, it would take you hours.
There won't have been too many civil servants who took on Q118 with the same fresh vigour, assiduity, and bright-eyed willingness to provide their helpful input for the greater good as they did with Q1.
Ain't nobody got time for that. At a certain point, even the most dedicated employees will just want to finish, muttering whatevs to themselves as they speed through it.
In the science of surveys this is referred to as "satisficing" and it's been proven to hamper a survey's usefulness. Too many questions shift the survey taker's attitude from wanting to be accurate to wanting to get it over with.
Then there are the questions themselves. Some of them are straightforward, but others — often the most important — can be overly complex.
This is Question 58:
Harassment is normally a series of incidents, but it can be one severe incident that has a lasting impact on the individual.
Harassment refers to any action, conduct or comment, including of a sexual nature, that can reasonably be expected to cause offence, humiliation or other physical or psychological injury or illness to an employee, including any prescribed action, conduct or comment.
Q58. Having carefully read the definition of harassment above, have you been the victim of harassment on the job in the past 12 months?
Yes
No
Ummm, maybe? I mean, did any comment by any colleague offend me over the last year? Well, I guess, probably yeah. But... is that what's really meant by that question? I know it says any comment — or action or conduct — but they can't really mean any. Can they? So I guess the answer is no? Sort of? I dunno…
You get the point. There are too many variables in this question and too many opportunities for survey takers to inject their own simplifications and interpretations -- what exactly do they mean by "prescribed"? -- for it to generate anything like usable data.
The solution would be to scrap the PSES and switch to properly designed and administered surveys. But that would mean admitting just how flawed the PSES has been, and having to live with accurate results generated by a rigorous assessment of civil servants’ views.
It would be a good start, though.
The people who work for Canada are too important for us not to listen to what they have to say, whatever that might actually be.
____________________