Thursday, July 27, 2017

Is Astronomy a Hostile Workplace for Women and Minorities?

The authors of a recent paper in the Journal of Geophysical Research think so. In a survey they conducted in early 2015, astronomers reported how often they heard negative language, experienced verbal and physical harassment, felt unsafe, and skipped work activities due to concerns about their safety.

88% of respondents reported hearing negative language from peers at their current career position, 51.9% reported hearing negative language from supervisors, and 88% reported hearing negative language from others. Thirty-nine percent of respondents report experiencing verbal harassment at their current position, and 9% report experiencing physical harassment. Twenty-seven percent of respondents report that they have felt unsafe at their current position, and 11% report that at their current position they have skipped attending at least one professional event such as a class, meeting, conference, or fieldwork opportunity because they felt unsafe attending.

They go on to say that their results "suggest there is not only a hostile climate in the astronomical community but that the community is experienced differently depending on one’s gender and race." The paper has received broad coverage in the press,* where "widespread harassment" in astronomy is now reported as an established fact. "The sciences are overwhelmingly hostile to women," wrote Rae Paolotta at Gizmodo, "and in astronomy, it’s doubly bad for women of color." Kate Clancy, the lead author of the paper, called it "great piece" on Twitter.**

The astronomy community appears not to have any objections to this characterization. The American Astronomical Society and the American Geophysical Union issued a joint statement, acknowledging the findings and promising "positive change". When I asked the executive director of AAS, Kevin Marvel, to comment specifically about the paper's claim that there is "a hostile climate in the astronomical community" he offered the following response:

We are glad the report has come out and it is now important for the community think carefully about the information it presents and the resultant recommendations. Although we have made good progress in some areas regarding harassment, there is more to do, and reports like this play an important role in moving the ball forward as we say in the US. We will use this, and other sources of information and recommendations to constantly work toward ensuring a professional environment free of harassment. Let us hope we achieve it sooner rather than later.

As far as I know, no one other than me has publicly challenged the assertion that astronomy is a hostile workplace for women and minorities. If the representatives of the astronomy community wanted to defend its members and its culture against this charge it could easily do so by looking more closely at the results. The AAS's Committee for the Status of Women in Astronomy sponsored the survey, and one the authors is a member, so I assume they have access to the raw data. As I never tire of pointing out, this would allow them to disaggregate the results according to the never/rarely/sometimes/often scale of the questionnaire. Since the authors of the study completely ignore me, I have been estimating what this would reveal based on the slides of the preliminary results that were presented at DPS in 2015 and AAS in 2016.

Here is why I don't believe the hype. Although 39% reported verbal harassment, less than 13% appears to have reported it occurring more than "rarely", and less than 2% seem to have reported it happening often. That's about seven people in a sample with a strong self-selection bias towards people who have something to report, including witnesses and allies, and an (intentional) oversampling of women. In the most extreme self-selection scenario we can imagine, everyone in the astronomy community who feels they're often harassed would have reported. In that case, .07% of astronomers (pop. 10,000) experience verbal harassment often. I estimate the upper bound on the amount who experience verbal harassment often at 60 people (2% of female astronomers); in that case, about .6% of astronomers (men and women) experience verbal harassment often.

A similar approach applies to all the other results. First we disagreggate the total percentage into never/rarely/sometimes/often and discover that a majority of respondents in the sample report the measured experiences of "hostility" never or rarely. Next, to get a realistic sense of prevalence in the whole population of astronomers, we adjust (always downwards) for the self-selection of victims, witnesses and allies, and the oversampling of women.

I believe astronomers are being let down on multiple fronts. Journalists are clearly not covering this story in an impartial, or even competent, manner. They are not applying even a modicum of skepticism to some obviously sensational claims being published with obvious political ambitions. They don't seem to have even a basic understanding of sampling bias or the now very well-known problems associated with significance testing. (We'll get to that in another post.) Social science is also letting astronomers down by making overblown claims based on underpowered studies and promoting their spread through the media. The journal editors and reviewers here also don't seem to have thought about the reputations of astronomers, either as a field or as individuals, when accepting this study for publication (and promoting it thereafter). Finally, I think the political bodies that are supposed to tend to the interests of astronomers have let their membership down. Not only did the AAS fund this study, they have not offered any critical pushback on behalf of the community it smears as steeped in sexism and racism.

All this, of course, is just my opinion as an outsider looking on a field that I used look at with awe and envy. I hope astronomers will find a way to get their house in order. To riff on Kevin Marvel's statement: let us hope they achieve it sooner rather than later. I am here to help in any way I can.

____________
*Here are some representative headlines: "Women of color face staggering harassment in space science" (Sarah Kaplan, WaPo). "Women of Color in Astronomy Face Greater Degree of Discrimination, Harassment" (Calla Cofield, Space.com). "Survey data point to widespread problems" (Colleen Flaherty, IHE). "There’s a lot of bias in astronomy" (Angela Chen, The Verge). "Female astronomers of colour face daunting discrimination" (Rachael Lallensack, Nature). "A new survey of astronomers and planetary scientists reveals a workplace harassment problem in the space sciences" (Francie Diep, Pacific Standard). "Unprecedented study reveals widespread bias in space science, and it's particularly terrible for women of color" (Miriam Kramer, Mashable). "Astronomer survey reveals gender and racial harassment" (Michael Banks, Physics World). "Widespread harassment reported in astronomer survey" (Toni Feder, Physics Today).
**Update: I forgot to mention that Clancy endorsed Paelotta's article even though it got the study's methodology completely wrong. "Clancy and her team surveyed 474 astronomers and planetary scientists between 2011 and 2014," says Paolotta, though the study was conducted from January to March of 2015. "All subjects identified as women or non-binary," she also says, though the sample was almost one third male.

Tuesday, July 25, 2017

Exclusion

Here's question 7 of the CSWA Workplace Climate Survey:

In your current position, how often have you been VERBALLY harassed because of the following characteristics?

{Choices include: Often, Sometimes, Rarely, Never}
Race or Ethnicity
Physical Disability Status
Mental Disability Status
Sexual Orientation
Gender Identity(Cisgender or Transgender)
Gender (Female, Male, or Non-binary)
Religion or Lack Thereof

Question 9 has the same form, with "PHYSICALLY" replacing "VERBALLY". Now, here's hypothesis 1:

Female respondents will report more verbal and physical harassment than men.

This sentence appears in the first paragraph of the results section:

Thirty-nine percent of respondents report experiencing verbal harassment at their current position, and 9% report experiencing physical harassment.

And this one appears in their support for hypothesis 1:

Women were also significantly more likely than men to report that they experienced both verbal and physical harassment because of their gender.

I have underlined that phrase because it draws attention to the glaring absence in the questionnaire of a "characteristic" that is likely to have been focus of harassment directed at white male astronomers. Indeed, though the paper doesn't tell us this (we know it only from Christina Richey's preliminary presentations at DPS in 2015 and AAS in 2016), race and gender account for the great majority of characteristics that people felt they had been harassed for. In the case of verbal harassment, they account for 65% of the reports. In the case of physical harassment they account for more than 80%.

For obvious reasons white male astronomers are not likely to report being victims of harassment because of their race or gender. If they are also straight, cisgendered, protestant, able-bodied and do not suffer from mental illness, they would seem to have no way to report their experiences on the survey. And yet, surely, they might experience harassment. Most commonly, they will experience verbal (and at times physical) harassment by professional rivals, for whom they are competing for publication, promotion and research funding. This basis for harassment has been completely excluded from the Workplace Climate Survey. The questionnaire did not even provide a generic "other" characteristic in which to report harassment.

Now, if the hypotheses tested had confined themselves to race- and gender-based forms of harassment, this wouldn't be such a big problem. Except that the conclusion that women and people of color experience more gender- and race-based harassment than white men is a bit underwhelming. But Clancy et al. claim to have found support for hypothesis 1, and they are promoting the result widely as suggesting that women, and women of color, experience more harassment than men full stop. As it turns out, this conclusion emerges from a measurement instrument that excluded the most typical experiences of harassment among white men.

I'm going to take some time to think about the consequences of this. But my initial reaction is that it completely undermines the validity of the survey, given the hypotheses it claims to be testing. Also, it raises the interesting question of whether women and minorities experience harassment based on professional rivalry (they would also not be able to report it). And that, finally, raises a question that concerns me greatly: is it possible that women and minorities are getting the axis of their harassment wrong? Is it possible that they experience harassment that is really grounded in ordinary competition as grounded in racism and sexism? If so, surveys like this are distracting them from the fight they should be fighting; and this, ultimately, will be to their disadvantage.

Comments are welcome, as always.

Monday, July 24, 2017

Hypothesis 0

According to Clancy et al., the CSWA Workplace Climate Survey was designed to test four hypotheses:

1. Female respondents will report more verbal and physical harassment than men.
2. Respondents of color will report more verbal and physical harassment than white respondents.
3. Trainees will report more verbal and physical harassment than those scientists of a higher rank.
4. Women of color experience double jeopardy where they are especially at risk for verbal and physical harassment compared to white women or men of color.

While they do not make the null hypothesis explicit, it is clear that they are testing these against their simple negation. That is, the null to be rejected is that men and women, whites and non-whites, trainees and seniors, will experience equal amounts of verbal and physical harassment.

This null, I want to argue, is not very compelling. To see why, consider what these hypotheses would look like if they were not about the sample (i.e., the "respondents") but about the population (i.e., "astronomers"). Here we need to add some information to capture the wording of the survey that the respondents were responding to, fixing both the situation and the basis of the harassment (I have marked the additions with square brackets):

1. Female astronomers experience more verbal and physical harassment [from other astronomers based on their gender] than male astronomers.
2. Astronomers of color experience more verbal and physical harassment [from other astronomers based on their gender or race] than white astronomers.
3. Astronomers in training experience more verbal and physical harassment [from other astronomers based on their gender or race] than astronomers of a higher rank.
4. Female astronomers of color experience double jeopardy where they are especially at risk for verbal and physical harassment [from other astronomers based on their gender or race] compared to white female or male astronomers.

This, it should be noted, is roughly how the results have been presented to the public by press releases and news coverage. But the importance of turning the hypotheses into statements about astronomers lies in pointing to what we can call the Astronomy Effect on the likelihood of experiencing verbal and physical harassment. Does becoming an astronomer expose you to a particular risk of such harassment? Consider four hypotheses about the general population:

1. Women experience more gender-based verbal and physical harassment than men.
2. People of color experience more race-based verbal and physical harassment than white people.
3. Trainees experience more race- and gender-based verbal and physical harassment than people of a higher rank.
4. Women of color experience double jeopardy where they are especially at risk for gender- or race-based verbal and physical harassment compared to white women and men.

It would be surprising if these statements weren't true in the general population, right? So the implicit null of the CSWA survey was that joining the astronomy community would completely eradicate race- and gender-based differences. Why would we begin there? Why would we not begin with the reasonable hypothesis that becoming an astronomer would reduce your exposure to the risk of harassment, and rising in the ranks of astronomy would reduce it still further? We might then be distraught, as Clancy and Richey have said they were, to learn that it actually has no effect on your chances of being harassed, or, indeed, actually increases it.

Such an effect (which I think is unlikely, but possible) could be explained by the continuing, if shrinking, gender and race disparities in astronomy. If becoming an astronomer increases her exposure to the company of men, she might find her exposure to harassment increase as well. Likewise, by deciding to join the community of astronomers, people of color will generally be exposing themselves to the company of white people. If their baseline experience has been among people of color, they might well experience a change similar to moving from a black to a white community.

My point isn't to argue for or defend any level of harassment. My point is just that a null hypothesis does not have to have a zero value. It just has to suggest a zero effect on the dependent variable of the hypothesized force, which, here, as far as I can tell, is the particular "hostility" that the astronomical community allegedly directs toward women and minorities. Even if astronomers are not particularly hostile to women and minorities, I want to say, we would expect them to experience more gender- and race-based harassment than white men in the same field. It's just that we'd expect the overall level of that harassment to go down when they are at work.

Finally, I would assume that the baseline level of race-based and gender-based harassment that white men face (before going into astronomy) is quite low (almost by definition, you'll note) and might drop to virtually zero in the protected space of the observatory. In a small sample, it's hard to predict what this will do for the statistical significance of the disparity between genders and races. Indeed, I suspect (but I will deal with this in latter post) that the low p-values (often < 0.001) stem from the fact that we are comparing groups that (again, essentially by definition) don't face the relevant form of harassment with those who actually might.* All this seems completely obvious to me and I'm at a loss to understand how an esteemed scientific journal like JGR:Planets came to publish the result in the form it did. Again, I'm happy to hear from people who think I'm wrong. Quantitative analysis isn't my strong suit either.

________
*Think, for example, of what would happen if we compared unqualified harassment of men and women in the general population. I think we would find that men get pushed around, as it were, as often as women. They just don't experience this as having anything to do with being men. This survey did not give white men much room to report these experiences.

Was the CSWA survey confined to recent experiences?

As I mentioned in my last post, a commenter on my first post on its publication pointed out that it does not seem true that the CSWA Workplace Climate Survey asked respondents to confine themselves to the years 2011-2015 when answering the questions. Clancy et al. are, it should be noted, adamant on this score:

At this time, we want to remind the reader that the findings of this study cannot be attributed to events from long ago: respondents were asked to only report experiences they had had in their current career position over the last 5 years. The events and experiences reported in this survey happened between 2011 and 2015. (p. 11)

But here is what respondents were told in the cover letter to the survey:

The survey is designed to request information during the respondent's current position and previous position (if the respondent has changed positions within the last five years). [...]

Directions: Please respond to Section 1 regarding your personal experiences in your current position. If you have changed career positions in the past five years, then please also complete Section 2 regarding your personal experiences in your previous position.

This does not seem to me to confine responses to the years 2011-2015. (While it's ultimately a moot point, do notice that "the last 5 years" would cover 2010-2014, not 2011-2015, since the survey was administered in early 2015.) If a respondent had been working somewhere for, say, 15 years they would reasonably interpret this as covering the whole time there. If they have changed position within five years, they are being asked to answer also for the entire time of their previous position. In other words, they are being asked to think back at least five years, not at most.

This strikes me as a serious issue, especially given Clancy et al.'s "reminder" to the reader. If the frequency of experiences is really spread over, say, 20 years and the analysis assumes they are concentrated within 5 years, this will strongly distort the result. If anyone knows what I'm getting wrong here, do let me know in the comments.

Friday, July 21, 2017

The CSWA Survey in Plain Language

It's going to take a bit of work to properly critique the the CSWA Workplace Climate Survey as published in the Journal of Geophysical Research. In this post, I want to take a sentence-by-sentence look at the plain language summary. In later posts, I will elaborate on each point by way of a critique of the substance of the paper. I have already pointed out that the press releases spin the survey in ways that the paper itself belies. As it turns out, this spin is also present in the summary. Indeed, it is also present in the abstract, but slightly more subtly. Like I say, I will go through it one sentence at a time.

Women generally, and women of color specifically, have reported hostile workplace experiences in astronomy and related fields for some time.

This is, of course, true. As Kate Clancy has noted elsewhere, it's true of every field of human endeavor, and it is true of all races and genders. Everyone has experienced hostility at work. Work is done by humans in human environments and hostility is a human capacity. Indeed, humans are "capable" of hostility in both directions: they can dish it out and they can take. In short, the first sentence is a truism. The second sentence gestures at something less trivial.

However, little is known of the extent to which individuals in these disciplines experience inappropriate remarks, harassment, and assault.

It's true that this specific question hasn't been studied directly in astronomy. But there is some evidence to suggest that astronomy and related fields are not especially hostile places, specifically to women. (Women of color are, as is often noted, very underrepresented in astronomy and do seem to get lost in such studies.) One study found that women don't think about leaving the discipline more frequently than men; another found that, while they are 1.64 times more likely to have negative experiences than men, the average level of hostility was on the order of occasionally hearing a sexist joke. But it must be granted that the extent to which individuals have particular experiences is not well understood. The next sentence implies that this study will do something about this gap in our knowledge.

We conducted an internet-based survey of the workplace experiences of 474 astronomers and planetary scientists between 2011 and 2015.*

What they here imply is misleading since the paper explicitly states that "these data cannot provide a direct assessment of prevalence". That is, their "plain language summary", presumably intended for the non-expert (or journalist) gets the reader to think that they have done a study to gain the knowledge we lack, even though the authors are well aware that the study was precisely not designed to gain that knowledge, i.e., knowledge of the "the extent" (prevalence) of hostility in astronomy. That is of course also why they don't present general findings of prevalence, only comparisons of groups within the sample:

In this sample, in nearly every significant finding, women of color experienced the highest rates of negative workplace experiences, including harassment and assault.

This may seem like a quibble, but it is worth noting: the survey asked about "verbal harassment" and "physical harassment", not harassment and assault. They don't actually know the extent to which people in their sample were reporting assaults, except on a definition of assault in which any unwanted touching constitutes an assault (I'll discuss this in another post). Note that we are not told whether they experienced these things at a generally high or low rate, mainly because the study sets no threshold to make such judgments. The next sentence does report some alarming levels of hostility, however.

Further, women of color reported feeling unsafe in the workplace as a result of their gender or sex 40% of the time, and as a result of their race 28% of the time.

This sentence is simply a misinterpretation of the relevant result. It distorts and exaggerates their actual finding, as stated in the abstract: "40% of women of color reported feeling unsafe in the workplace as a result of their gender or sex, and 28% of women of color reported feeling unsafe as a result of their race." That is, it is not true that respondents felt unsafe 40% of the time; rather, 40% of respondents felt unsafe some of the time—or, more accurately, had felt unsafe at some time in the past. Indeed, they were specifically asked whether they had "ever felt unsafe" in their current position (see also footnote*). Answering "yes" here says nothing about how long or how often they felt unsafe. If 40% of respondents had ever felt unsafe, surely the population doesn't feel unsafe 40% of the time.

Finally, 18% of women of color, and 12% of white women, skipped professional events because they did not feel safe attending, identifying a significant loss of career opportunities due to a hostile climate.

As far as I can tell, this is a completely accurate summary of the result. I have said before that this is an important one, since it shows that there is a difference between feeling unsafe and doing something about it. It has been established in other studies that women feel more unsafe than men (even when both sexes feel very safe) and it stands to reason that this would translate into more absenteeism among women. It needs to be stressed that the survey found that only 9% of respondents reported "physical harassment", i.e., arguably an actual violation of personal safety. This suggests that women generally feel less safe than they are. This isn't a particularly surprising result, especially in a climate where women are told (by scientists and politicians) that harassment is "rampant". This study, of course, is one of the things that might be making women feel unsafe. Indeed, the authors say women are unsafe explicitly:

Our results suggest that certain community members may be at additional risk of hostile workplace experiences due to their gender, race, or both.

My standing objection to this way of putting it is that it does not account for the fact that "certain community members" would be in other environments if they were not doing astronomy. The authors don't give us any way to decide what the comparative ("additional") risk of hostile work experiences would be if they went into banking, politics or even another scientific discipline. As I said at the outset, there is a risk of hostility in any human environment. If a woman of color took away from this study that she best stay out of astronomy and choose another line of work then that would be a reasonable, if tragic, conclusion to draw from the "plain language" of this summary for the public. But, since the study itself eschews any claims about prevalence, it is not actually a reasonable conclusion to draw from the survey itself. I think that is a serious problem in the communication of this result to the public. It is not only astronomers that should take issue with this; the whole ear of the public is rankly abused.

Like I say, I will be looking at the paper more closely to support these various points of criticism and raise a few more in future posts. As is my custom, I will also be asking the authors for comment. To my knowledge, there has so far not been any serious criticism of the study in the press or the science blogs. It would be to Clancy's credit if she engaged with at least one critic as part of the discussion she so insists it is important to have. But I am not holding my breath.

____________
*I will cover this in a separate post, but as a commenter on my last post pointed out, it does not seem true that the survey asked respondents to confine themselves to the years 2011-2015. As I read the questionnaire, the respondents might well have thought they were being asked "Have you ever experienced...?" I believe that the authors thought they had limited the responses in this way. But I don't think the respondents would generally understand it as limited to five years.