First Impressions Count, But How?

Today we published a paper in PNAS about how people form first impressions based on everyday images of faces, of the kind you find on the internet.

The four authors (L-R), Richard Vernon, Clare Sutherland, Andy Young and Tom Hartley - we also co-wrote this blog post. Underneath are reconstructions of photos of our faces which can be loosely thought of illustrating the way such images are “seen” by our model. They are actually reconstructions based on the 65 numbers we used to describe each face, using a model trained on a large number of such images (but not these ones). Note the subtle differences in shape between the photos and the reconstructed image - the model does not (yet) capture some of the information.
The four authors (L-R), Richard Vernon, Clare Sutherland, Andy Young and Tom Hartley – we also co-wrote this blog post. Underneath are reconstructions of photos of our faces which can be loosely thought of illustrating the way such images are “seen” by our model. They are actually reconstructions based on the 65 numbers we used to describe each face, using a model trained on a large number of such images (but not these ones). Note the subtle differences in shape between the photos and the reconstructed image – the features we use do not (yet) capture some of the information in the images, but sufficient for the model to make accurate predictions about social impressions.

By first impressions we mean the way we rapidly form judgements about others’ social characteristics. Although we can make an astonishing range of social inferences based on appearance (trustworthiness, intelligence, dominance, extraversion etc.) these different traits tend to go together in predictable ways, so that they fall along two or three more or less independent underlying dimensions:

  • approachability (do they want to help me or to harm me?)
  • dominance (are they capable of carrying out these intentions?)
  • youthful-attractiveness (perhaps representing whether they’d be a good romantic partner – or a rival!)

These judgements are formed very quickly (in as little as a tenth of a second) and can influence our subsequent behaviour. The impressions we create through images of our faces (“avatars”/“selfies”) are increasingly important in a world where, more and more, we get to know one another online rather than in the flesh. So how can we go from an image of a face to a judgement about someone’s character?

To answer this, we measured physical features in 1000 images of faces drawn from the web and used them to develop a model that could accurately predict people’s first impressions of the same images. We looked at 65 different features, things like “eye height”, “eyebrow width” and so on. By combining them we could explain over half of the variation in human raters’ social judgements.

We got our best results with a “linear” model. Effectively, we simply weight the different physical features and add them together to predict first impressions. This is really the simplest kind of model we can imagine. We tried more complex models but these didn’t do any better. The fact that a very simple model of this sort can explain such a high proportion of the variation in human raters’ responses suggests that the underlying processes in the brain may also be quite basic and linked to the processing of visual features rather than being highly abstract. You can contrast this with, say, reading a word. When you look at a picture of a word, you know what it means and in that sense the meaning is present in the image, but the relationship between the meaning and pattern of lines and dots is very indirect. It depends on specific combinations of cues and on a person’s idiosyncratic experience with letters, words and their meanings. For faces the relationship between the shapes and colours that make up a face and our perception of that person’s character seems more direct and straightforward.

We were also able to summarise the features involved in different social impressions by constructing cartoon images based on the relationships in our data. We tested the cartoon images and found they produced predictable first impressions in a new set of judges. This shows how what might seem to be relatively impoverished images can capture the essential cues present in photos of our faces and it bears out the point that the mechanisms needed to make what appear to be complex social judgements may be more straightforward than has often been thought. Subjectively some of the changes in these images are quite subtle, and it can be quite difficult to pin down what is changing!

Cartoon faces depicting the changing facial features associated with different trait impressions derived from images in our database. The way we construct these is described in more detail below and in the paper. When we presented images of this sort to a new set of judges, their ratings of the social impressions the cartoons elicited correlated with the social trait manipulations we used to construct them.
Cartoon faces depicting the changing facial features associated with different trait impressions derived from images in our database. The way we construct these is described in more detail below and in the paper. When we presented images of this sort to a new set of judges, their ratings of the social impressions the cartoons elicited correlated with the social trait manipulations we used to construct them. Modified from Vernon, Sutherland, Young and Hartley (PNAS, 2014)

Every picture tells a story

Our results suggest that some of the features that are associated with first impressions are linked to changeable properties of the face or setting that are specific to a given image. So things like expression, pose, camera position, lighting can all in principle contribute alongside the structure of our faces themselves. To illustrate this (although it’s not covered in the peer-reviewed paper) look at two pictures of Clare:

Two different pictures of Clare Sutherland show how are model predicts different social impressions from the same face. The reconstructions underneath give an insight into how the model "sees" each face. Credit: Clare Sutherland, Richard Vernon
Two different pictures of Clare Sutherland show how are model predicts different social impressions from the same face. The reconstructions underneath give an insight into how the model “sees” each face. Credit: Clare Sutherland, Richard Vernon

The images underneath give you an idea of how each image of Clare is “seen” by the model – they’re actually reconstructed based on the 65 numbers we use to describe her face. When we put these images into the model, we get different answers. The passport-style picture on the right is rated (by the model) as someone less approachable, more youthfully-attractive and also more dominant. These differences are clearly result of differences in the photographs, because the face, the person depicted, is the same.

Graph showing the model's different predicted social trait ratings for each image of Clare's face.
Graph showing the model’s different predicted social trait ratings for each image of Clare’s face.

Our study implies that, with a comprehensive model based on sufficient data, we can fairly accurately able to gauge people’s likely impressions of a given image. We could use this, for example, to select an image which conveys a desirable impression, perhaps even automatically. Until then, if you’re thinking about attaching a picture to your CV, resume or online dating profile, maybe you should take a look at our paper first!

In some ways our model parallels or makes explicit the kinds of judgement that might be relevant to casting directors, animators, and portrait photographers who select or manipulate images to create certain impressions. So we can think of many potential applications and of course our work builds on earlier research which has helped to make the importance of first impressions clear.

The trouble with first impressions

We’re not normally aware of how we arrive at our first impressions. They affect our future behaviour and can be hard to overturn. So it’s useful to know how we’re being judged on our appearance, especially since these judgements might not be accurate – think of effects on court cases or democratic elections, for example. Should we really trust a smiling face? Some previous research suggests that there may be a “kernel of truth” in some of our first impressions but that we overgeneralize so that, for instance, someone with a young-looking face (i.e., features resembling those of a young child) is judged to have other immature characteristics. However, often “judging a book by its cover” will be just plain wrong! It’s not hard to imagine that unconscious bias resulting from appearance-based first impressions might contribute to less overt forms of discrimination.

FAQs

In this section we answer some of the questions that we’ve been asked about the study.

Where do first impressions originate? Are they hard-wired?

It’s important to understand that our study doesn’t tell us very much about where these snap judgements originated – they may have their roots in evolution, but it’s equally possible that individual experiences and culture play a part. Because we find that one simple model accounts for a lot of the variation in average judgements, it’s hard to believe they are determined by individual experiences alone (since these would be highly varied), but common experiences (e.g., cultural influences) could play a big role. This is something we’re looking at in current work, by comparing first impressions formed by people from different cultures. However, there is already some evidence from social psychology that the dimensions we tap into in understanding other people are universal, which might suggest an evolutionary influence. For a social species like humans, getting along with others is a key part of survival, and so being able to identify potential allies and aggressors, potential mates and those most and least able to carry out their intentions may plausibly have been an advantage to our ancestors.

Which features are most important for social judgements?

Our model performs very well with information about the physical features (especially the shape) of the faces, so we can be pretty confident that these play an important role in determining our social judgements. However, it is difficult to nail specific traits to particular subsets of features (because multiple features vary together). Here’s an analogy: the average salary is correlated to the cost of an apartment and also to the cost of a car. It’s hard to know which causes which, because they all vary in the same way, when one goes up so do the others. So we could say the salary is ( x * the apartment + y * the car), but we can get equally good answers with lots of different combinations of values for x and y. So we cannot (from our study alone) pin down with certainty the causal relationship between any single feature and the social impression. With this important caveat in mind we have some strong circumstantial evidence. It looks as if the shape of the mouth and jaw is really important for approachability; perhaps not surprisingly smiles make faces look friendly, but a downturned or poker-faced mouth looks grim, even angry. So far as attractiveness is concerned our results (and some earlier research) suggest the size of the eyes is important. In our model smaller eyes play a part in predicting lower ratings for attractiveness, everything else being equal. However, the difficulty in interpreting the effects of specific features is one reason why we wanted to generate new images to encapsulate the variations that are expected to produce different perceptions. Because we tested these with new judges, we know that (at least in combination) they produce the expected perceptual effect.

Do your results apply to all faces?

People might be concerned that we only looked at faces of ‘Caucasian’ appearance in the current study. The reason for this is that we already know that people sometimes perceive faces of other ethnicities differently from their own. This might be because of cultural stereotypes but also more subtle things such as the level of experience we have with different kinds of variation in the face. As it’s not practical to incorporate faces and judges from every possible geographic, cultural and ethnic background, we instead try to keep these factors fixed by focusing on one ethnic/cultural group at a time. We can then investigate the ways in which different groups may rely on different facial features (and perhaps reach different social judgements) in a step-by-step way. We are currently working on cross-cultural studies now to expand our understanding; we think this is a really interesting and important future direction.

What’s the story behind this study? What made you want to do it?

Tom: I’ll give you my own story, the other author’s might see it differently. It involves two things I really love about science: teamwork and exploration. At the University of York each graduate student has a “Thesis Advisory Panel”, where different members of the faculty help guide each student through a PhD. I am on the panel for Clare’s PhD – she is studying face perception supervised by Andy. They’re interested in faces, and Clare’s thesis specifically brings together face perception and social psychology. As part of this she’s been investigating the way that first impressions arise from natural images. Andy, Clare, Isabel Santos and their colleagues had spent a long time collecting 1000 face images from the internet and having them rated for a wide variety of attributes. Using a fantastic program called Psychomorph, they had already made average images from faces that  were judged as having a certain social characteristic (such as perceived approachability, or intelligence), and shown how these averages still retained the characteristic in question. They had pointed out that the fact that image averaging methods worked so well implies that a set of consistent cues must underlie each type of judgement, because averaging largely gets rid of variations between images and leaves what is consistent.

These “averaged” images are pretty compelling, and the first time I saw them I was very struck the way that the most unapproachable-looking people seemed to have an air of grim desperation. How can you sense something like “grim desperation” just by looking at a face?

Grim Desperation?
Grim desperation? (from Sutherland et al., Cognition, 2013)

Because of the way they’d averaged the faces, by marking corresponding locations on each one, I thought we must have enough information to find which features were driving these impressions, and that we could use machine learning techniques that I knew about. So I suggested this as a project for Richard Vernon who worked incredibly hard with Clare’s advice to extract the information from the image database and to then to bring the information into models which could weight the different features to predict people’s ratings. Because of the huge variation in the images, we all expected this to be very difficult and inexact, so I was very surprised that it turned out to be quite straightforward and when we saw how well the simplest model worked, we realised we were onto something interesting.

Andy and I then realised it should be possible to try to visualise the stable relationships between facial features and social impressions, by reversing the modelling processes – going from specified social traits to pictures which would evoke those impressions. Richard and I did this in two steps. First we trained a set of neural networks to reproduce a set of coordinates given the 65 attributes we used to represent the shape of the face in our model. We could see this was working because we were able to reconstruct reasonably accurate and identifiable faces just from the attributes (you can see this in our own faces at the top of this article for examples). We then added a new layer to the input of the model which translated social trait ratings to attributes (based on the relationships in our 1000 images). Once we had done this we could generate cartoon-like faces which would be expected to produce specific impressions.

Finally, Andy pointed out that we could “close the circle” by giving these images as stimuli to a new set of raters to confirm that they saw them as we expected, as Clare and her colleagues had done with the original averaged photographs. This gives a useful test that the model has captured meaningful rather than accidental or chance properties of the data we used to create it.

What’s new about this?

Our work builds on previous research (see “Further reading” below) which has established that:

  • objective features such as age and sex can be accurately assessed from images
  • people largely agree in their judgements about more subjective characteristics such as extraversion, intelligence, trustworthiness, even though these impressions are of questionable validity
  • that these impressions can be summarised in terms of a few (~twothree) dimensions.
  • that they are formed very rapidly
  • that they influence our behaviour
  • that different images of the same person can evoke different impressions

There is also evidence from social psychology that the critical dimensions may be common across many cultures (“universal“) and that subjective judgements seem sometimes to be overgeneralizations of more objective patterns.

What’s new about the current work is that it shows for the first time that these impressions can be recovered from objective features in highly-variable everyday images; the sort of images we might see while browsing online, for example. Previous studies had mostly used highly standardised images, because image variability was recognised as an important constraint. For example, images used in previous studies would often be in a full-face pose, with standard lighting, and resized so that distances between the eyes were always the same. It was by no means clear how any cues to social impressions might be extracted from objective features in ordinary images. For example, this might be a very complex and indirect process, akin to understanding the meaning of a word from its image. In fact we found that, once we had measured the facial features, the first impressions could be predicted by simply weighting the measurements, and adding them up. This is surprising and it gives an indication that the brain processes underlying social judgements may be relatively straightforward compared to say reading.

Importantly, we also show how an accurate prediction of judges’ social impressions can be derived from these features (i.e., exactly which combinations of features are required to generate an accurate prediction from an arbitrary image). Of course just because our predictions are reasonably accurate, it doesn’t mean these appearance-based perceptions themselves are valid!

Finally, we generated synthetic face-like images that capture the relationships between objective features and trait impressions. By giving these images as stimuli to a new set of raters we confirmed that they saw them as we expected. This gives a useful test that the model has captured meaningful rather than accidental or chance properties of the data we used to create it.

Further reading

Oosterhof, N. N., & Todorov, A. (2008). The functional basis of face evaluation. Proceedings of the National Academy of Sciences, 105(32), 11087–11092. doi:10.1073/pnas.0805664105

Bruce, V., & Young, A. W. (2012). Face perception. London; New York: Psychology Press.

Jenkins, R., White, D., Van Montfort, X., & Burton, A. M. (2011). Variability in photos of the same face. Cognition. doi:10.1016/j.cognition.2011.08.001

Oldmeadow, J. A., Sutherland, C. A. M., & Young, A. W. (2013). Facial stereotype visualization through image averaging. Social Psychological and Personality Science, 4(5), 615–623. doi:10.1177/1948550612469820

Sutherland, C. A. M., Oldmeadow, J. A., Santos, I. M., Towler, J., Burt, D. M., & Young, A. W. (2013). Social inferences from faces: Ambient images generate a three-dimensional model. Cognition, 127(1), 105–118. doi:10.1016/j.cognition.2012.12.001

Todorov, A., & Porter, J. M. (2014). Misleading First Impressions Different for Different Facial Images of the Same Person. Psychological Science, 0956797614532474. doi:10.1177/0956797614532474

Willis, J., & Todorov, A. (2006). First impressions: making up your mind after a 100-ms exposure to a face. Psychological Science, 17(7), 592–598. doi:10.1111/j.1467-9280.2006.01750.x

Zebrowitz, L.A. & Monteparne J.M. (2008), Soc Personal Psychol Compass 2(3): 1497. doi:  10.1111/j.1751-9004.2008.00109.x

Advertisements

6 thoughts on “First Impressions Count, But How?

  1. Amazing, and interesting, but I sort of knew some of this? I’m a Transgerdered woman, and have wondered for years how I get “read” as not being a CIS gender woman. I’ve asked people, and they say you have a man look, but I don’t think I do? This could be really useful to TG people if you could pin down what says “Female” etc ? Would love to submit you a photo if you are doing further work ?

    1. I think we probably already have the data to determine which features say female or male. We’ve done some preliminary analysis of male and female faces, and of faces rated according to masculine/feminine cues. We should look more deeply at this data. However, I think this can be difficult territory for apolitical scientists to enter, since some will argue about the definitions of “male”, “female”, “masculine”, “feminine”, “sex”, “gender” and so on, so that whatever our analysis it might be seen as controversial. What seems interesting from our current results is that a single model, that makes no explicit distinction between male and female faces can still make accurate predictions about social impressions. This suggests that a common set of cues contribute to judgements of both male and female faces.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s