Tuesday, October 30, 2007

Certainty in sight records

In late August 1982 I walked into the weedy fields at the South Cape May Meadows in search of rare birds. Among many species that I had in mind as possible "prizes" that day was Loggerhead Shrike and, lo and behold, a scan of the bushes ahead revealed a small, white-breasted, dark-masked bird perched conspicuously on top of a slender post - a Loggerhead Shrike!

I looked for a couple of seconds and then quickly moved closer hoping to sit down to study and sketch this rare find, only to discover that it was gone. Not only was the shrike gone, I couldn't even find the post it had been on! In place of both was a Great Egret calmly hunting the grassy edge of a pond.
Such is the power of suggestion. I thought I might be rewarded that morning with a sighting of a Loggerhead Shrike, and I managed to create the vision I desired out of the pattern of shadows on a Great Egret, complete with dark mask, long tail, and the right shape and posture.

If the situation was different and I was only able to see it from a distance for those first few seconds, it's very possible that I would have convinced myself that what I had seen was real and that I would have reported seeing a Loggerhead Shrike. And if anyone had questioned it I would have said I was absolutely certain. After all, I was an experienced observer and I saw multiple diagnostic field marks. What else could it have been?

This is the fundamental problem with all of the recent Ivory-billed Woodpecker sightings. Claims of certainty and lists of diagnostic field marks are simply not as meaningful when they are based on such brief views.

Proponents still emphasize the number of sightings and the fact that some auditory and visual encounters are clustered. They ask "What are the chances that all of those people were mistaken?" Referring to one of his own brief sightings, Geoffrey Hill asks "What are the chances that just as I was misidentifying a Pileated Woodpecker as an Ivory-billed Woodpecker, I would hear another pileated give a double-knock?"

First of all, these questions are spurious and attempt to shift the burden of proof to the skeptics. It is not up to the skeptics to show how all observers could be mistaken. The simple answer is "Yes, that is possible." There is no proof and whether these observations are more likely to be correct or incorrect is simply a debate over percentages.

I would argue that the chances are very good that one misidentification led to another, since these events are not independent. Here's a possible scenario:
An observer hoping to see an Ivory-billed has an ambiguous view of a bird flying away. In the moments after, while processing the flickering black-and-white pattern of the wings and while most susceptible to suggestion, a single double-knock-like sound is heard. The sound (even though it too was ambiguous and was clearly not produced by the bird that flew away) helps cross a decision threshold - that Ivory-billed is likely, that the white really did seem to be on the trailing edge of the wings, and that the bird that just flew away must have been an Ivory-billed.

That decision in turn influences the perception of the double-knock-like sound, which then seems less ambiguous and "must have been" a second Ivory-billed. And as the observer reconstructs memories of the event and adds other subtle impressions to support the identification ("that was no Pileated Woodpecker!"), a circular reinforcement occurs. The retrospective perception of the wing pattern and sound actually change as positive elements are replayed and negatives ignored. The more certain the wing pattern seems, the better the double-knock sounds, which reinforces the interpretation of the wing pattern, and so on.

This may not be exactly what happened in this case, but all of these effects are well-documented in psychological research.

No intentional falsification or fabrication is needed, simply a subconscious selection of evidence supporting the favored conclusion, and a subconscious omission of refuting evidence. This generates false confidence. Once the perception is formed and "confirmed" it becomes nearly immune to question or revision. Claims of certainty and "multiple field marks seen" must be judged in the context of the situation. Longer and better views of a bird require less interpretation and give the observer more information and more opportunity to correct mistakes. Views as poor as all of the reported Ivory-billed sightings are far from certain.

This does not mean they should be ignored, and they have not been ignored. The reported sightings have inspired and guided massive search and conservation efforts in the last three years and before. Sightings should continue to be carefully reviewed and followed-up, but we have to be realistic about the strength of those sight records. If unprecedented search efforts fail to find what a few people glimpsed three years ago, it might indicate that those observers were mistaken.

15 comments:

birdchaser said...

I nominate this post for best birding blog post of the year!

Luneau Atheist said...

The view that making an identification error during a few seconds of viewing is "just not possible" shows either a lack of experience or a failure to learn. All birders who have spent any real time in the field have made errors like this, so the opportunity to learn is there.

My personal definition of a better birder is not synonymous with a birder who is fast at ID. It is a birder who understands the potential for errors and tries to adjust accordingly.

I've known birders who were very fast and very good, and this display of skill made them stand out. The problem was that their error rate was higher than other birders that were slower, more methodical, and not so quickly definitive. So despite having great field skills, their judgment skills fell short. They may have been the "hot" birder, but they failed to be one of the better birders. Some of the most reliable birders I know would not be considered "hot". The truly gifted birder is both.

David Sibley said...

Rob and Luneau Atheist -
Thanks for your comments.

Luneau Atheist, if you must use a pseudonym here, I'd appreciate it if you didn't adopt the name of any real person involved in the debate. That seems a bit provocative. But please keep commenting, you make an excellent point.

Chris W said...

You certainly have a point here.
I had a very similar experience just recently. I was scoping a pond in the southern part of my county and found all the usual shorebirds. Gr & Lesser Yellowlegs, SemiP sand, Least Sand, ect...
The prize that day would have been a Wilson's Phalarope. Anyway, I was scanning the pond when I came across a bird that looked very similar to one. So similar that I immediatly yelled out: "Wilson's Phalarope".
Upon closer inspection, the bird turned out to be a Gr Yellowlegs wading in deep water. The bird had been turned slightly so that I couldn't see the bill and the movements it was making tripped the ID switch in my mind that said Phalarope.
With your story here you make a very good point that even the most expert among us can still make mistakes.
Happy Birding! --Chris

Unknown said...

I don't think it counts as an ID until you have finished the process. So, for me, initial mis-identifications are no big thing. In the case at hand, you thought you had what you were looking for, moved in for a better view and discovered it was something else. No big deal.

On the upside of the process, the knowledge of what is possible in a given place at a given season more often results in noticing birds that might have slipped by simply because you are thinking about them and taking the time to look.

Case in point, one August at Brigantine, I thought I ought to scan the bay once for out-of-season loons. Sure enough, there was a Red throated Loon. Had I not been diligent, I wouldn't have seen it, not that the bird would have cared but I felt good about my birding "chops".

David Sibley said...

David -

You commented "I don't think it counts as an ID until you have finished the process". So the mistakes that Chris and I describe are "No big deal" because we were able to correct them in the course of the process. But how do we decide when the identification process is finished?

When we find out after a snap-judgment ID that we were wrong, we generally think "what a silly mistake! How could I have been so careless...". In retrospect then we dissect the process and go over all of the things that were wrong with our first impression, emphasizing why we should have been uncomfortable with the identification.

This promotes the idea that we have extended the identification process, taken a second look, and been good careful observers, when usually the reality is that we saw some unwelcome facts that we could not reconcile or ignore, and had to start the identification process over.

My shrike story and Chris's phalarope story both describe silly mistakes that were almost impossible to ignore, but lots of other mistaken bird IDs are not so obvious. In most cases the identification process ends after the first snap-judgment, and unless some contrary feature is observed, we consider that a successful end result. The identification process generally ends when we're happy and comfortable with the result, not after a rigorous and objective study.

I agree completely with your other point - that the pattern-recognition system that leads to these mistakes is at the same time our most powerful tool for successfully finding and identifying birds. Which leads to the larger point, that we can't prevent these kinds of mistakes, we can only be aware and use extra caution when they are most likely to occur, and accept the fact that a lot of our identifications are something less than "100% sure".

Unknown said...

Indeed, finishing the process is the variable. So, maybe we get to a spot (closer, better light, song) where we now have enough information to be confident. Or, the bird flies or skulks off and we never get any more data. So, we are uncomfortable enough to run everything we saw through our minds against everything we know and we come up with the most satisfying judgement. Usually, accompanied by a phrase such as "Damn, I wish I had a better look but if I had to call it, I would call it a .....".

This is half the fun of the whole thing. Phalarope, shrike and out of season loon sightings are not important in themselves. A good time had while puzzling out an ID on a minimal amount of information is the point. What could be a more pleasant way to spend a few hours?

Bennet said...

Both Davids seem to be making similar points but from opposite ends. David Sibley is reminding us that the end of the ID process is usually not a firm and final one, while for the other David (David Simply?) tis the process that's the thing, the goal, the reason for trying.

I think it's obvious that for both parties, myself, and for most if not all birders, it is the struggle to corral a finite number of clues gathered in a short (usually) amount of time into an ID that gives us the most pleasure from birding. We don't want to be the boy or girl who cried "Phalarope", but we needn't have to measure every primary, either. The beauty of birding is the balance between limiting our options based on available clues, and recognizing that there are more clues and more options that we have to teach ourselves still.

I'm a mathematically minded person, I must admit. To me, it comforts me to know when I'm birding that there must be an answer to what kind of bird I'm looking at. I may be frustrated if I'm not able to come up with that answer, but there is this creature I'm looking at and it is something. If I convince myself that it's something that it is not, well, I hope that I find out what it truly is, but, like David Sibley was saying, to paraphrase, I can't let it stop me that there is a chance that I'm wrong nor can I expect my ID to be unassailable. And if I never find out what it is, as in David Simply's hypothetical case, I hope I enjoyed the attempt.

And may I say this has been a thoroughly enjoyable post and comment thread.

David Sibley said...

Emupilot,

After careful consideration I've decided not to publish your comment submitted Friday at 4:57PM. Your tone is too personal and confrontational, and you simply misrepresent my position and then accuse me of "impugning" specific people. I have tried to be respectful of all parties.I believe that all the sightings are honest mistakes.

I've already tried to make clear in my posts that the most important point is not the details of individual sightings, but the fact that sight records are simply NOT verifiable evidence. They could be correct, they could not. There is NO WAY TO KNOW, regardless of who reports them or how well the bird was seen. Please read Gilovich or Risinger et al., and then you will understand how "fuzzy" human perception is, and how easy it is for observers to make honest mistakes.

If an Ivory-billed is photographed tomorrow, that would add credibility to the sightings but still wouldn't prove that past sightings were correct. That is just a fact. Since the scattered sightings over the last 70 years have failed to result in any confirmation, that leads me to the conclusion that those sightings were most likely mistakes.

My skepticism is a result of the lack of confirmation, not the cause. Everyone is free to follow their own convictions and I can respect your decision to continue searching, of course, but you can't expect all to share your faith.

emupilot said...

I am not contending for a minute that sight records constitute proof of the Ivory-billed Woodpecker's existence. A blanket assertion that all sightings were mistakes tries to avoid unconfortable accusations which are inherent in such as assertion. David Kulivan had an extended look at close range of a pair of Ivory-billed Woodpeckers and noted all the field marks. This is not a circumstance under which a "mistake" can be made with such a distinctive bird.

I have no problem with anyone who is not convinced of the bird's existence, only with those who assert the opposite: the bird's probable or definite extinction. That is because such an assertion is personal toward those who have reported the bird. It is not possible to separate the assertion from what it would mean about the integrity of Kulivan or the professionalism of many others. Perching on the fence with the Ivory-bill, considering its status unknown, allows one to express scientific skepticism while respecting the integrity and professionalism of the observers.

David Sibley said...

Emupilot,
You seem to have some things confused. If you are "not contending for a minute that sight records constitute proof", then you can't turn around and imply that some sightings are unquestionable. All sightings are questionable.

You still insist (falsely) that questioning the sightings is automatically "impugning" the observers. You seem to reject the idea that people can make honest mistakes. This leads you to suggest censorship - the skeptics should keep quiet simply because you think skepticism is disrespectful to the observers!

I have assumed that all the people involved are fundamentally honest and professional, so other than treating them with respect the observers' personal feelings have no bearing on the evidence. What is relevant is the actual physical evidence, and my primary obligation is to assess that carefully and objectively and to report honestly. Nothing personal.

Paul said...

Emupilot said that people who assert the extinction of the Ivorybill insult those who have reported the bird, that is, such an assertion is "personal". Unfortunately, this is true in the case the observer believes his/her role is to vouch for the truth of the sighting. Records committees must confront this confusion all the time. Many observers believe their role is analogous to the role of a Notary Public: I hereby certify that I really did see an Ivory-billed Woodpecker. Naturally, such an observer would be personally insulted by a skeptic.

In very few cases have I seen this problem fixed by counseling the insulted observer.

emupilot said...

When a records committee decides against accepting a record, they don't indicate the observer probably saw something else, just that the observation isn't verified. That is the distinction between questioning and rejecting which seems to be lost on many skeptics. There is, of course, nothing wrong with questioning. Professor Van Remsen famously questioned the heck out of David Kulivan. That his sighting survived the line of questioning doesn't mean the sighting is verified or that others can't independently question the sighting, just that his observation appeared genuine and was consistent with Ivory-billed Woodpecker.

The Kulivan case is the ultimate test of Ivory-bill skepticism. Kulivan is given the presumption of being fundamentally honest and professional, so his report must accurately reflect what he perceived. His observation lasted for 10 minutes, was at close range and included views from multiple angles of a pair of birds, so it is clearly not an "extremely brief glimpse" subject to the frailties of human perception fooling the observer. For that matter, he was not looking for Ivory-billed Woodpeckers, so there would be no expectation bias anyway. That leaves plain old misidentification, no doubt for a Pileated Woodpecker. Kulivan noted (at least) extensive white on the folded wings of the perched birds, white bills, one bird had a red crest while the other had a black crest curled forward, and they made calls which sounded like tin horns. I don't own the Sibley field guide, but I doubt it indicates that any of these features, much less all of them, are consistent with a pair of Pileated Woodpeckers. Nobody needs to take Kulivan's word for it, but granting Kulivan's honesty and then saying the birds he describes were probably Pileateds (as it must be for the Ivory-bill to probably be extinct) makes no sense at all.

Falco columbarius said...

emupilot said... When a records committee decides against accepting a record, they don't indicate the observer probably saw something else, just that the observation isn't verified.

That's not necessarily true. The records committees that I've participated in allow for a decision that the evidence more likely supports a different identification. I've seen this happen with photos that were mistakenly identified by the person submitting the record. The decision by the committee is actually a rejection of the evidence rather than a statement that the evidence is inconclusive.

I personally have not voted on a record where this was done with a written record, but then most written records are rejected for lack of details or conflicting details.

Anonymous said...

"...so his report must accurately reflect what he perceived. His observation lasted for 10 minutes..."

One of my best friends in the world once told me a truly amazing story. For years later he didn't really want to talk about it, which I thought was strange because I was so impressed. Many years went by until he shamefacedly told me he had made it up for a joke, then was too embarrassed to admit the truth. Kullivan's sighting was, of course, on April Fool's Day. That kind of thing happens commonly, a lot more often than the rediscovery of presumably extinct species.

Another issue is his sighting lasted ten minutes. TEN MINUTES! After five minutes or so, who would be afraid of possibly spooking the birds by using a camera?? And if he could watch a PAIR for ten minutes, how come nobody can see one for long enough to get a photo? Of course there are photos from back when the bird was alive.

"For that matter, he was not looking for Ivory-billed Woodpeckers, so there would be no expectation bias anyway."

Actually, according to this article there was: http://freakonomics.blogs.nytimes.com/2007/08/15/the-lord-god-bird-and-the-power-of-suggestion/

Wright [Kullivan's professor] was in many ways already a believer in the continued existence of ivory-billed woodpeckers. He’d been fielding reports of sightings for twenty years, and though he himself had never seen the bird, he firmly believed it was out there. It was Wright who had told his class about the ivory-bill, along with several other animal species presumed extinct but still rumored to live in the heart of the swamps and forests of Louisiana. In some sense, he had prepared his students for a sighting by telling them the bird was still out there, which, depending on your point of view, increased the likelihood of a credible sighting or diminished it by planting the image of the bird already in his students’ minds.

Almost nobody sees an apparition of the Virgin Mary without first having a mental image of what she might look like. On the other hand, a great deal of birding is based on knowledge acquired before you go into the field. This paradox is amplified a thousand times when birding for extinct birds.