One of John le Carré’s boyhood memories is clutching his mother’s hand while waving to his father, who stood high up behind a prison wall. Ronnie Cornwell was a charming rogue, a confidence man who ran frauds and visited jails all over the world. He once sent the teenage le Carré to St. Moritz to talk a hotel manager out of an overdue bill — “and while you’re there, have yourself a steak on your old man.” Yet his love for his sons overflowed in guilty tears, and his longest con was to finagle expensive private educations for them. (He later sent an invoice.) As for waving to him in prison, however, Cornwell insisted that le Carré had misremembered. Cornwell had done a stretch at Exeter Jail, yes — but everyone knows that at Exeter you can’t see into the cells from the road.

Is the memory real? This question in various forms drives le Carré’s remarkable memoir, The Pigeon Tunnel: Stories from My Life. Le Carré began his career working for British intelligence and went on to revolutionize the intrigue genre with over twenty novels, like Tinker, Tailor, Soldier, Spy and The Constant Gardener. Now eighty-four, he finds that pure memory is elusive “after a lifetime of blending experience with imagination.” He believes what his father said, but he also knows that after that remembered day, a part of him never saw Cornwell wearing anything but convict’s stripes. One of the most haunting scenes of the book recounts the way the otherwise exuberant Cornwell would stand meekly at doors, waiting for them to be opened. As a prisoner, he had not been allowed to do this himself. And yet this memory is filtered as well: it comes secondhand, from le Carré’s mother, who abandoned him in childhood and was an enigma thereafter. It too is clouded by time’s opaque haze.

The Pigeon Tunnel is an episodic rather than a chronological memoir, jumping from scene to scene in le Carré’s fascinating life. The deeply affecting chapter on Cornwell is not merely the best part of the book, it may well be the best thing le Carré has ever written. Other sections are less personal. Le Carré avoids writing about his marriages, lovers, friendships, and children, focusing instead on the relationship between his work as a spy and his chosen life as a novelist. The two careers have much in common with one another and with his father’s line of work, he asserts, not least because they involve a complicated relationship to the truth. “To the creative writer, fact is raw material, not his taskmaster but his instrument, and his job is to make it sing.”

Parts of this book sing a little too much. Fans of le Carré’s intricate novels will recognize here his jargon-filled, world-weary dialogue and suspect embellishment. Such dialogue fills the chapter on spymaster Nicholas Elliott, who interviewed the traitor Kim Philby after Philby defected to the Soviet Union. Similarly, le Carré seems to make a good story even better as he recounts trying to collect a debt for his father from the Panamanian ambassador to France at age sixteen. He writes that the ambassador’s wife, “the most desirable woman I had ever seen,” played footsie with him under the table and then nibbled on his ear as they danced into the night. Such print-ready scenes are evocative but not entirely believable, and once again implicate the fraught relationship between memory and the creative act. Le Carré’s biographer Adam Sisman contends that le Carré “enjoys teasing his readers, like a fan dancer, offering tantalizing glimpses, but never a clear view of the figure beneath.”An air of mystery suits a thriller writer — especially one who used to be a spy.

The Pigeon Tunnel shows that le Carré is at his best not when he renders scenes or snappy dialogue but when he simply observes. He has a marvelous eye. His diplomatic cover in the 1960s required him to escort dignitaries from place to place, including translating for a German politician meeting with Harold Macmillan. The British prime minister’s patrician slur “was like an old gramophone record running at a very low speed,” le Carré writes. “A trail of unstoppable tears leaked from the corner of his right eye, down a groove and into his shirt collar.” Le Carré briefly describes his great-grandfather, “whom I remember as a white-bearded D. H. Lawrence lookalike riding a tricycle at ninety.” Best of all is a surreal meeting with Yasser Arafat, which occurred while le Carré researched the Israeli-Palestinian conflict for a novel. As they embraced, le Carré sized up his man, whose brown eyes were “fervent and imploring.” Arafat’s famously patchy beard “is not bristle, it’s silky fluff. It smells of Johnson’s Baby Powder.”

The meeting with Arafat illustrates a central concern of le Carré’s working life: his commitment to research. It is an ironic preoccupation for a self-confessed fabulist. Although his early spying looms large in the public imagination, le Carré has gathered far more material for his novels during civilian trips to dangerous places. Many of the memoir’s chapters recount these adventures. He spent time in the eastern Congo and Khmer Rouge Cambodia; he interviewed Russian oligarchs and Middle Eastern terrorists. In every town he tried to find the watering hole where spies, diplomats, journalists, and men of fortune sought comfort and camaraderie. These settings and characters worked their way not just into his fiction but into his consciousness; they have set his novels apart from all other stories of intrigue. “An old writer’s memory is the whore of his imagination,” he confesses, late in this book. That arresting quote might well have been his epitaph.