The Foreign Office has unlawfully hoarded more than a million files of historic documents that should have been declassified and handed over to the National Archives, the Guardian has discovered.
The files are being kept at a secret archive at a high-security government communications centre in Buckinghamshire, north of London, where they occupy mile after mile of shelving.
Most of the papers are many decades old – some were created in the 19th century – and document in fine detail British foreign relations throughout two world wars, the cold war, withdrawal from empire and entry into the common market.
They have been kept from public view in breach of the Public Records Acts, which requires that all government documents become public once they are 30 years old – a term about to be reduced to 20 years – unless the department has received permission from the lord chancellor to hold them for longer. The secret archive is also beyond the reach of the Freedom of Information Act.
The Foreign Office is not the only government department that has been unlawfully hoarding files. This month the Guardian disclosed that the Ministry of Defence was unlawfully holding more than 66,000 historic files at a warehouse in Derbyshire, including thousands of files from the army's Northern Ireland headquarters.
However, the Foreign Office's secret archive, which is estimated to hold around 1.2m files and occupies around 15 miles of floor-to-ceiling shelving, is believed to be far larger than the combined undisclosed archives of every other government department. One of Britain's leading historians describes its size as "staggering".
A basic inventory of the hidden archive gives a clue to its enormousness: batches of files are catalogued according to the length of shelf space they occupy, with six metres and two centimetres dedicated to files about Rhodesia, for example, and four metres and 57 centimetres holding files about Guy Burgess and Donald Maclean, the KGB spies who operated inside the Foreign Office and MI6. There are 50 metres of files on Hong Kong, 100.81 metres about the United States and 97.84 metres of "private office papers".
No length is given in the inventory for other categories such as Colonial Office files or records from the permanent under-secretary's department, the point of liaison between the Foreign Office and MI6.
The inventory says there is one bag of records from the Foreign Office's now notorious cold war propaganda unit, the Information Research Department. And buried away within the archive, wedged between files from the British military government in post-war Germany and lists of consular officials, are papers about the treaty of Paris, which concluded the Crimean war in 1856.
The Foreign Office's realisation that it would eventually need to admit to the existence of such a vast repository appears to have come at a time when its lawyers were waging a court battle with a group of elderly Kenyans. It was a battle that it eventually lost, with the result that it was obliged to issue an unprecedented apology and pay millions of pounds in compensation to thousands of men and women who suffered severe mistreatment during the 1950s Mau Mau insurgency.
During those proceedings the Foreign Office repeatedly denied the existence of a much smaller secret archive of 8,800 colonial-era documents, known as the migrated archive. It was eventually obliged to admit that this did exist, and that its contents corroborated the Kenyans' allegations about widespread acts of murder and torture by the colonial authorities.
As a first step, the Foreign Office gave its colossal secret a name, the Special Collections. Then last November the justice secretary, Chris Grayling, was asked to sign a blanket authorisation that is said to have placed the retention of the files on a legal footing for 12 months. No announcement was made.
Finally, a written statement about "public records" by the Foreign Office minister David Lidington was quietly issued in the Commons on a Friday afternoon. The statement included two sentences that referred to a "large accumulation" of documents.
As a result of the manner in which the matter was handled, the existence of the archive has remained all but unknown, even among historians. Anthony Badger, the Cambridge history professor who has been overseeing the declassification of the migrated archive, has written that he believes "it is difficult to overestimate the legacy of suspicion among historians, lawyers and journalists" that resulted from the concealment of those 8,800 files.
The discovery that the colonial-era documents are just a very small part of a hidden archive of more than a million files is certain to cause enormous damage to the Foreign Office's reputation among historians and others. A Foreign Office spokesperson said the archive had accumulated over time and that "resources have not been available to review and prepare" them for release.
The handful of historians who have become aware of the archive are deeply sceptical about this claim, however. Richard Drayton, Rhodes professor of imperial history at King's College London, said the size of the hidden archive was staggering, and it was "scandalous" that papers of such significance could be concealed for such a long time. "It's a working archive, for a department which believes it has a long-term, historic interest in many parts of the world," he said.
It was unclear whether there is any "truly explosive" material within the files, Drayton said, or whether officials were attempting to manage the country's historic reputation. "It may be that from the perspective of the state, 50 years is a short time. But the idea that the British state today has an obligation to protect the reputation of the British state of 50 years ago seems to me wholly inappropriate. It would be a manipulation of history, which we associate with iron curtain regimes during the cold war, regimes that managed and controlled the past."
Mandy Banton, senior research fellow at the Institute of Commonwealth Studies, said it was "extremely likely" that the archive had been culled to remove material that would most damage the reputation of the UK and the Foreign Office. Banton, a Colonial Office records expert who worked at National Archives at Kew, south-west London, for 25 years, said she had been "very angry" when she discovered that the migrated archives had been withheld. "I would have been incandescent had I learned while still working there. In lying to me, the Foreign Office forced me to mislead my readers."
Freedom of information campaigners believe that the hoarding of such a huge amount of papers is symptomatic of a culture of secrecy and retention at the Foreign Office and across many other UK government departments. Maurice Frankel, director of the Campaign for Freedom of Information, an NGO that works to ensure the Freedom of Information Act is properly implemented, said: "The FoI system depends on people knowing what they hold and being transparent about what they hold."
The archive is kept at Hanslope Park, a sprawling Foreign Office and MI6 outstation in the heart of the Buckinghamshire countryside. Sometimes referred to by Foreign Office staff as "Up North" – although it is only 60 miles north of London – Hanslope Park is also home to Her Majesty's Government Communications Centre, a facility where hundreds of government scientists and technicians develop sophisticated counter-espionage measures.
They include measures intended to protect the UK government and its allies from the sort of surveillance that Edward Snowden's leaks have shown to have been perfected by the National Security Agency and Britain's GCHQ.
Two wire fences, one 10ft high and topped with razor wire, encircle the cluster of buildings at Hanslope Park. Between them is a no man's land with intruder alarms. CCTV cameras are positioned every few yards and the entire perimeter is covered by floodlights. Inside, posters on the walls carry the half-joking warning: "Careless talk costs jobs."
Curiously, many of the offices are said to house row after row of typewriters rather than computers, with incinerators at the end of each room for the disposal of typewriter ribbons – a measure to reduce electromagnetic emissions, which can travel for hundreds of yards and be deciphered by foreign governments.
Hanslope Park is not only a highly secure facility, it is also a place that appears to be accustomed to handling – and destroying – large amounts of paperwork. This, possibly, explains why the special collections have been held there.
The blanket authorisation signed by Grayling put the secret archive on a legal footing for 12 months, during which time the Foreign Office is expected to devise a plan for its declassification and transfer to Kew. A spokesperson said a plan would be presented next month to a committee that advises the National Archives and the Ministry of Justice.
It will be quite a task. Declassification of the migrated archive has taken two and a half years, with the final tranche of documents due to arrive at Kew next month. At that rate, clearing up the special collections would take around 340 years.
