By Andrew Lownie

I have never considered myself as a dangerous radical or enemy of the state and my background and activities would not suggest it – I have been a Cambridge history fellow, Parliamentary candidate for the Conservative Party and been a member of several smart London clubs. Yet I’m being spied on by the State.

The monitoring by the Cabinet Office and Foreign Office includes my social media accounts, collecting a flyer for a talk I gave at a private club, details of a lecture at a Cambridge Alumni weekend and a library talk with an internal heading by the Cabinet Office of ‘Not just any cook-along this week’. I know this from various Subject Access Requests under data protection laws to those two departments relating to the personal data which they hold on me and which I have applied for over the last few years.

Though public authorities are required by law to supply such information, both departments have been reluctant to do so, citing a series of excuses ranging from costs compliance to vexatious behaviour. The Foreign Office, and only after seven months and the intervention of the Information Commissioner who regulates Information Rights, provided four brief mentions covering the period March 2021 to March 2022 when I had requested all data held over the last five years, a period which included repeated FOI requests, Parliamentary Questions and several Information tribunals.

The Cabinet Office eventually admitted that they held so much material on me for the past five years – they estimated it would take over 656 hours to collect the information – that my requests needed to be broken down into six monthly intervals. Their releases showed that my activities were brought to the attention of the Permanent Secretary, Alex Chisholm, and the ‘Cabinet Office COPRA team’, that my speaking engagements, newspaper articles and crowdfunding activities were monitored and that information was also collected on other parts of my life. This included employment tribunal and defamation cases which I had successfully defended, and which had nothing to do with my FOI requests or activities as an historian. The inference was that this information might be useful to smear me.

My crime? As an historian to push back against the censoring of our history by government departments and to highlight their failures to adhere to various Public Records Acts and the Freedom of Information Act.

My concerns about historical curation go back to researching a biography of Guy Burgess over a decade ago, where I found huge gaps in the record. There was nothing on his time in the Information Research Department, a secret unit set up at the beginning of 1948 to counter Russian propaganda and which he betrayed months after it was set up. Likewise, nothing on his time in the News Department, in the private office of Foreign Secretary Ernest Bevin’s deputy Hector McNeil nor the British Embassy in Washington between 1950-51 – though there were papers for the period either side of his time in Washington for diplomats doing the same job. In historical parlance this is known as ‘dry cleaning’ the records.

My suspicions about Establishment cover ups were further confirmed when I began researching a book on Dickie & Edwina Mountbatten, the last Viceroy and Vicereine of India, in 2016. Their letters and diaries had been extensively quoted in previous books and a major fundraising campaign had been mounted by Southampton University in 2010 to buy their papers so they could be ‘open to all’.

I was therefore surprised to be told by the Southampton archivist that they knew nothing about these diaries and letters, part of a £2.8 million purchase under the Acceptance in Lieu scheme and with contributions from the Heritage Lottery Fund, Hampshire County Council and other organisations.

Eventually after several years, numerous FOI requests, the intervention of the Information Commissioner and the unprecedented threat of contempt proceedings against Southampton University, in 2019 a Decision Notice was issued ordering the release of the material.

Southampton and the Cabinet Office appealed the decision but then, just before the November 2021 hearing, dumped 99.9% of the material (over 30,000 pages) on the internet. The material that they had kept closed for a decade, and fought so hard to prevent being made publicly available before my book The Mountbattens: Their Lives and Loves was published in 2019, proved to be entirely innocuous.

And they knew this because in March 2018, some eighteen months before they appealed the Information Commissioner’s Decision Notice, a review of the material organised by Southampton and the Cabinet Office had concluded there was nothing sensitive in the personal diaries and letters.

The tribunal, however, ruled that the Cabinet Office still had the right to apply FOIA exemptions to the diaries and letters which meant that just over a hundred redactions – some a single word, others several paragraphs – were applied on the grounds they were communications with the Sovereign, or that they would damage International Relations or National Security.

Until just before the four-day hearing in November 2021, Southampton had argued they were bound by the mysterious Ministerial Direction controlling the letters and diaries, but they dropped this argument saying that simply specific FOIA exemptions would be applied. On such grounds the Cabinet Office had no role to play in the hearing, no right to ‘review’ anything or to dictate to Southampton what to do with FOIA, but this was ignored by the tribunal.

There was no evidence that the diaries and letters had ever been ‘closed’ – neither the Cabinet Office nor Southampton could cite a specific notice but tried to argue that by implication they had been caught by the ‘undertakings’ concerning Dickie’s official papers in agreements in the 1960s and 1980s, but this could not be. The diaries and letters are expressly defined as AIL Chattels in the 2011 Purchase Agreement – not ‘Excluded Records’ i.e. the papers that the Cabinet Office had closed. The 2011 agreement expressly stated that the vendors were free to sell all AIL Chattels and that they are not subject to the Undertakings. Thus, the proviso in the Ministerial Direction couldn’t apply in any event to AIL Chattels. All this was ignored by the tribunal.

The upholding by the tribunal of various requested redactions is also baffling. Some names from the Royal Household were redacted – even if they were already in the public domain from the London Gazette or other books or, indeed, unredacted on other pages of the diaries. Other similar roles were not redacted so there was no consistency on how the FOIA exemptions were applied.

The tribunal also ruled on the grounds of National Security that a reference in Lord Mountbatten’s 1943 diary to an intelligence organisation under Captain G.A. Garnon-Williams should be redacted even though his P Division has extensively been written about and there are numerous references to him and his organisation in the database of the Mountbatten papers.

It was also ruled that a reference to the Pakistan leader, Muhammad Ali Jinnah, in Edwina’s private diary should be redacted on the grounds that it would be prejudicial to relations with Pakistan even though the test is that ‘The public authority must show that there is some causative link between the potential disclosure and the prejudice and that the prejudice is real, actual or of substance. The harm must relate to the interests protected by the exemption.’ It is hard to believe that such a reference in a private diary seventy-five years ago would still damage relations with another country, especially when there is plenty of evidence already in the public domain from books about Edwina or by family members drawing on their access to the diaries, that Edwina had a low opinion of Jinnah. Aren’t India and Pakistan entitled to know what the Vicereine thought in this anniversary year of Independence? If not now, when?

This FOIA exemption – section 27 – has a public interest test and Southampton even then had no obligation to apply it but they did so. An FOIA exemption is available to a public authority in respect of any FOI request but it is not bound to plead it; the authority has a discretion – unless providing the info would otherwise be unlawful (under Data Protection Act, Official Secrets Act etc). Southampton were perfectly free to publish the material that allegedly would damage relations with India/Pakistan but Southampton choose not to do so in what looks like an academic institution censoring history.

It is quite clear, contrary to the Cabinet Office and Southampton’s claims, that the diaries and letters were open when purchased – they would not have satisfied the Acceptance in Lieu scheme otherwise – and that the reasonable course of action would have been to review the collection to see what could be released when it was acquired in 2011 and not only after they had been forced to do so a decade later. This could easily have been done by experts at Southampton. Instead, through to the hearing in November 2021 Southampton claimed that all the diaries and letters were so sensitive they had to be closed, that digitalisation would take years, the material was illegible and fragile etc. None of this was true.

Access to the diaries and letters before my book was completed would have made my book richer and more nuanced and, as I was not awarded my costs, it left me personally with a legal bill of over £400,000. No private individual should be financially penalised seeking access to material which was purchased with taxpayers’ money on the basis that it would be open to the public, but that is the position I now found myself in.

Millions of pounds of public monies were spent purchasing the total Broadlands Archive (even though we don’t know exactly what was apportioned to the diaries and letters) to make this important collection publicly available. And then, given that Southampton and the Cabinet Office deployed two top QCs and a plethora of lawyers, probably well over £1 million has been spent suppressing them. However, neither Southampton nor the Cabinet Office will say, even after Questions in Parliament and FOI requests, how much public money has been spent on pursuing this needless appeal against the Regulator.

This was only the start of my problems with officialdom. After I discovered a wartime FBI file which claimed Mountbatten was ‘a homosexual with a perversion for young boys’, I requested other listed files held on him, only to be told they had been destroyed. When I asked when that destruction had taken place, the American authorities candidly admitted, “After you had asked for them.” Clearly this had been at the request of the British Government, previously unaware that such damaging material existed.

The Irish police, the Garda, accepted that they had car logs for the visitors to Mountbatten’s holiday home in Ireland for August 1977, the month two sixteen-year-old boys claimed he had abused them, but they would not release them on the grounds that they were part of the investigation into Mountbatten’s murder – which took place two years later.

Even though we now have a twenty-year rule for deposit of historical records, I found that no files on Mountbatten’s 1979 murder had been deposited in archives, either in Ireland or Great Britain. The Garda claimed it was still ‘an active investigation’, even though the bomb maker had been convicted, served a sentence and was released under the Good Friday Agreement in 1998.

Indeed, many of the files relating to Mountbatten’s funeral, seen by millions around the world on television, are closed because they reveal sensitive information about the procession route, who sat in which carriage etc.

For my next book, researching the Duke of Windsor’s time in the Bahamas during the Second World War, I discovered that, while the Colonial Office Files in the National Archives were thin on him, there were mirror copies of the files in the Bahamas. These were much more extensive and full of revealing detail – such as the Duke posting the Commissioner of Police to Trinidad on the morning of a murder which the Duke wanted covered up.

Last year I requested a 1932 police protection file relating to the Duke of Windsor. Dozens of similar files have been available at the National Archives for twenty years. They contain useful titbits on the then Prince of Wales’ movements but nothing remotely secret. The Metropolitan Police refused to release the file on the grounds that it would jeopardise the present safety of the Royal Family.

That decision was upheld by the Information Commissioner’s Office (ICO) so I took the matter to a tribunal. A judge asked if I would supply examples of information from other protection files of the period but, when I sought to do so, I discovered that the twenty files I had highlighted in my submission, and which had been publicly available for over twenty years, had been withdrawn from the National Archives. They included MEPO 10/35 which reveals Wallis Simpson’s affair with a car salesman called Guy Trundle, which has been copied and quoted numerous times by historians and is published in all its juicy detail on the website of the National Archive. Yet historians cannot look at the original file.

No terrorist has mounted an attack after spending hours wading through such files, yet on no evidence whatsoever the file was closed. Incidentally, I was told by the Special Branch weeder that there were dozens of other Special Branch reports on Edward and Wallis but only this representative file had been preserved. The others were not deemed worthy of preservation. Says who?

That relatively simple tribunal was held in November yet there has been no judgement. Justice delayed is justice denied.

I looked into writing a biography of Prince Philip to find almost all the files, even after his death, remain closed though I had some successes in securing releases. These included a file relating to his application to join the Royal Navy in 1938 consisting of written reports by his headmaster, briefings for various foreign trips going back to the 1950s and two pages were released relating to the Duke of Edinburgh Award scheme in India in 1983. Quite why these files were ever closed in the first place beats me. Other requests failed such as files on his tour of South America in 1975, a briefing for a 1967 visit to Greece and a 1975 tour of Poland.

For my next book on Prince Andrew, I have given up FOI requests because, though some files are listed at the National files, often relating to foreign trips which were widely reported at the time, I know that data protection will be used to deny access.

The preservation of royal records is a real problem as the division between family and state records is unclear. It is known that Princess Margaret burnt huge quantities of the Queen Mother’s papers. The reputable author Christopher Wilson gave up writing his life of the Duke of Kent’s father after being refused access to his papers at Windsor. The Royal Archives give no access whatsoever to files on the reign of Elizabeth II, which include correspondence not just with prime ministers of the UK but premiers and governors-general of the Commonwealth realms. They also decide which historians they want to let in or not. Historians wishing to gain access to files from previous reigns are obliged to sign a form to say they will inform Buckingham Palace how any material will be used. Cameras are forbidden and there is no public inventory – rather like a restaurant with no menu.

Recently the campaigning organisation Index on Censorship published a report on censorship of Royal records pointing out that almost 500 files at the National Archives were closed including:

Royal Family flying training 1977 -1978. Record opening date 1^st January 2066.

Family name of Royal Family members 1952-1960. Record opening date 1st January 2027

Remains of the Russian Royal Family 1993 Jan 1 – Dec 1993. No release date.

Family name of the Royal House 1952. Record opening date 1st January 2053.

Air travel for the Royal Family: Containing information relating to the financial arrangements for and other matters relating to the Royal Family 1936-1952. Record opening date 1st January 2053.

Visits overseas by members of Royal Family 1954. Record opening date 1st January 2055.

Declassified recently reported that over 200 files on overseas trips made by King Charles going back to the 1970s remain closed. They include a 1983 visit to Australia which will only be released when Charles is 121 years old.

Historians cannot look at important historical material from almost a century ago, yet Prince Harry can spill intimate secrets from a few months ago. As the former MP Norman Baker, author of And What Do You Do? What the Royal Family Don’t Want You to Know has said; ‘There’s no reason for these to be kept secret. The normal excuse given is that it’s to uphold the dignity of the crown. But the dignity of the crown is upheld by them not behaving in an undignified manner.’

There are lots of techniques used by public authorities to avoid disclosure. They can kick the can down the road as long as possible, sometimes amounting to over a year. They can keep changing the exemptions deployed as each is addressed and shown not to apply. They can simply not answer requests and hope the requestor gives up. They can play with semantics in carefully phrased replies which are economical with the truth. They can agree to release documents and then do nothing or redact them so heavily as to make them worthless. They can aggregate separate requests and then refuse on grounds of costs of compliance. They can claim a request is vexatious or deploy FOI exemptions without a Public Interest Test and which cannot therefore be challenged.

Time and time again, authorities hide behind national security or law enforcement or claim not to have material, only to miraculously find it when evidence of its existence is presented. Intriguingly, only the most sensitive documents are ever affected by damp or asbestos. A favourite trick is to use Section 22 (where the information is held by the public authority with a view to its publication, by the authority or any other person, at some future date) but where the material mysteriously never finds its way to the National Archives. A weeder has personally told me that when in doubt reviewing material, they are told to just use an absolute exemption, such as Section 23 National Security.

My experience with Mountbatten is a good example of public authorities, often pleading scant resources when responding to FOI requests, yet deploying costly lawyers to battle invariably under-represented requestors and to try and break them financially. It is clear from the recent House of Commons Public Accounts Committee report that the Freedom of Information Act 2000 – both in terms of legislative reach and enforcement power – is simply not fit for purpose and Parliamentary unease at the antics of the Cabinet Office and the weakness of ICO as a regulator is justified.

There are too many loopholes (“exemptions” which are very broadly drawn), and the government – particularly the Cabinet Office – has become extremely adept in exploiting them. They include from my own experience:

(i) The licence given to public authorities to entirely change their reasons for refusing to disclose information at almost any stage – however late – in proceedings. This leads to what could be described as the absurd game of ‘whack-a-mole’ – and of course drives up costs. In my case the Ministerial Direction was used to justify closure for a decade until I questioned its existence.

(ii) The lack of any mechanism to ensure public authorities adhere to deadlines, or even Decision Notices. Because they can ignore these with impunity, researchers are faced with the prospect of incurring costs to bring a delinquent body into line. My lawyers constantly had to chase both Southampton and the Cabinet Office for responses, even though they had statutory time limits, to the extent of bringing contempt proceedings. All this cost me money which was their aim.

There is scope for some simple reforms such as:

• Statutory deadlines for an authority to respond. There are, for example, no enforceable deadlines for Internal Reviews which should take no more than forty days. It wouldn’t be difficult to tweak FOIA to include unequivocal – and actionable – timetables across the process.
• “Deemed refusals”. Scotland’s FOIA includes this provision, by which the absence of a response within the required timetable is taken as a formal refusal, which can then be appealed by the applicant.
• Sanctions for failures to comply with timetables. Public authorities, routinely flout deadlines – whether statutory or in ICO guidance. The way to discourage this is by an automatic financial penalty – payable to the applicant – for every deadline missed. Train companies (for example) are now required to pay passengers what amounts to a fine for failures to arrive on time: there is no reason why Whitehall should be any different.
• Severely reduced licence for public authorities to ‘change horses midstream’. If the Cabinet Office pleads Section 22 at first FOIA request, it should not be allowed to amend that to a different exemption without the explicit permission of the regulator. And the bar for being allowed to do so should be set extremely high, with accompanying statutory requirements for the disclosure of evidence supporting any such request.

The culture of cronyism needs to go. Either archives are secret, or they should be made available to everyone. Tame journalists are often tipped off about document releases well in advance of the rest of the media and there are a select number of writers who are given privileged access to write commercial books.

There needs to be proper, separate oversight. Internal reviews are conducted by the same department, and, from my experience, they have all upheld the original decision. There is the Advisory Council on National Records and Archives Committee, but it has little power and its members appointed by the DCMS. It needs to be replaced with a much more robust body and given stronger powers.

The ICO requires more money and staff, it needs to be truly independent of the Cabinet Office (who are the worst abusers of FOIA), and it has to be prepared to use its enforcement powers. Indeed, I believe the ICO should be left with just its data protection role and a new regulatory for Information Rights set up.

There also needs to be a sea change in attitudes in Whitehall. I have seen very senior civil servants lie to the media, Parliament and my lawyers. After the former Foreign Secretary, Dr David Owen, called on the Cabinet Secretary to reimburse my legal costs on the Mountbatten case, he was fed a tissue of lies by a senior member of the Cabinet Office. The weeders need to have a lighter touch and FOI requests need to be dealt with more quickly while a rather more enlightened attitude must be taken towards FOI exemptions to really protect what is important.

The balance between accountability and transparency on the one hand and protecting national security on the other is a difficult one to strike. Once records are released the genie is out of the bottle, but it is hard to argue that records, which in many cases are over 60 years old and where the officials involved are dead, should not be released. If our history is to be written accurately, we will have to have all the records made available – not just those a government department believes we should have – and historians should not be penalised for seeking to ensure that happens.