MICROSOFT 365, SHAREPOINT AND RECORD TRANSFERS
audience: Information management advisors, Compliance manager, Solutions designer
author: Jonathan Stuckey
long-read article.
Why does "transferring records" appear to be such an uphill struggle for organisations living in M365 and SharePoint Online? Surely its just a file-up/down load ? ..er No. Its not.
I work across a number of verticals in New Zealand and Australia. Unfortunately in most cases the people involved "transferring records" rarely understand the nuances when dealing with transfers between systems of record this is not the same as a copying files or data-transfer...
With SharePoint and M365 people seem think its either:
just like moving folders from file-server, or
there must be tools built-in for a task like this.. (like in an EDRM).
Unfortunately, SharePoint and the associate "Purview Records management" services treat transfer as though its just file-download/upload. There's no validated process built-in.
We know how to do this, right?
Industries been dealing with the challenge of records transfer for a long-time e.g. Central government, local government, legal services, financial services, utilities etc - basically any organisation handling money, infrastructure or civic services.
But I hear of lots of examples of where organisations obviously:
don't understand their obligations (with "records") - or even what a record actually is,
haven't looked up the requirements for transferring them between entities,
have not figured out the physical activities necessary
...to support what should be a simple process.
There are a number of global standards, so it should be easier then it seems. In fact, EDRMs and ECM platforms be signing-off to say they meet standards, like ISO15489, 13008 and 23081, and VERS v3 with VEOs..., for years.
So why is there a problem when it comes to applying this to M365 and SharePoint?
Answer:
because SharePoint was built for the individual and easy end-user experience, not for large-scale administration or compliance needs
and
accreditation under the standards is a check-box exercise on features, and vendors are not obliged to prove ease-of-use at scale.
How do you address records transfer with SharePoint?
Transferring records from one organisation to another is actually quite common place when divestiture of a subsidiary or other entity occurs, or when ownership of process or activities transfers to another group or body.
But when we deal with regulatory obligations then the transfers must ensure preservation, authenticity, accessibility, and accountability of the records (documents) being relocated. So no "drag-n-drop" of your files to an upload box, or Zip-up the folder structure to compressed file and courier it over on USB is not good enough.
What is required is the file/document and context for it to be the record that is transferred. So, I thought I'd break it down for those of us who do not live life in amongst the 'stacks' and boxes in storage facilities.
There are 4 sections:
What are the obligations we need to support?
The following outlines the requirements, for an organisation undertaking records transfer - as described in international standards. ...but once you've skimmed this jump ahead to the practicalities of achieving it using M365 and SharePoint [LINK]
WARNING: boring bit next..
The legal processes for record transfer between the systems requires that you (as the sender), must approach the following - before passing your content (records) to the recipient:
1. Legal and Procedural Preconditions
Authority and Eligibility:
Only records that have been sentenced as suitable for transfer must be included, and a current disposal authority required (i.e. documented, and approved/signed)
In government (and ancillary sectors) any transfer of public records retains their legal status unless formally discharged by the Chief Archivist.
Notification and Timing:
In NZ, for public sector orgs. Section 23(2) of Records Act 2005 requires the transferring or receiving organisation to notify the Chief Archivist within 3 months of the transfer via the Section 23 Transfer Notification Form.
In Australia the equivalent is Section 24 of the Archives Act 1983 (Cth), stipulates you can't transfer of custody or ownership of any Commonwealth record without the prior written consent and formal permission of the National Archives of Australia.
Regardless of public sector, or not, early consultation with the receiving organisation is essential, especially for transfers as a result of organisational changes, mergers, or disestablishment.
Transfer Planning:
A transfer plan must define:
Identified records to transfer.
Current locations of those records.
Roles and responsibilities of the transferring and receiving organizations.
Security and access arrangements.
Handling of records in secondary storage or contracted services.
Management of records in various physical and digital formats.
Cost allocation and continuing ownership confirmations.
2. Record Metadata and Documentation Requirements
According to both ISO standards, and regional Archives guidance, the following minimum metadata must be captured and maintained to ensure authenticity, context, and usability of a (electronic) document record transferred:
Core Record Metadata:
Unique identifier.
Record title or name.
Date of creation.
Business activity documented.
Creator (person or system).
Original software application and version used to create the record.
Current and historical file location or repository.
Versioning and Change History:
Item version history (all revisions).
Record of later actions (assessments, modifications, disposal activities).
Actors and dates for all such actions.
Contextual Metadata:
Classification or access status (open/restricted).
Extended properties relating to business use, such as commentary or narrative.
Audit trail capturing full history of record interactions.
Security and Access Metadata:
Existing access authorities, security levels, caveats, or classifications.
Any transfer-specific access restrictions or modifications.
Disposal Metadata (if applicable):
Date of disposal of previous versions or control actions.
Authorities governing disposal actions.
Persons responsible for disposal.
Retention Requirements:
Metadata must be retained for a minimum of 10 years if used in transfer or disposal processes.
3. Technical and Preservation Requirements
In order to support the life of the records that have been transferred to the new organisation or entity there must be:
Format and Platform Compatibility:
Records must be in formats readable and accessible on the receiving system.
Preservation of original digital integrity (checksums or cryptographic verification is recommended).
Security and Confidentiality:
Measures must ensure that sensitive records remain protected throughout the transfer process.
Control of permissions and monitoring during transfer is required.
Creation and Verification of an Audit Trail:
Document the transfer sequence, actors, and dates to ensure full traceability and accountability.
Basically it's still usable when it arrives, but we can traceback what happened to it and when in the previous organisation(s).
4. Submission and Approval
Behind the scenes we need all the basic prep and wrap-up protections i.e. full set of auditable reporting and inventories - before, during and after the transfer:
Prepare a transfer schedule detailing:
Record identifiers, classes, and quantity.
Dates of creation and last updates.
Metadata and context necessary for interpretation.
Submit to Archives [or recipient organisation representative] for validation:
Completeness checks.
Integrity and usability verification.
Formal legal custody transfer occurs only after validation.
5. Records Still in Active Use
Implement a transitional retention strategy.
Copies may remain on operational systems while the recipient (e.g. Archives) retains the complete, authentic archival copy.
Summary
A transfer, a proper real transfer of records requires adherence to legal, procedural, metadata, technical, and documentation standards. You have to ensure that the records remain: authentic, traceable, and accessible for future reference - by the receiving organisation, and by regulator (or Archival organisation).
All phases of the activity, from identification, preparation, transfer, to validation post move, must be systematically documented and compliant.
In reality, usually a heavy dose of practicality creeps in but you have to ensure that you are doing the right-thing, and that what is transferred is useful - or you've just wasted everybody's time, money and patience.
What's the real-skinny for SharePoint Online?
IMPORTANT: this bits the practical knowledge..
Does SharePoint do what you need? Well sort-of. Actually, to support real-lifecycle obligations you need a combination of services in M365 platform:
SharePoint system properties
SharePoint extended configuration (content types, metadata etc)
Purview Retention Labelling (lifecycle stage)
Purview Information Protection Sensitivity Labels (classification)
Purview Audit (assuming configured, and retained more than default duration)
along with some serious up-front leg work on configuration and background processes - None of which are available in the system, product services or out-of-the-box setup.
On a quick check-list... and because its not practical to do the lot in this article, lets look at M365 platform ability to support just the metadata requirements of items being transferred:
After you factor this lot together, with (the assumed) effort, to enable all the related services to function as intended - it covers ~75% of requirements ...without adding stuff.
75%
Just think about that. The platform (not SharePoint on its own), the platform can cover 75% of metadata requirement support for transfer under the standard's definition. This does not mean you can turn-it-on and we're almost there. No. This means if I spend a lot of time configuring and setting up first I can get 3/4's of the way.
...and there still wont be an integrated out-of-the-box process or workflow which will export and package-up all the related bits of information I need about each record exported from each (disparate) platform service. No, Microsoft make you build that yourself.
Brass-tacks: What is a realistic?
OPINION: based on personal experience
Well what's realistic effort vs. what's practical to meet the needs are often very different from what the standard's ask for. There are a couple of critical questions to answer at this point, before you head-off down a particular path:
Are we likely to need to do a "regulatory compliant" level of records transfer?
If you are - are you likely to do it often enough to warrant putting in the effort to establish a "proper" process for it?
Do we have the money, resources and skills in invest in the (extensive) configuration requirements of M365?
What's our best option for filling the process requirement? Our internal support model is "Bend | Buy | Build" (delete as appropriate).
If you answered 'No' to 1, and / or 2 then you can probably get aware with a basic approach (and no that is still not just "download files" from library to .zip and ftp to the recipient)
If you answered 'Yes' to 1 and 2, then you need to do something - so then it just comes down to: money, time and risk.
If you answered 'Yes' to 3 - then really what you need to do is decide if you are going to take on the Herculean task of design, build and support the solution and integration yourself or just buy something that does the job.
If you answered 'No' to 3 then really all that is available from the out-of-the-box features has to be "good enough" - In which case I recommend the minimum check-list in the next section.
If you got through to 4, and you want to do a good job (bearing in mind you cannot actually achieve 100% with M365 alone) then you are into vendor add-ons like: AvePoint, RecordPoint, Preservica etc - or very expensive custom development.
And if I don't have buckets of spare cash for a Hygiene project?
If you are, like most organisations, in the middle of a funding crisis and need to address Records Transfer with a degree of compliance, its best to work through the practicalities of supporting the useful majority of requirements. In which case there is:
List (inventory) all files to be exported, itemise with location and associated metadata (system properties, business class. schema etc)
Classification labelling - with Purview you can have this captured on export of the item - using the Information Protection label
Retention logging - also with Purview you can access this and the sentencing reference data - Using Retention Labels and File-plan
Audit log for items can be exported - but here it gets trickier - assuming:
you have audit logging captured and stored for periods of 10yrs or greater, and
you have identified the required events necessary to support your requirement.
After this things go down hill a bit:
Access permissions - who, and which groups have what sort of access to the item - current is retained on the item, but its not easy to retrieve without dev or tools
Activity logging - M365 only captures recent activity, and expires it - so be quick
Disposal record for historic versions of the file - export from Purview Disposal is feasible, but you need to have document internal Id and associated data to search Disposal log and then export it (per item, one-by-one)
Trailing versions - all the retained versions of an item - technically possible, terribly painful with out a tool and way of referencing number sequence when exported.
Other transfer information e.g. transfer specific restrictions etc, - would need to be captured externally from the process as they are not natively part of export; or you encode capture in custom Workflow logging - Messy.
What can you achieve without breaking the bank?
Well it pays to have put in some effort to ensure you have the related information (or at least appropriate means to have captured it):
Unique Id - have your sites setup with feature enabled i.e. Make sure you have Document ID feature enabled on across you SP sites - or at the very least across the business critical and authoritative sites.
Extended metadata - Insist all your sites have basic common metadata, besides system properties e.g. Business function, activity and document type (no - not format)
Source metadata - If you migrated your documents into SharePoint from another platform first, make sure you have a record of the the file original location or if you are undertaking an import capture: original created, created by, modified, modified by, original location path, original file name
Version history - have this configured to an appropriate level. It's captured, but its not easy to export and keep track of the sequence without PowerShell or a tool.
Classification labels - requires Purview licensing to support this, but if configured and pushed out it's attached as a standard library metadata attribute to your documents, comms etc
Retention labelling & File-plan - captures the related regulation, legislation or business rule for applying retention, disposal, business functional properties etc - if configured properly
Audit logging - have someone who understands what needs retaining in the audit - and then setup a policy for require period i.e. 10yrs min.
But some things are frankly just not worth struggling over e.g.
for 99% of files we can infer the original application - if not the exact version - from the file-format extension.
or Activity history - in old EDRM speak narrative, life-history - ...if you're lucky you could query (max) 90-days of activity on file stored in SharePoint/OneDrive - but exporting is custom code only.
Net result
From all of this, the practical mathematics of necessity has the M365 Platform, with additional services and extensive setup and configuration, delivering 75% of support...
Then we can subtract:
approx. 10% of the metadata, and supporting configuration tasks are too-hard to do properly or export usefully
lose another 10 - 15% of critical items with broad adoption of key features like Document ID, extended metadata for BCS are not done consistently (if at all)
drop a critical 10% when don't have use of Purview for Retention - which is often poorly implemented, if at all, and usually not with Labels (required)
with a little additional of negative numbers, we end up with ~45% of standard required feature coverage in M365 is usually readily available and usable.
All up most organisations can do a passable 45% of the outcome required to support a transfer.
This is still better than file-download as .zip.
Close
If you have questions about this, or need a hand setting yourself up for a more achievable outcome give me a call. I do coffee and will listen.
References
Acknowledgement
Generative AI has been used in this article to: clean-up and review of article copy (spelling, grammar, tone), the creation of the article images, and QA review prior to publishing. Tools used: Wix Photo Studio AI Image Creator tool, Microsoft 365 Copilot for reference research. The article topic and written copy, ideas and visual imagery, key discussion points, frustrations and similes were created by the author based on personal experience and interactions with customers and business professionals. Any errors or issues with the content in this article are entirely the author's responsibility.
About the author: Jonathan Stuckey
Comments