File sharing software test: thirty files, seven timed tasks

A ranked list of file sharing products would have to assume it knows what you keep, who you share it with, and what you are obliged to prove about both. Change any one of those and the order changes. A timed test on your own files does not have that problem.

The test needs one afternoon of preparation and the same seven tasks in every candidate. The results do the sorting, and the field is the familiar one: Dropbox Business, Box, Google Drive, Microsoft OneDrive and SharePoint, Egnyte, Citrix ShareFile, Nextcloud, Tresorit, and M-Files.

What to take away

Build the fixture set first

Assemble a folder of about thirty files that stands in for your worst cases before you look at a single product. It is the single highest value hour in the whole evaluation.

Fill it from the estate you already carry: file counts, path depth, oversized items, and odd formats some engineer standardized on in 2014. Copy the set into every candidate, with the same files, the same order and the same tasks.

A tidy folder of small new documents passes every test and teaches you nothing. Keep the files that already give you trouble in the set, because everything after this is comparison rather than impression.

Seven tasks, each timed

Run the same seven tasks in each product, and time them with a watch rather than a feeling. Write the time or the observation next to each task as you go, because a task with no number beside it did not happen.

Task five is the one that most often separates tiers rather than products, so run it on the tier you plan to buy. Task seven is the one people skip, and it decides whether leaving is possible later.

On task two, the outside person's view matters as much as yours. Ask them what they could see and do, and compare it with what you intended. On task six, the delay is the finding: a change that takes long to apply everywhere is a window in which the old access still works.

Week one against week three

Almost every product wins week one. The interface is new, the search is fast because the account is nearly empty, and nothing has gone wrong yet.

Week three reveals the differences, and by then the account holds enough that search quality becomes a real question. Someone shared something they should not have, and a sync client made a conflicted copy. At least one person asked why they cannot see a folder everyone talks about.

Extend the trial instead of shortening it, and keep a plain log of every time someone had to ask for help, because that log is your comparison.

What the demo will not tell you

Ask these directly, and treat a vague answer as an answer.

Two of those questions deserve more than a vague answer, and neither has a single right answer. The first is who holds the encryption keys: with end-to-end encryption the provider cannot read your files, and it also cannot recover them for you. Provider-held keys make password reset and e-discovery simpler, and they put your content within reach of a subpoena served on the vendor.

The second is which attestation or certification covers the service you would actually buy, not the corporate parent. Ask to see the SOC 2 report produced under AICPA standards, an ISO/IEC 27001 certificate, HIPAA safeguards where health data is in scope, and FedRAMP authorization if US federal work is in scope, and ask which region stores your files.

Check that two offers are the same kind of thing before comparing them. The NIST definition of cloud computing is composed of five essential characteristics, three service models, and four deployment models.

The right to receive your own data in a usable form is settled law in some jurisdictions, not a favor. Article 20 of the GDPR gives people in the EU and UK a right to data portability, and the ICO's right to data portability guidance names CSV, XML and JSON as structured, commonly used and machine-readable formats, a decent benchmark for an export even where the law does not reach you.

From timings to a decision

Split your criteria into two lists and keep them apart. Requirements are pass or fail: the product either does the thing or it is out, and there should be very few of them. Preferences are traded against each other, and they are where price belongs.

Score independently, then compare notes. A group scoring session converges on whoever speaks first, and the disagreement you most need to see is usually confined to one row.

One honest warning before you start. If the real complaint is that nobody knows where the current version of anything is, the product may not be the problem. That is a decision about who owns a document and when it stops being a draft, and no amount of storage fixes it.

Common questions

How many products should be in the trial?

Two, or three at most, chosen after disqualifying requirements have cut the field. A trial of five is a research project that nobody finishes properly, and the quality of the testing falls faster than the number of options rises.

What if the two finalists are genuinely equal?

Then say so and break the tie on the easier exit. Time spent separating two adequate options is time not spent on the decisions where nothing adequate is on the table.

Can we run two products at once?

Deliberately, yes, if the boundary is a rule anyone can state in one sentence. Accidentally, which is the common case, it produces duplicate versions and a search that never finds anything on the first try.

Why time the tasks instead of rating them?

A rating records an impression, and week one flatters every product. A time written down during the trial is the only record you will still trust at renewal.