Measuring adoption without mistaking logins for value

A rise in logins can mean a new service is useful. It can also mean staff have been told to open it, cannot find what they need or must keep returning to finish a task. Usage is evidence of activity, not a complete account of value.

I would still collect usage data where appropriate. It helps identify reach and abandonment. But the review should connect that activity to the work the service was introduced to improve.

Define the unit of useful work

For a fictional internal purchasing portal, a useful unit might be an accurate request ready for purchasing review. A login, draft and submitted form are intermediate events. The process may still fail if purchasing staff have to phone the requester to reconstruct missing information.

Define completion across that boundary. Include the quality conditions that make the output usable and distinguish a completed request from one returned for correction. If the process spans several tools, agree how the team will follow a request without collecting unnecessary personal information.

Then decide which outcomes matter. The portal may aim to reduce rekeying, make requests easier to track or improve approval consistency. Those are different propositions. A service can improve one and disappoint on another.

I would avoid collapsing them into a single adoption score. Show task completion, quality and effort separately, with a short explanation of the tensions. Faster submission alongside more downstream corrections is a result to investigate, not an automatic success.

Frequency also changes interpretation. A tool used once a quarter may be valuable despite low weekly activity. Compare use with the number of relevant opportunities to use it, rather than treating daily engagement as a universal goal.

Establish what the baseline can support

Before launch, observe a sample of the existing process. Record hands-on work, elapsed waiting, corrections and help from other people. Use consistent definitions so a later comparison describes the same task.

A baseline collected during an unusually quiet period may not represent month-end demand. A sample drawn only from experienced staff may understate the effort new starters need. Write those limitations beside the results rather than putting them in a footnote nobody reads.

Sometimes the old process has no reliable records. Retrospective estimates can still help frame a hypothesis, but label them as estimates. Do not turn staff recollections into a precise improvement claim. It may be better to establish a dependable post-launch baseline and then measure subsequent changes.

Comparison groups can also differ in ways the service did not cause. A pilot team with dedicated assistance is not equivalent to a later group working without it. Demand, policy changes and staffing may affect results. A responsible report identifies plausible alternative explanations instead of attributing every favourable movement to the software.

The aim is decision-grade evidence, not a claim of experimental certainty where none exists. Small operational samples can inform a change while remaining too limited to justify a broad productivity promise.

Look for effort that moved elsewhere

Ask who does the work before and after the change. A self-service form might reduce service desk calls while increasing the time requesters spend interpreting fields. Automated routing might save administration but create an exception queue that requires specialist judgement.

Include that displaced effort in the review. Sample the difficult cases, not only successful automated runs. Record manual intervention and the time spent maintaining rules, explaining results or repairing records.

Useful measures for the fictional portal could include:

• Requests accepted without further clarification.
• Requests abandoned or returned, with a reason where known.
• Hands-on effort across the requester and purchasing team.
• Waiting time at approval and exception stages.
• Recurring support issues that prevent independent completion.

These measures require care. Counting individual activity can feel like performance monitoring and may invite gaming. Explain why the data is being collected, restrict access and prefer process-level reporting where individual detail is unnecessary. Staff should be able to report a workaround without assuming it will be used against them.

Pair the numbers with observation. A reduction in support contacts may mean the service improved, or that people stopped asking and built a spreadsheet. A short conversation with the people doing the work can distinguish those possibilities.

Use the review to change a decision

Set the measures before asking for more funding. Define what result would justify expanding the service, changing it or investigating further. That does not require arbitrary universal targets. It requires an agreed interpretation of the business need and acceptable operating effort.

A service review might conclude that the portal is useful for standard purchases but creates too much work for complex requests. Keeping a supported alternative for that category could be better than forcing every request through one route. Record the boundary and review whether it remains justified.

This evidence also improves the business case for workflow automation. Capacity released on paper becomes meaningful when managers can explain what work has changed and what staff can now do with the time.

I would report adoption as a developing picture: who can use the service, whether the task completes and what it costs to operate. Logins remain part of that picture. They simply stop carrying a conclusion they cannot establish by themselves.