Getting useful feedback from a small internal pilot

"It didn't work" is a valid report from a busy colleague. It is not yet enough information to diagnose the problem. The challenge is to gather the missing context without making the person complete a technical incident report just to be heard.

For a small internal pilot, I would use a light reporting route, a named triage owner and short follow-up conversations. The collection method should respect both the participant's time and the information they handle.

Ask for the task before the rating

Begin with what the participant was trying to achieve. A fictional document-request pilot might receive the comment that search is poor. The useful follow-up is whether the person was looking for a known document, exploring a topic or checking which version was current.

Those are different tasks. A search result can be relevant to the topic and still fail a version-checking task. Asking only for a satisfaction score loses that distinction.

Keep the initial form short. Ask for the task, what happened, what was expected and whether there is a safe workaround. Capture the tool version automatically where appropriate, and allow the person to provide a contact route for follow-up. Do not require a screenshot if a short description is enough.

Give severity a business meaning

A participant's frustration matters, but severity should describe consequences. Distinguish an inability to complete essential work from a cosmetic issue or an improvement suggestion. A privacy concern needs prompt specialist review even if only one person reports it.

The triage owner can apply severity after a short conversation. Users should not need to understand an internal priority scheme. Explain which problems require stopping the task and how to report those immediately, rather than waiting for the next pilot meeting.

Frequency and impact should remain separate. Several reports may describe the same underlying fault. One report may reveal a serious boundary failure. Counting comments alone is a poor way to decide what gets attention.

Reproduce without copying the person's data

Ask for the minimum information required to investigate. A time window, non-sensitive record reference and role may be sufficient to locate diagnostic evidence. Where content is necessary, use an approved secure route rather than asking staff to paste it into a general chat.

Warn participants that screenshots can include names, browser tabs and unrelated records. Give them a safer alternative, such as describing the message or arranging a supported reproduction with prepared data. Redaction can help, but it should not become an excuse to collect more than the investigation needs.

Control access to the feedback store and agree retention. Pilot comments may include personal opinions about work, colleagues or management as well as product faults. A broad project audience rarely needs every raw comment. Share findings at the level needed to make a decision.

Tell participants what is being recorded during observation sessions. Do not quietly turn usability research into individual performance monitoring. If an issue needs a separate employment or safety process, handle it through that process rather than hiding it in the product backlog.

Preserve enough detail for a useful fix

The triage record should connect the original task to a reproducible case. Note the role, starting conditions, steps, expected result and actual result. Include the relevant version and any assistance that changed the outcome.

Separate facts from interpretation. "The page displayed an empty result" is an observation. "The index is broken" is a proposed explanation until someone checks it. Keeping that distinction prevents a confident early guess from shaping all subsequent investigation.

When several reports describe one issue, group them without erasing differences. A failure occurring only on a particular device or permission group may be hidden by an overly broad label. Preserve those conditions so the team can test the correction properly.

For improvements, record the problem before the proposed feature. A request for an export button may be a request to compare two records, brief a manager or preserve evidence. There may be a simpler solution than the first suggestion.

Close the feedback loop in plain language

Give each item a visible disposition: investigating, planned, corrected, deferred or outside the pilot's scope. Explain deferrals. Silence can encourage repeated reporting or make participants conclude that feedback was ceremonial.

Invite the original reporter to check a correction where practical, but do not make them responsible for technical verification. Retest related cases as well. A fix that helps one participant can affect someone using a different role or workflow.

Publish a short digest of themes and decisions without unnecessary personal detail. It should show what changed, what remains unresolved and what the team learned. A catalogue of every comment is less useful than a clear account of the work affected.

Keep the evidence tied to the pilot decision

Small pilots produce useful detail but limited coverage. Note which roles and conditions were absent. Do not describe silence from an unrepresented group as evidence that its needs have been met.

The findings should feed the pilot's exit decision: expand, investigate a defined gap, change direction or stop. Retain examples that explain the decision and dispose of unnecessary raw material under the agreed arrangements.

The aim is a feedback process participants can use during an ordinary working day. A brief report followed by a respectful, focused conversation can produce better evidence than a compulsory form that people abandon halfway through.