Making 3D Bugs Visible: What I Learned Building Somewhere Together with AI

I’m an experienced technical innovation consultant, and Somewhere Together is my first experience developing a game. Building a multiplayer 3D world with AI has given me a practical education in how much can hide behind a simple description like “the vehicle jumps.”

A jump on screen could come from the simulation, a network correction, prediction, the camera, or a frame that took too long to render. Asking AI to fix the jump still leaves all those explanations open. I needed ways to show what was happening and test which explanation fit.

Some of the most useful work went into the tools around the game: visible build labels, X-ray views, short diagnostic histories, and controlled reproductions. These eight techniques helped us investigate specific problems. Each also had limits that mattered when deciding what to change next.

1. Put the loaded build on screen

During phone testing, we needed to establish whether I had actually refreshed to the latest version. A screenshot of the game could show the problem clearly while leaving the version uncertain.

We added a small development label with the preview name and Git revision. When relevant source changes were still uncommitted, a fingerprint distinguished those changes too. The label described the page that had loaded. It did not update itself to a newer revision while the old page remained open.

That detail made a screenshot more useful. We could connect what I was seeing to a particular version before investigating the behavior. It also gave AI a concrete reference when comparing feedback across revisions.

The badge identified application source; it did not prove that every asset or server component matched. It was deliberately a development tool, excluded from production builds. Its job was to remove one avoidable uncertainty from the conversation.

2. Show the different versions of a moving vehicle

Racing movement made the word “position” surprisingly ambiguous. The server’s position, the client’s predicted position, and the vehicle drawn on screen could differ for legitimate reasons.

We added an X-ray driving view with gold server collision rings, a cyan predicted chassis, and a white rendered chassis. The guides remained visible through scenery, with a legend explaining what each represented.

Now I could describe which representation moved unexpectedly. We could examine whether the camera followed the displayed vehicle, whether prediction diverged, or whether the authoritative state arrived late. AI could inspect those same relationships instead of inferring hidden state from the final image alone.

A gap between the gold and cyan guides was not automatically a bug. Network delay normally creates separation. The diagnostic work also did not prove that the intermittent phone jitter was resolved. It gave us a way to capture and discuss that symptom more precisely.

3. Keep a short history of what just happened

A brief movement problem can be over before I open a menu or explain it. A screenshot captures the aftermath but loses the sequence that led there.

The racing diagnostics therefore included a driving report covering up to the previous 30 seconds, bounded to 300 samples. It recorded server, predicted, and rendered poses alongside snapshot age, frame peaks, contact, and recovery state. Recording happened while X-ray diagnostics were enabled.

That history connected a visible event to timing and state. An old snapshot and a long frame suggested different investigations, even if both appeared as a stutter. The report also gave AI something structured to compare with the image and reproduction steps.

We kept the instrumentation bounded and released the overlay resources when leaving or disabling it. The report did not capture the whole system, and its local measurements could not establish what happened elsewhere on the network. It was a focused record for a particular investigation.

4. Reproduce the conditions that separate explanations

In shared rides, I reported returning to a position several seconds earlier. An ordinary latency test had passed, so repeating that test would have left the report unexplained.

We tested a different condition: pause upstream traffic, then deliver the real queued movement messages. That produced a large rollback in a controlled local run. The server was limiting movement using message arrival time. Messages representing seconds of legitimate travel arrived close together and received too little movement allowance.

The repair used bounded movement-sample timing, with validation for invalid timestamps. The investigation also separated passenger extrapolation and name-label projection from driver correction. Several visible symptoms had different causes.

This historical reproduction gave us a failing case against which to check a repair. A later injected-pause replay still showed backward adjustment; it is not a passing replay of that historical repair. It did not establish that my original connection had experienced that exact pause, identify a provider fault, or prove that all ride synchronization problems were solved. The conditions and the conclusion needed to stay attached.

5. Profile the work and preserve the comparison

Performance problems required the same discipline. During an instrumented walk in Akihabara, the profile pointed to procedural clothing construction and garbage collection. Camera work alone did not explain the stalls.

Temporarily suppressing the relevant resident construction helped test that explanation. The delivered solution prepared residents during travel while keeping them in the world. Removing content was useful as an experiment; preserving it was part of the product requirement.

Other experiments challenged plausible optimization ideas. One reduced draw calls without producing a useful measured FPS gain. On the measured Android device, HUD backdrop blur was a consequential cost. Counting rendering calls could not substitute for measuring the experience.

We also had to control the comparison: device, camera, resolution, population, warmup, and browser state. Extra tabs and attached debugging could confound results. Local desktop profiles, phone emulation, and physical-device samples answered different questions. None justified a claim about every device or long-session performance.

6. Draw the gameplay parameter I am trying to tune

Choosing a nearby action raised a spatial question: how close should I be, and which object should count when several are nearby?

We drew the avatar-facing interaction cone in the world and exposed reach and width controls beside the live experience. I could move around furniture, change a value, and see how the eligible action changed. In the October 13 tuning pass, I selected 1.00 metre of reach and a 70-degree full cone width. Those are historical values; later defaults changed to 1.1 metres and 100 degrees.

The visible guide made direction and nearest-object selection discussable. It also exposed a distinction that mattered: turning the camera alone did not change the avatar-facing target. The sliders were temporary preview tools rather than saved player preferences.

This became an example of a broader practice: review the schematic, units, and important calculations before building a mechanic. Values give feedback somewhere specific to go. Our selected dimensions were product choices, not universal interaction standards, and a mathematically consistent mechanic still needed playable review.

7. Give AI evidence it can inspect together

A useful screenshot could still omit the awkward side of a building, the route back, or a failure visible only in motion. We needed broader coverage without losing the specific question under investigation.

The project’s review process brought together the loaded revision, reproduction steps, intended behavior, actual captures, measurements, and remaining uncertainty. For world construction, that included arrival and return views, boundaries, elevated angles, and relevant lighting or weather. Passing a walking test did not establish that the visible coastline was convincing.

The same principle applied to AI review. An overlay needed its legend; a performance result needed its conditions. I wanted the next experiment to follow from observations that we could both inspect.

We formalized independent visual critique and correction as part of that process. Those workflow additions did not, by themselves, prove better visuals or fewer corrective prompts from me. That outcome would need its own before-and-after evidence.

8. Check what reached the player

A merged fix and a verified release were separate milestones. The deployed game also had to support people whose browsers still held an older client.

Our release checks preserved earlier content-hashed assets, exercised current and previous clients, and checked admission, reconnect, and travel. The verified staging artifacts and server image were then promoted to production, rather than treating a source push as proof of delivery.

This added evidence about a different boundary. Local tests could support a movement repair while hosted checks established whether the assembled application actually worked through its delivery path. A notes-only publication could be checked separately without implying another gameplay validation.

Even a verified release did not establish sustained phone performance, every character-and-outfit fit, or independent player acceptance. Keeping those distinctions explicit helped me describe what we had checked and what still needed observation.

These are the methods I’m carrying into the next game-development problem. I can give AI a clearer starting point when I can identify the build, expose the relevant state, and preserve a reproduction. When the evidence leaves a question open, that question becomes part of the next test.

You can explore the game at Somewhere Together.