Microsoft builds on Recall with Gaming Copilot — fails basic privacy tests
doublepulsar.com
Gaming Copilot, rolling out now to Windows 11, adds a new attack surface to Windows.

The software is labelled Beta, but is being pushed out to existing PCs anyway because… well… why not beta something in prod, it’s 2025 after all.

It appears in Game Bar, which is accessible from Windows Key + G. You may not have it yet as they are staging deployment. I’m not in Windows Insiders, but I have it installed — so it looks like the roll out is becoming wide. Note that I had uninstalled Copilot from my PC… but Gaming Copilot silently installed anyway.

It is similar to Recall, except not all the processing is done locally — it relies on the cloud. It screenshots gameplay, and then extracts elements of the screen (such as symbols and text) to work out what the player is doing. The idea is it can help you game, e.g. you can ask questions about what you’re doing in the game at a given moment.

  • utopiah@lemmy.worldEnglish
    2·
    17 hours ago

    Thanks again. Well the first sentence started so good, correct game, neat,… but then wrong weapon… so totally pointless.

    Again this can eventually be fixed. It’s “just” a data problem, and that’s exactly what models (and the entire infrastructure of data centers and researchers funded by VC money) excel at. So I think one can safely bet it will get there.

    But… today, can one genuinely imagine playing Dark Souls (or any other game) without… knowing it? Like how does search for the wrong weapon and sometimes the right one help? How is that more convenient that picking a weapon up the searching manually for its name on desktop or mobile knowing with 99% certainty it will be the right one and advice will be genuine and relevant?

    • Domi@lemmy.secnd.meEnglish
      1·
      16 hours ago

      But… today, can one genuinely imagine playing Dark Souls (or any other game) without… knowing it?

      Absolutely not, but I would love to see someone do a challenge run where they play through the entire game just with Copilot. No thinking allowed.

      • utopiah@lemmy.worldEnglish
        2·
        16 hours ago

        lol, performance art. That’d be interesting. I’d watch too.

        Plus 20K people did watch a fish play Pokemon.

        Also now that I think about it, it shouldn’t be too hard to feed a vision model a specific subset, e.g. https://darksouls.wiki.fextralife.com/Weapons of the visuals of all equipment and only then give advice. There is so much hierarchical information in there, e.g. one doesn’t get an Elden Ring weapon in Dark Souls, or does not get an end of the game weapon (except with glitches) after 1h of play time, etc so it’s possible to narrow the search space a lot.

        I imagine a lot can be done with just few curated sources. Now… again (and I apologize for repeating myself so much while possibly sounding pedantic), why? Like what’s the actual point?

      • Hudell@lemmy.dbzer0.comEnglish
        21·
        16 hours ago

        Do you mind trying it with some random skyrim door puzzle or something like that? I imagine it’s closer to what they expect people would use it for, but I don’t imagine it handling any better.

        The whole thing is specially useless if you consider that people are much less likely to try it on games that are old enough for the AI to have had data about it than on brand new games that they know nothing about.

        • Domi@lemmy.secnd.meEnglish
          2·
          16 hours ago

          Stolen from https://www.eurogamer.net/skyrim-golden-claw-door-puzzle-solution-bleak-falls-barrow :

          Gemma 3 27b:

          AI generated content

          User: Can you solve this puzzle for me?

          Gemma 3 27b: Okay, this is the “Blessing of Akatosh” puzzle from The Elder Scrolls V: Skyrim in the Temple of Kynareth.

          Here’s the solution:

          You need to activate the stones in the order representing the stages of a dragon’s life cycle. From left to right, these represent:

           Egg:  The stone with the coiled serpent (looks like a dragon egg)
 Hatchling: The stone with the dragon head emerging from a spiral.
 Dragon: The stone with the full, flying dragon.

          So, the order you need to press the stones is: Left, Middle, Right.

          Activating them in this order will open the pathway to the Blessing of Akatosh. Good luck! Let me know if you get stuck elsewhere.

          Correct game, wrong puzzle. Did detect that it is a puzzle though.

          ChatGPT:

          AI generated content

          User: Can you solve this puzzle for me?

          ChatGPT: This image shows the pillar puzzle from Bleak Falls Barrow in The Elder Scrolls V: Skyrim.

          The correct order of symbols for this puzzle is:

          Snake – Snake – Fish (Whale).

          Explanation: If you look around the room before the lever, you’ll see a broken pillar lying on the ground above the gate. It shows the solution pattern in that exact order. Set the three rotating pillars to match:

          Left pillar → Snake

          Middle pillar → Snake

          Right pillar → Fish (Whale)

          Then pull the lever — the gate will open.

          According to the Eurogamer article, that is correct.

          • Hudell@lemmy.dbzer0.comEnglish
            1·
            14 hours ago

            Thanks! And yeah, if it can’t even handle stuff like this properly I don’t think Microsoft will have any luck getting people to use it.