Dev Spy

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91

# Dev Spy

A HUD in the corner of your screen shows what a Rebind script would see right now: the pointer position, the monitor under it, the active window, the color under the pointer, and the keys you hold. Use it while you write a script to find coordinates, colors, and window names without guessing.

Dev Spy can also record. An observation saves every key, click, scroll, and window change, with a screenshot at each action and when things settle, as a file an AI model or a script author can read. A macro recording saves only keys, mouse, and controller input, and turns it into a macro Rebind can play back. From the same menu you can try `rb.ai.decide`, `rb.ai.find`, and `rb.ai.read` on the screen in front of you before you put them in a script.

Needs Rebind 3.13.0 or newer on Windows or macOS. It requests the `ai` permission and clipboard access. On macOS, Rebind needs **Screen Recording** permission for screenshots and the color under the pointer. The AI console needs Rebind Pro (or its free trial); everything else is free.

## Use

Press **Ctrl + Alt + D** (**Control + Option + D** on macOS) to open the menu. Press a number, or move with **Up** and **Down** and press **Enter**. **Escape** closes it. While the menu is open, those keys go to the menu only.

From the menu you can start an observation or a macro recording, stop it, mark moments while it runs, hide or show the HUD, open the AI console, copy the pointer and color as a line of Luau (such as `rb.hid.move_to(1287, 644) -- #1e1e2e`), and copy the full path of Dev Spy's data folder. Hiding the HUD, observing, recording a macro, and copying the pointer and color can also get their own keys under **Keys** in the settings; none has one by default, so Dev Spy takes no other key from your apps.

While the HUD is hidden and a recording runs, a strip at the top of the screen shows its time, its screenshots or events, and that it is logging keys. It also shows a stopped recording that is not saved yet.

Dev Spy only watches the rest of your input. Its own keys, and the keys its menu takes, are left out of recordings. The Ctrl and Alt of the menu chord stay in an observation's log, marked as Dev Spy's, and are left out of saved macros.

## What the HUD shows

- **pointer**: screen coordinates, then the same point relative to the active window.
- **monitor**: which monitor the pointer is on, its size, and where it sits on the desktop.
- **window** and **process**: the active window's title, its app, its size, and its position.
- **pixel**: the color under the pointer as hex and as red, green, and blue. Over Dev Spy's own HUD or AI panel it reads "under Dev Spy" instead.
- **keys**, **mods**, and **last**: the keys held now, the modifiers held, and the last input with how long ago it was.
- **device** and **pad**: in hardware mode, the device the last input came from and any controller stick or trigger that is moved. Software mode cannot tell devices apart.
- **timer**: how often Dev Spy's own timer actually runs, to the nearest 10 ms. It reads 100 ms when Rebind keeps up.

The top row shows how long Dev Spy has been running. During a recording, the HUD shows the recording's time, its events and marks, its screenshots, crops, and their size, and the size of its log.

## Recordings

An observation and a macro recording log every key you press in every app, including passwords and messages, for as long as they run. The HUD and the strip say so while one is running. Stop recording before you type anything private, and check a recording before you share it.

A recording goes into the `sessions` folder in Dev Spy's [data folder](https://docs.rebind.gg/reference/sdk-reference#file), outside your scripts folder, written as it happens, so Dev Spy keeps very little in memory. Each one is a folder with `events.jsonl`, one event per line. An observation adds a `frames` folder of PNG screenshots and a `crops` folder with a sharp image around each click. Each key event names the latest screenshot taken before it.

An observation takes a screenshot:

- when it starts;
- at once when you click, press Enter, Escape, or Tab, or use a shortcut with Ctrl, Cmd, or Alt;
- at the first key press, scroll, mouse movement, or stick motion after a pause, showing what you reacted to;
- when things settle: 400 ms after you stop moving the mouse, scrolling, typing, or moving a controller stick, and after each action above, so the result is on screen;
- 250 ms after the active window changes, including its title, position, or size, and at most once a second while it keeps changing;
- after a quiet stretch, 1 second by default, so a change nobody caused is placed within about a second.

A screenshot identical to the one before it is not saved again; the log points to the earlier file, so a still screen costs almost nothing. A screenshot taken for a click or key names that input. Each click also saves a 160 by 160 crop around the pointer at the display's full resolution, which a script can search for with `rb.screen.find_image`, and the log notes the color at each click before and after, for up to 8 clicks in one burst. A click made while the previous crop is still being taken is logged without a crop.

**What this recording is for** in the settings is saved at the start of each recording. While a recording runs, the menu opens with **Mark this moment**, **Mark success**, **Mark failure**, and **Mark with clipboard text**, so an AI can tell the steps and the outcomes apart. A mark is timed when you opened the menu.

When you stop, open the menu to save it:

- **Save to observations/** moves it to `observations/<name>` in the data folder.
- **Save to macros/** moves it to `macros/<name>` in the data folder and also writes `macros/<name>.luau`. Key presses, mouse buttons, pointer movement, and scrolling play back, and pointer movement is replayed as the steps you made from where the recording started. A key still down when the recording stopped is let go at the end. Controller buttons play back on a Rebind Link whose USB profile has a joypad. Stick and trigger movement stays in the log only.
- **Name from clipboard** takes the name from the text you copied. Otherwise the name comes from **Name for the next save** in the settings, or the time the recording started. A name already in use gets a number added, and the menu shows the name it will use.
- **Discard** deletes the recording after you confirm.

The HUD and the save notice show paths starting with `@data/`, which is how Dev Spy's code names its data folder. To run a macro on its own or load it from another script, copy its `.luau` into your scripts folder.

A recording you have not saved is kept. If the script restarts, including after you change one of its keys, Dev Spy says so and the menu offers it again.

## AI console

Open **AI console** from the menu. Each call runs once, and the result shows in the AI panel at the bottom left with how long it took.

- **Decide** asks the question from the settings and picks one of the choices, with the model's confidence. It looks at the display under the pointer, or at the window you name.
- **Find** looks for what you describe and marks the spot it found for 5 seconds.
- **Read at pointer** reads the text in a 400 by 160 box around the pointer. **Read in window** reads in the window you name, or the active window.
- **Question from clipboard** uses the text you copied as the question or description instead of the settings.

During a recording, each call and its answer are saved in the log. Each call counts against your monthly AI allowance.

## Settings

- **Keys**: the menu key, and optional keys for hiding the HUD, observing, recording a macro, and copying the pointer and color. A change restarts the script, which stops a recording that is running.
- **HUD**: the theme, and how often the HUD refreshes.
- **Sessions**: how long a quiet stretch lasts before an observation takes a screenshot anyway (0 turns this off), the screenshot size, when a recording stops on its own (30 minutes by default), what a macro recording includes, what the recording is for, and the name for the next save.
- **AI**: the question and choices for Decide, what Find looks for, what Read reads, and the window to look at.

## Limits

- A recording stops at 512 MB, crops included, or at the time limit, whichever comes first. One screenshot is taken at a time. Settle, window, and idle screenshots stay at least a quarter second apart, while a click, an action key, or the start of a burst is taken as soon as the one before it finishes.
- A window whose title keeps changing, such as a terminal with a spinner, adds about one screenshot a second while it changes.
- A macro is written from a log up to 4 MB. A longer recording still saves its log, without the `.luau`.
- A macro moves the pointer by the steps you made, so clicks land where you made them only when it starts where the recording started, on the same monitor layout. In software mode on macOS the steps are exact; through a Rebind Link or on Windows, pointer acceleration can shift them.
- A game in exclusive full-screen mode on Windows hides the HUD and the menu.
- Recordings stay in Dev Spy's data folder when Dev Spy updates from the marketplace. Deleting Dev Spy in Rebind deletes them, and cloud backups leave them out, so copy the ones you want to keep.
- Live use is checked on macOS. Windows remains unverified.

## Stop

Stop the package from the editor toolbar, or press **Left Ctrl + Left Alt + K** (**Left Control + Left Option + K** on macOS) to stop all scripts.