diff --git a/README.md b/README.md index da1448f..e8940a0 100644 --- a/README.md +++ b/README.md @@ -13,13 +13,21 @@ directly. - Deciding **fast** whether a request is even achievable client-side, instead of burning hours on a paywall that is enforced by a server. +- Deciding **what form the deliverable must take** before any work starts — an + unrooted, self-contained artifact is a different problem from "make it work on this + machine", and confusing the two is the most expensive drift in this domain. - Choosing the **safest patch layer** for a given change, and avoiding the layers that break the app. - Separating **your own mistakes from the app's or the server's problems** — a feature-scoped failure (login, registration, payment) is often a TLS/certificate issue on - one code path, not a consequence of the patch you just built. + one code path, not a consequence of the patch you just built. Device state, a dead device + server and clock drift masquerade the same way. +- Establishing **which architecture and which library are actually executing**, rather + than trusting what the manifest ships or what the device claims. - Working through **packed/hardened targets**: identifying the packer, unpacking, and turning a memory dump back into a patched, installable APK. +- Keeping a **long task honest**: a live record, graded conclusions, calibrated timeouts, and + bounded waits, so progress is not lost and the same mistake is not made twice. - Avoiding the specific mistakes that produce an APK that builds perfectly and dies at runtime. @@ -31,10 +39,13 @@ references/ loaded on demand, one topic each recon.md identify packer, SDKs, code location, tamper checks; unpacking packers.md hardened targets: rejection signals, measuring the validation boundary with single-variable tests, choosing a native host - framework-runtimes.md Flutter / React Native / Unity: which layer owns the UI + framework-runtimes.md Flutter / React Native / Unity: which layer owns the UI, and how to + find logic when there are no symbols (string encoding traps) native-and-so.md .so hosts, DT_NEEDED vs JNI_OnLoad, relocation limits, - relocation-free bootstrapping, replacing Java methods natively - long-task-discipline.md live record, conclusion grading, drift control, handover + relocation-free bootstrapping, replacing Java methods natively, + and which ABI/library is *actually loaded and executing* + long-task-discipline.md live record, conclusion grading, drift control, timeout and + wait calibration, deliverable-form drift, handover ad-removal.md ad taxonomy, wrapper mapping, callback trap, global gates, verification membership-and-limits.md server vs client authority; what is and is not patchable server-api.md probe an app's API; prove who owns the gate @@ -42,9 +53,11 @@ references/ loaded on demand, one topic each third-party-builds.md auditing a "cracked"/"modded" APK before trusting it dex-patching.md patch-layer table + dexlib2 technique in depth repack-and-sign.md repack rules, unpack-and-repack, signing, post-install hazards - runtime-data.md DataStore / SharedPreferences / SQLite / protobuf + runtime-data.md DataStore / SharedPreferences / SQLite / protobuf; when the app + rewrites your edit, and decoding a value that looks encrypted dynamic-frida.md Frida setup, version pinning, the four-layer probe, hook strategy - environment.md device/emulator setup, ADB, UI automation, offline devices, log signals + environment.md device/emulator setup, root, ADB, offline devices, log signals, + emulator console control and recovery, preflight, look-at-the-screen verification.md the claim ladder; what "done" means pitfalls.md the failure catalogue -- read before building scripts/ parameterized, path-agnostic @@ -65,6 +78,15 @@ scripts/ parameterized, path-agnostic frida_probe.js four-layer runtime probe (app net layer + OkHttp + java.net + exceptions) run_probe.py inject the probe, stream it to a log file, stay resident tls_check.py strict certificate check for one or more hosts + preflight.py environment check before every experiment block (device, root, + ABI/translation, clock skew, leftover proxy/forwards, dead server) + lib_map.py what is *actually mapped* into a live process: per-library path, + base, architecture, and whether it came from the APK or was + materialized at runtime + blob_decode.py search, don't guess, the framing of a stored value + (base64/hex x rotation x deflate); re-encode the edited payload + snap.py bounded burst screenshots + control-tree capture with a stall + detector, and a verdict on whether the tree is usable at all ``` ## Install diff --git a/SKILL.md b/SKILL.md index f4c1bf6..87b02bb 100644 --- a/SKILL.md +++ b/SKILL.md @@ -7,10 +7,11 @@ description: Reverse engineer, debloat, de-ad, patch, or re-sign Android APKs, a Goal of this skill: get to a **verified, installable, still-working artifact** fast, and avoid the whole class of mistakes that silently destroy an APK. -Two rules override everything else in this skill: +Three rules override everything else in this skill: -1. **Never ship an unverified APK.** "It assembles" is not "it works". Install it, launch it, exercise the feature you changed, on a real device or a close emulator. -2. **Know which layer owns the behavior before you patch.** Ads, paywalls, and feature gates live in different places. Patching the wrong layer either does nothing or breaks the app. Classify first (see step 2 below), patch second. +1. **Deliver it in the environment the request actually requires.** "It works" is not the goal; "it works under the stated constraint" is. A result obtained with root, or with live instrumentation, or with a host proxy, frequently does **not** satisfy a request for an installable artifact that works on a normal device — and it is easy to present that result as a finished one. Write the constraint down as a testable sentence in the first minutes and re-read it at every checkpoint. If you cannot meet it, say so and label any privileged workaround as a fallback, never as the deliverable. Full treatment: `references/long-task-discipline.md` §the most expensive drift. +2. **Never ship an unverified APK.** "It assembles" is not "it works". Install it, launch it, exercise the feature you changed, on a real device or a close emulator. +3. **Know which layer owns the behavior before you patch.** Ads, paywalls, and feature gates live in different places. Patching the wrong layer either does nothing or breaks the app. Classify first (see step 2 below), patch second. ## Start here: classify the target in seven questions @@ -27,13 +28,23 @@ Answer these before touching a tool. Every one of them changes the whole plan. 3. **Is the app's own code in plain dex, or moved to native/Flutter/Unity?** Plain dex → you can patch. Flutter (`libflutter.so` + `libapp.so`) / Unity (`libil2cpp.so`) / pure native → different toolchain entirely. See `references/recon.md` §code-location and `references/framework-runtimes.md`. **Runtime check (cheap -- do it before committing to a layer):** hook the obvious Java classes for the UI you care about, then reproduce that UI. If those hooks fire, the behavior is Java-owned. If they fire **zero times** while the UI is plainly on screen, the behavior is drawn by the runtime or by native code, and a dex-only plan will stall. Do not keep hunting in dex after a zero-hit probe -- that is the most expensive wrong turn in this skill's history. -4. **Does the app verify its own signature, or does the server?** +4. **What must the deliverable be able to do?** Write the answer as a testable sentence before + planning anything, then re-read it at every checkpoint. This is the drift guard, and the drift it + guards against is the most expensive one in this skill: a runtime-only result (a data edit, a live + hook, a blocked hostname, a host proxy) can look like success while failing the actual requirement. + The axes that decide it: **privilege** (unrooted?), **modification form** (a rebuilt, installable + artifact, or is live instrumentation acceptable?), **ABI/device class**, **network** (must it work + online?), **persistence** (survives restart / upgrade / fresh install?), **distribution** (must the + shipped file be self-contained?). → `references/long-task-discipline.md` §the most expensive drift. +5. **Does the app verify its own signature, or does the server?** App-side → you must bypass it. Server-side → re-signing silently breaks the app later. See `references/repack-and-sign.md` and `references/server-api.md`. -5. **What is your device situation?** → `references/environment.md` - Rooted real device (best), emulator with root, or no device (static only). Also: this determines whether Frida is usable. -6. **Is one *specific feature* failing at runtime — login, registration, payment, an API-backed screen — while the rest of the app works?** +6. **What is your device situation?** → `references/environment.md` + Rooted real device (best), emulator with root, or no device (static only). Also: this determines whether Frida is usable. **Run `scripts/preflight.py` before your first experiment**, and again whenever a failure surprises you — device state, a dead device server, a leftover proxy, and clock drift all masquerade as a broken patch (`pitfalls.md` P9). +7. **Which architecture is actually executing?** → `references/native-and-so.md` §Cross-architecture + `getprop` reports what the device claims and `primaryCpuAbi` reports what the package manager chose — neither is what is running. Only the live mapping is ground truth (`scripts/lib_map.py`). If the library you meant to patch is not mapped, a translator is in play, or the ABI differs from your assumption, that changes the plan more than any patch will. +8. **Is one *specific feature* failing at runtime — login, registration, payment, an API-backed screen — while the rest of the app works?** → `references/tls-and-cert.md`. A feature-scoped network failure is very often a **TLS/certificate problem on one code path**, not a consequence of your patch. The app can even carry two independent trust chains, so "other requests work" proves nothing. Rule this out in minutes before hunting for a signature check. -7. **Was the input a build you did not produce** (a "cracked"/"modded" APK circulating online)? +9. **Was the input a build you did not produce** (a "cracked"/"modded" APK circulating online)? → `references/third-party-builds.md`. Audit it before adopting it: such builds are frequently re-protected (sometimes with *more* layers than the original) and may carry injected components or endpoints. Never use one as a patching workbench. **Long-task rule:** if this is likely to run long, open `references/long-task-discipline.md` now @@ -43,13 +54,13 @@ Answer these before touching a tool. Every one of them changes the whole plan. ## The workflow, end to end -1. **Recon** — `references/recon.md`. Manifest, package name, version, ABI, dex count, packer, embedded SDKs, where the app's own code lives. Ten minutes here saves hours. **If it is packed, unpack before anything else** (`references/recon.md` §unpacking): you cannot patch code you cannot read, the encrypted payload lengths tell you which dumped dex is the original, and a memory dump must be de-duplicated by hash and structurally validated before any of it is trusted. +1. **Preflight, then Recon** — `scripts/preflight.py` before anything else if a device is involved (it takes seconds and prevents a whole class of false conclusions), then `references/recon.md`. Manifest, package name, version, ABI, dex count, packer, embedded SDKs, where the app's own code lives. Ten minutes here saves hours. **If it is packed, unpack before anything else** (`references/recon.md` §unpacking): you cannot patch code you cannot read, the encrypted payload lengths tell you which dumped dex is the original, and a memory dump must be de-duplicated by hash and structurally validated before any of it is trusted. 2. **Extract strings and endpoints** — build a picture of the app's API surface and SDK inventory from the dex string tables. No decompiler needed for this, and it is fast. Scripts: `scripts/dex_strings.py`. 3. **Trace to the owning class** — find the class that wraps the behavior (the app almost always wraps third-party SDKs in one helper). Reverse-lookup instructions: `references/dex-patching.md` §finding-the-call-site. 4. **Decide the patch layer** — client SDK call / client rendering / client data consumption / server contract. See the table in `references/ad-removal.md`. 5. **Patch surgically** — `references/dex-patching.md`. Prefer **dexlib2 method-level rewriting** over whole-tree smali round-trip. Whole-tree round-trip damages R8-optimized dex in ways that only show up at runtime. 6. **Repack and sign** — `references/repack-and-sign.md`. **Do not strip the whole `META-INF/`.** This single mistake destroys otherwise-correct builds. -7. **Verify on device** — `references/environment.md` + `references/verification.md`. Check: launches, the changed behavior actually changed, nothing unrelated broke, and **the app reaches its normal UI with no blocking dialog**. First prove the artifact actually changed on the device -- a package manager reporting success does not prove an interposed confirmation was accepted (P18). Capture continuously for the first ~20 seconds after launch; sampling gaps are how a blocking modal goes unseen (P20). +7. **Verify on device** — `references/environment.md` + `references/verification.md`. Check: launches, the changed behavior actually changed, nothing unrelated broke, and **the app reaches its normal UI with no blocking dialog**. First prove the artifact actually changed on the device -- a package manager reporting success does not prove an interposed confirmation was accepted (P18). Capture continuously for the first ~20 seconds after launch, **and look at the captures** — sampling gaps are how a blocking modal goes unseen (P20), and a burst of images that were never inspected is not evidence. If the accessibility tree is empty, the image is the primary evidence rather than a fallback. 8. **Log what you learned** — if a failure cost you more than thirty minutes, add it to `references/pitfalls.md`. That file is the most valuable artifact in this skill. ## Non-negotiable constraints @@ -59,6 +70,8 @@ Answer these before touching a tool. Every one of them changes the whole plan. - **Verify structure after every dex edit.** `scripts/dex_classdiff.py` must report zero differences in class set and access flags for classes you did not intend to change. - **Do not patch a method that is widely shared.** Before patching any helper, count its callers (`scripts/find_refs.py`). A `Long.valueOf` wrapper with 30 callers is not an ad-specific hook. - **Do not make an API fail to suppress a UI element.** A 404/400 on an endpoint that other features depend on takes the whole screen down with it. Suppress at the data-consumption or render layer instead. +- **Look before you conclude — and look while you wait.** Execute, capture, and **inspect**; do not drive and sleep blind. A screen that is actually looked at answers in one step what coordinate-guessing cannot answer in five: the layout moved, a different dialog is up, a countdown is frozen, the text on screen says exactly why. Where the thing you are waiting on is visible, a sample you can inspect beats a duration you hoped was right, and byte-identical samples mean nothing is going to change. `scripts/snap.py`; `references/environment.md` §look at the screen. +- **Put a timeout on every command, and calibrate it from measurement.** An unbounded call turns a stall into "the task stopped making progress", which is indistinguishable from slow work and costs hours silently. Time the operation once, record it, then derive the bound from it — that is what makes slow and hung distinguishable. A deadline that passes is a measurement, not a verdict. `references/long-task-discipline.md` §bound every wait. - **Every claim needs evidence.** "Probably", "should be", "in theory" are not findings. Either you observed it, or you label it unverified. - **"Done" means the user-visible outcome**, not an internal signal. A blocking dialog still on screen means the task is not done, however many errors disappeared from the log. Absence of a log line is absence of evidence, never evidence of success. - **Do not discard a route on compound evidence.** If a failure followed two simultaneous changes, the attribution is a hypothesis, not a finding. Re-run it single-variable before writing the route off -- mis-attributed failures have removed viable approaches for a long time. @@ -74,19 +87,19 @@ Load only what the current step needs. | `references/recon.md` | Starting any new sample; identifying packer, SDKs, code location, ABI | | `references/packers.md` | The app is packed/hardened, or an edit makes it die before your code runs. Also load before discarding any route as "blocked by the shell" | | `references/framework-runtimes.md` | The UI is not native (Flutter / React Native / Unity / Cordova), or Java-layer hooks fire zero times while the UI clearly works | -| `references/native-and-so.md` | Patching in a `.so`, needing code to run before the app's own code, or hand-built native payloads that crash inside the linker | +| `references/native-and-so.md` | Patching in a `.so`, needing code to run before the app's own code, hand-built native payloads that crash inside the linker, or **deciding which library/ABI is actually loaded and executing** | | `references/ad-removal.md` | Task involves ads, trackers, sponsored cards, splash/interstitial/reward | | `references/membership-and-limits.md` | Task involves VIP, subscription, paid content, unlock, "fully cracked" | | `references/server-api.md` | The behavior is decided by a response, or you need to know if a patch can even matter | | `references/dex-patching.md` | Any actual editing of dex/smali, choosing a patch layer, choosing a tool | | `references/repack-and-sign.md` | Rebuilding, signing, installing, or a repacked app misbehaves | -| `references/runtime-data.md` | Local state matters: DataStore, SharedPreferences, SQLite, protobuf caches, tokens | +| `references/runtime-data.md` | Local state matters: DataStore, SharedPreferences, SQLite, protobuf caches, tokens — **or your data edit keeps being reverted, or a stored value looks encrypted** | | `references/dynamic-frida.md` | Frida setup, hooking strategy, tracing caller chains, finding the real call site | -| `references/environment.md` | Device/emulator setup, root, ADB, networking, offline devices | +| `references/environment.md` | Device/emulator setup, root, ADB, networking, offline devices, emulator console control and recovery, **the preflight check to run before every experiment block** | | `references/verification.md` | Defining what "done" means; building the evidence chain | | `references/tls-and-cert.md` | One feature fails at runtime (login, registration, payment, an API-backed screen) while the rest of the app works | | `references/third-party-builds.md` | The input is a "cracked"/"modded" build you did not produce — audit it before trusting it | -| `references/long-task-discipline.md` | The task will run long, or you are resuming one. Live record, conclusion grading, drift checkpoints, handover | +| `references/long-task-discipline.md` | The task will run long, or you are resuming one. Live record, conclusion grading, drift checkpoints, **deliverable-form drift (rooted-only vs shippable)**, bound-your-waits, handover | | `references/pitfalls.md` | Always worth a skim before building. This is the failure catalogue. | ## Script index @@ -111,3 +124,7 @@ All scripts are parameterized and path-agnostic; pass paths explicitly. Run `--h | `scripts/frida_probe.js` | Four-layer runtime probe: app network layer + OkHttp + java.net + swallowed exception messages | | `scripts/run_probe.py` | Inject a probe, stream it to a timestamped log file, stay resident while you operate the app | | `scripts/tls_check.py` | Strict certificate check for one or more hosts (expired / wrong host / untrusted CA) | +| `scripts/preflight.py` | Read-only environment check before every experiment block: device, root, ABI/translation, clock skew, leftover proxy/forwards, dead device server. Run this before blaming a patch. | +| `scripts/lib_map.py` | What is **actually mapped** into a live process: per-library path, base, architecture (`ELF e_machine`), and classification (system / from-APK / runtime-materialized). Answers "which library and which ABI is really executing". | +| `scripts/blob_decode.py` | Decode an opaque stored value by searching the parameter space (base64/base64url/hex × rotation × deflate/zlib/gzip) instead of guessing, then re-encode an edited payload with the same parameters. | +| `scripts/snap.py` | Bounded burst screenshot + control-tree capture, with a stall detector and an explicit verdict on whether the accessibility tree is usable at all. Use it so you *look* at the screen instead of driving blind. | diff --git a/references/environment.md b/references/environment.md index 7d4f976..e3c3efe 100644 --- a/references/environment.md +++ b/references/environment.md @@ -65,11 +65,86 @@ Notes: - Some SDKs bypass the system proxy entirely (many ad SDKs do). Proxy logs therefore **undercount** traffic — do not conclude "no ad traffic" from proxy logs alone; confirm with a runtime DNS hook. - Clean up when done: `settings put global http_proxy :0`. +## Preflight — run this before every experiment block + +`scripts/preflight.py` checks, read-only, everything that silently fakes a failure: device +reachability, whether more than one device is attached without an explicit serial, root, the +runtime translation layer, device/host clock drift, a leftover device-wide proxy setting, stale +port forwards, a dead device server, the ABI the package manager actually chose, and the free +space left for installs. + +```bash +python scripts/preflight.py --pkg --expect-root +python scripts/preflight.py --cleanup # also clears a leftover proxy and stale forwards +``` + +**The rule it exists to enforce:** *do not attribute a failure to your patch while preflight is +dirty.* A surprising fraction of "my change broke it" is a device that was already in a bad state. +Costing thirty seconds here is cheaper than costing several rounds to a wrong conclusion, and the +wrong conclusion is the one that gets written down and trusted later. + ## Emulator notes -- Verify the app can install at all. INSTALL failures on emulators are common (ABI, min SDK, vendor checks) and are usually not related to your patch. -- Snapshot/rollback is the main advantage — use it to A/B two builds quickly. -- Never quote emulator behavior as proof for a device-only question (and vice versa). +Emulators are the right place to iterate and the wrong place to conclude. + +- **Verify the app can install at all.** INSTALL failures on emulators are common (ABI, min SDK, + vendor checks, a stale copy with a different signature) and are usually **not** related to your + patch. Get a clean install of the *unmodified* build working before you change anything. +- **Snapshot/rollback is the main advantage** — use it to A/B two builds quickly, and to get back + to a known-good state after a destructive experiment. +- **Never quote emulator behavior as proof for a device-only question** (and vice versa). An + emulator that runs the app is not evidence about a physical ARM device. +- **Most vendors ship a console binary that is far more reliable than the GUI.** Learn yours early; + it is what you will need when the GUI is unresponsive and adb is already down. + + | Vendor | Typical console binary | Useful verbs | + |---|---|---| + | LDPlayer | `dnconsole.exe` (next to the player exe) | `list2`, `launch --index N`, `reboot --index N`, `quitall` | + | MuMu | `MuMuManager.exe` | `info -v all`, `control -v N launch`, `control -v N shutdown` | + | AVD / Android Emulator | `emulator.exe` | `-list-avds`, `@`, `-no-window` | + | Genymotion | `gmtool` | `admin list`, `admin start` | + +- **Recovery order when `adb devices` goes empty.** Do these in order; the first two are the ones + people skip: + + 1. `adb kill-server && adb start-server` — clears a wedged host daemon. + 2. Re-run `adb devices`. If still empty, check whether the **emulator process is actually alive** + (`tasklist` / `ps`). A running process with **no listening port** means the VM never finished + booting its adb bridge — restarting the *device* is the only fix. + 3. Restart the instance through the vendor console (`reboot --index N`). Restarting via the + player binary's "launch" verb frequently leaves you with a process and still no adb. + 4. Only then consider that something is wrong with the host. + + Symptom worth memorising: **the emulator process exists but nothing is listening on the adb + port.** Distinguishing "not running" from "running but not booted" is what makes step 2 worth + doing — they have different fixes and only one of them needs the GUI. + +- **Root is a per-vendor setting, not a given.** Most emulators expose it as a toggle in their + settings ("root permission" / "ROOT 权限"); some ship a separate rooted image. Always verify + rather than assume: + + ```bash + adb -s shell "su -c id" # want: uid=0(root) + ``` + + A rooted emulator is usually the fastest environment for everything in this skill, which makes it + easy to forget that it is also the *least representative* one. + +- **Running two instances.** Different serials make two emulators genuinely disjoint, which is + useful for holding a clean control (original build, untouched state) alongside your workbench. + Keep them labelled, always pass `--serial`, and **do not run the same experiment on both** — the + value of a second device is independence, and duplicating work destroys it. Operate one at a + time; the other is a reference point, not extra throughput. + +- **Apps can detect the emulator** and change behaviour or refuse to run. If the app behaves + differently here than on hardware, that is a finding about the emulator, not about the app. Note + it and move the question to a real device. + +## Which architecture is actually executing + +Device selection and architecture are the same question. `getprop ro.product.cpu.abi` reports what +the device claims; it does **not** report what is executing. Read `native-and-so.md` §Cross-architecture +before you choose which library to patch, and `scripts/lib_map.py` to see the live truth. ## Install / reinstall @@ -110,15 +185,64 @@ If the control still does not react, bypass the UI path entirely — `am start - `input text` additionally mangles or drops non-ASCII input. For CJK text, either install an ADB-driven helper IME or set the field from a runtime call rather than typing. -### Screen evidence without pixels +### Look at the screen — do not drive and wait blind -`screencap` returns a 0-byte file on some ROMs. First try writing on-device and pulling: +The most expensive habit in device work is: tap a coordinate, sleep, tap again, sleep, conclude +something about the app. A screen that is *looked at* answers in one step what coordinate-guessing +cannot answer in five — the layout shifted, a different dialog came up, a countdown is frozen, the +button is disabled, the text on screen says exactly why. + +**Treat a look as a routine step, not a debugging last resort.** Capture: + +- **immediately before** anything time-dependent, so you know the starting state; +- **during** a wait, at intervals — state changes are the information, and a single sample at the end + cannot distinguish "it progressed" from "it never moved"; +- **at every decision point**, before choosing the next action; +- **on any surprise**, before forming a theory about it. + +`scripts/snap.py` does this with sane bounds, and tells you which kind of evidence you actually got: + +```bash +python scripts/snap.py --out shots --tag before +python scripts/snap.py --out shots --tag waiting --count 6 --interval 3 +``` + +**Use the stall detector.** If consecutive samples are byte-identical, nothing is happening and more +waiting cannot help. Stop, and go find out why — that is a different investigation from waiting +longer. + +### Two kinds of screen evidence, and which one is trustworthy here + +| Evidence | What it gives you | When it is the right one | +|---|---|---| +| **the image** | exactly what is rendered: layout, which dialog, disabled states, drawn text | always available; the **only** evidence for runtime-drawn UI | +| **the control tree** (`uiautomator dump`) | precise `bounds`, exact text, diffable | only when it actually has content | + +The tree is more convenient *when it exists* — it gives you tap coordinates and text you can diff. But +**verify it has content before planning around it**: + +```bash +adb -s shell "su -c 'uiautomator dump /sdcard/ui.xml'" +adb -s pull /sdcard/ui.xml +grep -c ' the tree is useless for this screen +``` + +**A runtime-rendered UI frequently exposes no real controls at all.** Cross-platform runtimes, web +views and canvas-drawn surfaces often produce an empty (or text-free) tree, which is why a plan built +on "read the bounds out of the XML" stalls on exactly those apps — and also why tapping a control that +"should be there" silently does nothing. When the tree is empty, the image is the **primary** evidence, +not a fallback, and you read it directly rather than trying to derive coordinates from a tree that does +not exist. + +`screencap` itself returns a 0-byte file on some ROMs. Write on-device and pull instead: ```bash adb -s shell "su -c 'screencap -p /sdcard/x.png'" && adb -s pull /sdcard/x.png ``` -If that is also empty, do not fight it: `uiautomator dump`'s XML is **better** evidence — text, diffable, and it records the real control tree and its contents. (If the dump fails with `could not get idle state`, retry once; a paused animation is usually the cause.) +If that is also empty, do not fight it — but note that **a 0-byte capture is not evidence the screen is +blank** (`pitfalls.md` P20). And if `uiautomator dump` fails with `could not get idle state`, retry once; +a paused animation is usually the cause. ### Verify form input by reading it back diff --git a/references/framework-runtimes.md b/references/framework-runtimes.md index f20ec01..e827f02 100644 --- a/references/framework-runtimes.md +++ b/references/framework-runtimes.md @@ -72,11 +72,37 @@ the snapshot rather than by pattern-matching machine code. - **Strings first.** User-visible text (dialog bodies, feature labels) is the cheapest anchor in any runtime. Find it, then find what references it. +- **Search them in the encoding the runtime actually uses, or you will conclude they do not exist.** + This is the single most common false negative in cross-platform work. A runtime snapshot does not + necessarily store text as UTF-8; UTF-16 (little-endian on these targets) is common and entirely + valid. A UTF-8 search of such a snapshot returns **zero hits**, which reads as "the strings were + stripped / encrypted, this route is dead" — and that conclusion is wrong. + + ```python + blob = open('libapp.so', 'rb').read() # or whichever artifact holds the snapshot + for phrase in ('', ''): + print(phrase, 'utf-8:', blob.find(phrase.encode('utf-8')), + 'utf-16le:', blob.find(phrase.encode('utf-16-le'))) + ``` + + Try both, and try a short distinctive substring rather than a long phrase. +- **A phrase that is absent may simply never exist as one literal.** UI text is often assembled from + fragments or templates, so "the whole sentence" can be missing while both halves are present. + Search the shortest distinctive token, and expect the interesting anchor to be a *label* rather + than a sentence. +- **Landing on a string tells you where the text lives, not which code decided to show it.** Treat the + hit as an anchor for reference-hunting, not as the answer. - **Compare two builds.** The same feature in a slightly different version often reveals the code path. - **Watch the boundary, not the interior.** For cross-platform apps it is usually far cheaper to observe what crosses between layers than to reverse the interior of the runtime. - **Do not assume the obfuscated names are stable or meaningful.** They are not, and treating them as identities leads to conclusions that break on the next build. +- **Obfuscated identifiers can be non-ASCII and invisible.** Renaming passes can replace class and + member names with characters outside ASCII (combining marks, variant selectors and similar). If a + class exists at runtime but "cannot be found" by the name you read from a decompiler, suspect + encoding or normalisation in your tooling rather than a missing class. Resolve such members by + **shape** — parameter counts, types, interface implementation — instead of by name, and prefer + runtime enumeration of loaded classes over guessing identifiers. ## What this changes about your plan diff --git a/references/long-task-discipline.md b/references/long-task-discipline.md index f895262..711b63d 100644 --- a/references/long-task-discipline.md +++ b/references/long-task-discipline.md @@ -83,6 +83,116 @@ Do these cheaply, and only when they can change a decision — not as ceremony. - **When resuming**: read the record first, and treat anything not in it as unknown, even if it feels familiar. +## The most expensive drift: solving it in an environment the deliverable will never see + +There is one drift in this domain that costs more than all the others, because it produces a +result that looks like success and is not one. + +**A runtime-only result is easy to obtain and easy to mistake for a finished artifact.** + +Editing a data file, hooking a live process, blocking a hostname, holding a proxy open — each of +these can make the app behave correctly *on the machine where you did it*, with no repackaging +required. It is often the fastest path to a visible win. It is also frequently **not the +deliverable at all**, because the requirement was never "make it work here". + +The constraint axes that decide this — check every one against the original request, and write +down the answer as a testable sentence before you start: + +| Axis | The question | Why it silently invalidates work | +|---|---|---| +| **Privilege** | must it run **unrooted**? | a rooted-only result cannot be given to a normal user at all | +| **Modification form** | must it be a **rebuilt/installable artifact**, or is a live-instrumentation result acceptable? | hooks and data edits do not ship inside an APK | +| **ABI / device class** | which ABI, which device family? | an emulator-only or x86-only result is not evidence for a physical ARM device | +| **Network** | must it work **online**? | a fix that depends on being offline or on a host proxy fails the moment it is used normally | +| **Persistence** | must it survive restart, upgrade, and a fresh install? | in-memory and session-scoped state evaporates | +| **Distribution** | does the *shipped file* have to be self-contained? | anything that needs a helper on the machine is not a shippable artifact | + +**Checkpoint question, asked at the same moments as the drift checks above:** + +> *If I handed over exactly what exists right now, would it satisfy the constraint sentence I wrote +> at the start?* + +If the answer is no, you have made real progress on a **component**, and that is worth stating as +such — but it is not the task. Do not let "it works" stand in for "it works under the constraint". + +**What to do instead of over-claiming.** Split the result into two explicitly labelled parts: + +1. **What is achievable inside the constraint**, and how far along it is. +2. **The unconstrained workaround** (needs root / needs a host / needs a proxy), stated as a + deliberate fallback, with its requirements made obvious to the reader. + +A privileged workaround is genuinely useful — it can be the difference between using the app and +not. It becomes a problem only when it is presented as the deliverable. Report it as a fallback, +keep the constrained goal open, and say plainly which one you have. + +**Also watch the inverse**: if the constraint is "must be a rebuilt artifact", do not let a working +runtime result quietly close the investigation. Use it for what it is good for — it proves the +mechanism and identifies the exact code or data to change — then port that finding into the +artifact. The runtime result is the map, not the destination. + +## Bound every wait + +A long task stalls in ways that produce no information and consume the most valuable resource you +have: wall-clock time and the next person's patience. Three habits prevent almost all of it. + +**1. Every command has a timeout, and its absence is the bug.** A helper that shells out without a +timeout can hang forever, and the failure presents as "the task stopped making progress" rather +than "this call blocked". Pass an explicit timeout on every subprocess, every HTTP request, every +device call. When one expires, the result is **unknown**, not failed — record it that way and +re-check state before retrying. + +**2. Calibrate the expected duration from measurement, not from a guess.** A timeout only means +something if it is set relative to how long the operation *should* take, because that is what makes +"slow" and "hung" distinguishable. Guessed budgets are either so short that healthy work gets killed, +or so long that a stall looks like patience. + +So: **measure once, write it down, then derive the bound.** + +| Operation class | How to bound it | +|---|---| +| host-side tool (`baksmali`, `apksigner`, a compiler) | seconds to low minutes; time it once, then set roughly 3–5× observed | +| device shell call (`adb shell …`) | seconds — **except** the first `su` after a reboot, which can prompt and block | +| install / push of a large artifact | minutes; scales with size, not with the app's complexity | +| app cold start to first frame | tens of seconds — **longer with a packer**, which does real work before your code runs | +| a step needing a human or an external service | **unbounded in principle** — do not wait at all (see P22) | +| waiting for an on-screen state change | bound it *and* sample it (point 3) | + +Record the observed durations in the live record. They are exactly the kind of mechanical fact that +gets re-derived painfully after a context loss, and having them turns a later timeout into an +interpretable signal instead of a mystery. + +**Exceeding the expected window is itself the finding.** It tells you something about state — the +device is wedged, the app never reached that phase, the action never happened — and that is a +different investigation from waiting longer. Re-check state instead of extending the deadline. + +**3. Every wait has a deadline, an observable to sample, and a look.** "Wait for the operation to +finish" is not a plan. Name the observable, how often you will sample it, how long you will sample +before giving up, and what you conclude on expiry: + +``` +waiting for: e.g. the overlay is gone / the counter incremented +sample: e.g. every 15 s: screenshot + current window focus +give up at: e.g. 150 s +on expiry: e.g. "did not complete within the window" -- not "cannot work" +``` + +**Sample with your eyes rather than sleeping blind.** When the step is gated on something visible — a +screen, a dialog, a progress indicator — capture it and **look at it** instead of sleeping through the +interval. A fixed sleep either wastes time or measures the wrong instant; a look tells you what +actually happened, and byte-identical consecutive samples tell you nothing is going to change +(`environment.md` §look at the screen). This is the cheapest way to avoid spending rounds on a state +that was never going to arrive. + +Three rules that follow: + +- **A deadline that passes is a measurement, not a verdict.** "Still not finished after N seconds" + tells you about latency and reliability. It does not tell you the approach is impossible, and + writing it down as impossible is how a viable route gets discarded. +- **Never busy-poll a long job while you have independent work.** Start it, do the other thing, and + collect it when it settles. Polling wastes the same resource the task is already spending. +- **Prefer a bounded observable over a fixed sleep.** Poll the state you actually care about, with a + cap — and prefer a sample you can inspect over a duration you hope is right. + ## Handover A handover is the record plus three things, written for someone with **no** memory of the task: diff --git a/references/native-and-so.md b/references/native-and-so.md index 6a0ff7e..6413ab2 100644 --- a/references/native-and-so.md +++ b/references/native-and-so.md @@ -101,12 +101,82 @@ Stripped and obfuscated libraries still expose structure: ## Cross-architecture notes -- **arm64** is the common case on modern devices; prefer it and make sure the host library you pick - actually exists for that ABI. -- **x86_64** appears on emulators. Shells and payloads are ABI-specific: a file built for one ABI will - not be loaded on the other, and the failure mode looks like a missing library rather than a mismatch. -- Keep your build artifacts per-ABI and label them. Mixing them is a common source of "it crashed for no - reason" when the same APK behaves differently on an emulator versus a device. +Architecture is not background information here — it decides **which artifact you must edit** and +**whether your hook can observe anything at all**. Get it from the runtime, not from the manifest. + +### What the device claims vs what is executing + +Three different answers, and they disagree more often than people expect: + +| Source | Answers | Command | +|---|---|---| +| The device | which ABIs the system supports | `getprop ro.product.cpu.abi` and `ro.product.cpu.abilist` | +| The package manager | **which ABI it chose for this app on this device** | `dumpsys package \| grep primaryCpuAbi` | +| The live process | **what is actually mapped right now** | `scripts/lib_map.py --pkg ` | + +Only the third is ground truth. The first two are predictions, and they are wrong exactly when it +matters: an emulator whose primary ABI is `x86_64` can be running an `arm64-v8a`-only app through a +translator, and a fat APK can have the package manager pick an ABI you did not assume. + +**Consequence: if the library you patched does not appear in the live mapping, your change cannot +matter.** That is a plan problem, and no amount of re-patching will fix it. + +### Translation layers change what you are observing + +When an ARM-only app runs on an x86 host, a translator (Intel Houdini, `libndk_translation`, a +`native_bridge` in general) is executing the guest code. Detect it from the maps — look for +translator marker libraries and for app libraries whose architecture differs from the host's +primary ABI (`lib_map.py` reports both). + +What it changes: + +- **Timing.** Translation is slower and less predictable. Anything you measured about latency is + about the translator, not the code. +- **Native integrity and anti-tamper checks can behave differently** under translation, in either + direction — a check that fails on hardware may pass here, and vice versa. A pass under translation + is not evidence of a pass on hardware. +- **Hook behaviour.** Intercepting translated code is not the same as intercepting native code: + address spaces, calling conventions, and what the host sees at a syscall boundary all differ. If a + native hook reports nothing while the feature plainly runs, suspect translation before suspecting + your script. +- **Syscall-level observation shows the host's view.** A file or network call made from the guest + may look different on the host side than it would natively. + +**Rule: verify the artifact on the ABI the user will actually run.** An emulator-only result is a +mid-task checkpoint, never the final claim (`verification.md`). + +### Fat APKs: patching the ABI that never loads + +An APK can ship `lib/arm64-v8a/`, `lib/armeabi-v7a/`, `lib/x86_64/`… The package manager extracts +**one** of them into the install-time native library directory, and only that one is loaded. + +- Check `primaryCpuAbi` and the live maps before editing a `.so`. Editing `arm64-v8a` while the + device loads `armeabi-v7a` produces a build that is byte-different and behaviorally identical. +- If the deliverable must work for an unknown user, remember their device may select a different + ABI than yours. Either patch every ABI present, or state which ABI your artifact targets. +- If the app's own libraries are written at *runtime* rather than extracted from the APK, the live + paths will not be inside the APK at all (`lib_map.py` marks these `materialized`). Those belong to + whatever produced them, and a repacked APK will not carry them in that location. + +### Payloads and tooling are per-ABI too + +- A native payload, an injected library, or a shellcode blob built for one ABI **will not load** on + another. The failure usually surfaces as a *missing library* or a generic linker error rather than + a clean "wrong architecture" message — do not read it as "my payload is broken". +- **Pointer size differs** between 32- and 64-bit targets. A script or struct layout that assumes + 8-byte words misbehaves silently on a 32-bit target. +- Keep build artifacts per-ABI and label them with the ABI in the filename. Mixing them up is a + reliable source of "it crashed for no reason" when the same APK behaves differently on an emulator + and on a device. + +### Quick decision list + +1. `scripts/lib_map.py --pkg --app-only` → which app libraries are loaded, from where, and at + what architecture. +2. If a translator is present and the task needs reliable native behaviour, move to a matching + device. +3. If the library you meant to patch is absent, stop — pick the right library first. +4. Only then start editing. ## Verification diff --git a/references/pitfalls.md b/references/pitfalls.md index 35e819f..878d917 100644 --- a/references/pitfalls.md +++ b/references/pitfalls.md @@ -493,3 +493,113 @@ an impression of work being done. - Detect stalls by change, not by elapsed time: if N consecutive samples are identical, break out. - Prefer driving the prompt to completion over waiting it out — the same prompt usually recurs, so automating it once pays back immediately. + +--- + +## P23. The fix lives in a file the app rewrites + +**Symptom** +You change a value in the app's data, the write succeeds, the file reads back correctly — and after +the next launch the value is back to what it was. Often byte for byte identical, which makes it look +like nothing happened at all. + +**Root cause** +The stored value is a **cache, not a source of truth**. Either the app re-fetches it and re-persists +it, or it rewrites the file from its own defaults on every start. Your edit was never authoritative. + +Compounding it: the intuitive way to protect the file — a restrictive mode or a changed owner — **does +not work**, because the app does not open-and-write the existing file. It **deletes the file and +creates a new one**, and a new file is created with the app's own mode and owner. Nothing is inherited, +so `chmod`/`chown` are silently ineffective. + +**Why it is hard to see** +Verification is usually done immediately after writing, while the file is still correct. The rewrite +only happens on the next start, which is one step further along than you looked. And the "fix" that +seems obviously right (tighten permissions) fails without any error. + +**Do instead** +- **Verify after a relaunch, not after the write.** The write succeeding is not the finding; surviving + a restart is. +- **Distinguish the two causes with one offline launch.** If the value survives with the network down, + it came from the server. If it does not, the app is regenerating it locally — and then a data edit is + the wrong layer entirely; patch the read path instead. +- To make a data edit stick, use the **immutable attribute** and confirm it took effect: + ```bash + su -c "chattr +i "; su -c "lsattr " # expect the 'i' flag + ``` + It is enforced by the filesystem against the delete itself, which is why it holds where permissions + do not. Undo with `chattr -i`. +- Then **exercise the feature**, not just the value. A blocked write the app depends on can make it + misbehave; "the file still has my value" is not "the app still works". +- Remember a lock is **device state, not artifact state** — it does not travel with an APK. Record it + as an environment requirement. The only form that ships is a code patch (`runtime-data.md`). + +--- + +## P24. The artifact changed, but the wrong one is executing + +**Symptom** +The build differs from the original, the pipeline reports success, the file on disk is genuinely +modified — and the app behaves exactly as before. Or a native hook reports nothing while the feature +plainly runs. + +**Root cause** +Something other than your edit is being used at runtime: + +- **Wrong ABI.** A fat APK ships several `lib//` directories; the package manager extracts and + loads **one**. Editing `arm64-v8a` while the device loads `armeabi-v7a` produces a byte-different, + behaviorally identical build. +- **The library is not from the APK at all.** Some libraries are written into the app's data directory + at runtime rather than extracted from the package. Patching the APK copy changes a file nobody loads. +- **Multiple processes.** The work was done in, or the check lives in, a different process than the one + you are observing. +- **A stale install.** The package manager reported success for a request that did not replace what is + on disk (`P18`). + +**Why it is hard to see** +Every local indicator agrees: the diff is non-empty, the build is signed, the install returned success. +Nothing in the *build* pipeline can detect this, because the build is fine. The contradiction only +exists at runtime. + +**Do instead** +- **Ask the running process what it loaded**, before editing: `scripts/lib_map.py --pkg `. + Libraries whose path is not under the installed APK's lib directory were materialized at runtime + and belong to whatever produced them. +- Confirm the ABI the package manager actually chose (`dumpsys package | grep primaryCpuAbi`) + rather than the one you assumed from the manifest. +- **If the library you patched is not in the live mapping, stop.** No amount of re-patching helps; the + plan is wrong. +- Treat a behaviorally identical rebuild as **positive evidence that your edit is not being loaded**, + not as "the change had no effect". Those are different conclusions and only one of them is actionable. + +--- + +## P25. "The search found nothing" treated as "the data is not there" + +**Symptom** +You scan an artifact for a known-present value — a UI label, a marker string, an endpoint — get zero +hits, and conclude the content is stripped, encrypted, or otherwise unavailable. A route gets written +off on that basis. + +**Root cause** +The search used the wrong representation. A byte scan for UTF-8 text returns nothing against content +that is stored as UTF-16, or compressed, or framed inside a container, or split across fragments. The +data is present; the needle was encoded differently from the haystack. + +**Why it is hard to see** +"Zero results" is a clean, confident-looking output. It feels like a measurement, so it gets recorded +as a finding, and findings propagate into the plan. + +**Do instead** +- **Before concluding absence, search more than one encoding.** UTF-8 and UTF-16LE will between them + cover most text storage: + ```python + blob.find(needle.encode('utf-8')), blob.find(needle.encode('utf-16-le')) + ``` +- **Search the shortest distinctive fragment.** Text is often assembled from pieces or templates, so a + full sentence can be absent while its parts are present. +- **Try the value without its framing.** A hit rate of zero is also the expected result for content + that is inside a compressed or encoded container — decode the container first (`runtime-data.md`, + `scripts/blob_decode.py`). +- **State a negative result with its scope**: "no UTF-8 or UTF-16LE literal match in this artifact" + is a finding. "The string does not exist" is a guess wearing a finding's clothes. diff --git a/references/runtime-data.md b/references/runtime-data.md index 38c4506..103e0a5 100644 --- a/references/runtime-data.md +++ b/references/runtime-data.md @@ -105,8 +105,85 @@ If the effect must ship inside an APK, move it into code. Patterns, in order of Reference case: a promo popup was gated by `_expires_at`. Writing a large value via DataStore suppressed it on the test device, but a fresh install lost it. The durable fix replaced the dedicated map-lambda that produces the value so it always yields a far-future timestamp — a single-purpose class, safe to patch, no effect on the shared serialization helpers. +## When the app rewrites your edit + +A very common sequence: you change the value, relaunch, and the value is **back** — often byte for +byte identical to what it was before. That is not a failed write. It is the app re-establishing the +value from a source of truth you have not touched yet. + +Two causes look identical and need different fixes: + +| Cause | How to tell | Fix | +|---|---|---| +| The app **re-fetches from the server** and re-persists | take the app offline and relaunch: does your value survive? | lock the file, block the refresh request, or move the change into code | +| The app **rewrites the file on every start** from its own defaults | your value does not survive even with the network down | the value is not authoritative — patch the read path instead | + +Do the offline test first. It costs one launch and separates the two cases. + +### Making a data edit stick (in order of durability) + +1. **Immutable file attribute** — the reliable, reversible way to stop a rewrite: + ```bash + su -c "chattr +i " + su -c "lsattr " # expect the 'i' flag, e.g. -----i------- + # to undo: + su -c "chattr -i " + ``` + Verify with `lsattr`, not by assuming the command worked. + + **Why permissions do not achieve this.** Tightening the mode or changing the owner looks like the + obvious move and does not work, because the app does not open-and-write the existing file — it + **deletes the file and creates a new one**. The new file is created by the app, with the app's own + mode and owner, so a restrictive `chmod` or a `chown root` is simply not inherited. The immutable + attribute is enforced by the filesystem against the delete itself, which is why it holds. + +2. **Keep the app offline** so the refresh source is unreachable. Effective for a test, and + sometimes acceptable in use — but be explicit about it, because a fix that only works offline + fails the "works under normal conditions" constraint. See the deliverable-drift section in + `long-task-discipline.md` before presenting this as a result. + +3. **Block the refresh request** at the runtime or network layer — durable while your instrumentation + is running, and requires you to have identified the endpoint (`references/server-api.md`). + +4. **Move the change into code** — patch the read path or the decision. This is the only form that + ships inside an APK (`§Making the fix durable` above). + +### Two cautions about locking + +- **Verify the feature, not the value.** Blocking a write the app depends on can make it misbehave + or fail loudly. After locking, exercise the feature you care about — "the file still has my value" + is not the same as "the app still works". +- **A lock is device state; it does not travel with an APK.** If the recipe requires it, record it as + an environment requirement, so nobody later mistakes it for something baked into the artifact. + +## Encoded values: do not assume "encrypted" + +Preference values that carry server configuration are frequently stored as a single opaque string +rather than readable XML. Before concluding the value is encrypted, check whether it is merely +**framed**: a common shape is `base64( rotate( deflate( json ) ) )`, where the cyclic rotation exists +precisely so a naive decode fails. + +`scripts/blob_decode.py` searches the parameter space instead of guessing it — outer encoding +(base64 / base64url / hex), rotation offset, and compression type — and reports every combination +that yields a structured document, plus the exact parameters to re-encode your edited payload: + +```bash +# pull the value straight out of a preferences XML and decode it +python scripts/blob_decode.py --prefs-xml prefs.xml --name # (see --help for exact name) + +# after editing the decoded payload, rebuild the value with the winning parameters +python scripts/blob_decode.py --encode --file decoded.bin --outer base64 --inner raw --cut +``` + +The rotation search is cheap — a few tens of thousands of candidates resolve in well under a second +— so there is no reason to guess. **Distinguish framing from real encryption before spending time +on a key:** a framed blob becomes structured the moment you remove the framing, whereas a keyed blob +stays random-looking. If it stays random, treat it as opaque and move to the code path that consumes +it. + ## Cautions -- Editing data while the app is running is unreliable: in-memory caches win. -- The server may overwrite your value on next sync. If the app re-fetches and re-persists the authoritative value, a data edit is temporary by design — which is exactly why the durable version belongs in code. +- Editing data while the app is running is unreliable: in-memory caches win. Force-stop, edit, then launch. +- The server may overwrite your value on next sync — see *When the app rewrites your edit* above. - Do not edit a token you do not own and expect it to be accepted; tokens are validated server-side (`references/server-api.md`). +- A restored data directory needs ownership fixed again after every reinstall, because the uid increments. diff --git a/scripts/blob_decode.py b/scripts/blob_decode.py new file mode 100644 index 0000000..1ec82ea --- /dev/null +++ b/scripts/blob_decode.py @@ -0,0 +1,276 @@ +#!/usr/bin/env python3 +"""Decode (and re-encode) an opaque single-value blob cached by an app. + +Many apps cache server-issued configuration as ONE opaque string inside a +preferences file or a key-value store. The encoding is usually some stack of: + outer: base64, base64url, or hex + transform: a cyclic rotation of the byte stream (so the compressed stream + does not begin at its header), sometimes preceded by a short + header to skip + inner: raw deflate, zlib, or gzip + +You do not have to guess the parameters. This script searches them and reports +every combination that decodes to something structured, then can re-encode your +edited payload with the same parameters so it can be written back. + +Why rotation matters: a plain base64->inflate attempt fails on these blobs, and +the failure looks like "this value is encrypted". Exhaustively trying cyclic +shifts is cheap and frequently succeeds, which turns an opaque blob into an +editable JSON/XML document. + +Usage: + # decode the value of a key inside a prefs XML + python blob_decode.py --prefs-xml prefs.xml --name flutter.remoteConfig + + # decode a blob stored in its own file (or piped on stdin) + python blob_decode.py --file value.txt + cat value.txt | python blob_decode.py + + # keep searching harder (slower) and write the best result out + python blob_decode.py --file value.txt --max-skip 64 --out decoded.bin + + # re-encode an edited payload with the parameters that worked + python blob_decode.py --encode --file decoded.bin --outer base64 --inner raw --cut 15435 --out value.txt + +Notes: + * Output is written to --out; nothing on device is touched. + * The winning parameters are printed as a copy-pasteable line for --encode. +""" +from __future__ import annotations + +import argparse +import base64 +import binascii +import json +import re +import sys +import time +import zlib + +INNERS = [("raw", -15), ("zlib", 15), ("gzip", 31)] + + +# --------------------------------------------------------------------------- # +# outer encodings +# --------------------------------------------------------------------------- # +def outer_decode(text: str, which: str) -> bytes | None: + t = text.strip().replace("\n", "").replace("\r", "").replace(" ", "") + try: + if which == "base64": + return base64.b64decode(t + "=" * (-len(t) % 4)) + if which == "base64url": + return base64.urlsafe_b64decode(t + "=" * (-len(t) % 4)) + if which == "hex": + return binascii.unhexlify(re.sub(r"[^0-9a-fA-F]", "", t)) + except Exception: + return None + return None + + +def outer_encode(data: bytes, which: str) -> str: + if which == "base64": + return base64.b64encode(data).decode("ascii") + if which == "base64url": + return base64.urlsafe_b64encode(data).decode("ascii") + if which == "hex": + return binascii.hexlify(data).decode("ascii") + raise ValueError("unknown outer encoding: %s" % which) + + +def sniff_outer(text: str) -> list[str]: + """Cheap guess first so the common case is instant.""" + t = text.strip() + guesses = [] + if re.fullmatch(r"[0-9a-fA-F\s]+", t) and len(t) >= 16: + guesses.append("hex") + if re.fullmatch(r"[A-Za-z0-9+/\s=]+", t): + guesses.append("base64") + if re.fullmatch(r"[A-Za-z0-9\-_\s=]+", t): + guesses.append("base64url") + # always keep base64 last-resort, then everything else + for w in ("base64", "base64url", "hex"): + if w not in guesses: + guesses.append(w) + return list(dict.fromkeys(guesses)) + + +# --------------------------------------------------------------------------- # +# inner transforms +# --------------------------------------------------------------------------- # +def try_inflate(body: bytes, max_out: int | None = None) -> tuple[str, bytes] | None: + best = None + for name, wbits in INNERS: + try: + if max_out is None: + out = zlib.decompress(body, wbits) + else: + d = zlib.decompressobj(wbits) + out = d.decompress(body, max_out) + if out: + if best is None or len(out) > len(best[1]): + best = (name, out) + except Exception: + continue + return best + + +def inflate_raw(data: bytes) -> bytes: + co = zlib.compressobj(9, zlib.DEFLATED, -15) + return co.compress(data) + co.flush() + + +def score(out: bytes) -> float: + """How much does this look like a real document we can edit?""" + if not out: + return -1.0 + printable = sum(1 for b in out[:4096] if 32 <= b < 127 or b in (9, 10, 13)) + ratio = printable / min(len(out), 4096) + s = ratio * 10.0 + head = out[:64].lstrip() + if head[:1] in (b"{", b"[", b"<", b"#"): + s += 6.0 + if b'"' in out[:512] or b"=" in out[:512]: + s += 2.0 + if len(out) < 32: + s -= 3.0 + return s + + +# --------------------------------------------------------------------------- # +def search(blob: bytes, max_skip: int, log) -> list[dict]: + hits: list[dict] = [] + n = len(blob) + t0 = time.time() + + for skip in range(0, min(max_skip, n) + 1): + base = blob[skip:] + m = len(base) + if m < 8: + break + for k in range(m): + cand = base[k:] + base[:k] + got = try_inflate(cand, max_out=8_000_000) + if got: + name, out = got + hits.append({ + "skip": skip, "cut": k, "inner": name, "size": len(out), + "score": round(score(out), 2), "data": out, + }) + if hits and log and skip == 0: + log(" first hit found with skip=0 after %.1fs" % (time.time() - t0)) + if log and skip and skip % 16 == 0 and time.time() - t0 > 30: + log(" ... skip=%d, %.0fs elapsed" % (skip, time.time() - t0)) + log(" search finished in %.1fs (%d candidate(s))" % (time.time() - t0, len(hits))) + return hits + + +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + src = ap.add_mutually_exclusive_group() + src.add_argument("--file", help="file containing the encoded value") + src.add_argument("--prefs-xml", help="a preferences XML file to pull the value out of") + ap.add_argument("--name", help="key name when using --prefs-xml") + ap.add_argument("--outer", choices=["base64", "base64url", "hex", "auto"], default="auto") + ap.add_argument("--max-skip", type=int, default=0, + help="also try dropping 0..N leading bytes before the rotation search (slower)") + ap.add_argument("--min-size", type=int, default=200, help="ignore results smaller than this") + ap.add_argument("--top", type=int, default=5, help="how many candidates to print") + ap.add_argument("--out", help="write the best decoded result here") + + ap.add_argument("--encode", action="store_true", help="re-encode instead of decoding") + ap.add_argument("--inner", choices=[n for n, _ in INNERS], default="raw") + ap.add_argument("--cut", type=int, default=0, help="rotation used for write-back") + a = ap.parse_args() + + def log(msg: str) -> None: + print(msg, file=sys.stderr) + + # ------------------------------------------------------------------ encode + if a.encode: + if not a.file: + log("--encode requires --file (the payload to encode)") + return 2 + payload = open(a.file, "rb").read() + body = inflate_raw(payload) if a.inner == "raw" else zlib.compress(payload, 9) + if a.inner == "zlib": + body = zlib.compress(payload, 9) + cut = a.cut % max(len(body), 1) + rotated = body[-cut:] + body[:-cut] if cut else body + enc = outer_encode(rotated, "base64" if a.outer == "auto" else a.outer) + if a.out: + open(a.out, "w", encoding="utf-8", newline="\n").write(enc) + log("wrote %d chars to %s" % (len(enc), a.out)) + else: + print(enc) + log("write-back parameters: outer=%s inner=%s cut=%d" + % ("base64" if a.outer == "auto" else a.outer, a.inner, cut)) + return 0 + + # ------------------------------------------------------------------ input + if a.prefs_xml: + if not a.name: + log("--prefs-xml requires --name ") + return 2 + xml = open(a.prefs_xml, encoding="utf-8", errors="replace").read() + m = re.search(r'name="%s"[^>]*>([^<]*)<' % re.escape(a.name), xml) + if not m: + log("key not found in XML: %s" % a.name) + log("keys present: %s" % ", ".join(re.findall(r'name="([^"]+)"', xml)[:40])) + return 2 + text = m.group(1) + log("value length: %d" % len(text)) + elif a.file: + text = open(a.file, encoding="utf-8", errors="replace").read() + else: + text = sys.stdin.read() + + if not text.strip(): + log("empty input") + return 2 + + # ------------------------------------------------------------------ decode + candidates = [a.outer] if a.outer != "auto" else sniff_outer(text) + for which in candidates: + blob = outer_decode(text, which) + if blob is None or len(blob) < 8: + log("outer=%s -> not decodable, skipping" % which) + continue + log("outer=%s -> %d bytes; searching rotation (and skip<=%d)" % (which, len(blob), a.max_skip)) + hits = [h for h in search(blob, a.max_skip, log) if h["size"] >= a.min_size] + if not hits: + continue + hits.sort(key=lambda h: (-h["score"], -h["size"])) + print("=" * 66) + print("outer=%s candidates=%d" % (which, len(hits))) + for h in hits[:a.top]: + print(" score=%-6s skip=%-4d cut=%-7d inner=%-5s size=%d" + % (h["score"], h["skip"], h["cut"], h["inner"], h["size"])) + best = hits[0] + print("-" * 66) + print("write-back line:") + print(" python blob_decode.py --encode --file --outer %s --inner %s --cut %d" + % (which, best["inner"], best["cut"])) + print("-" * 66) + preview = best["data"][:1200] + try: + print(preview.decode("utf-8")) + except UnicodeDecodeError: + print(preview.decode("latin1")) + print("(non-UTF-8; first 64 bytes: %s)" % best["data"][:64].hex()) + if a.out: + open(a.out, "wb").write(best["data"]) + log("wrote %d bytes to %s" % (len(best["data"]), a.out)) + return 0 + + print("no decodable candidate found.", file=sys.stderr) + print("Next steps:", file=sys.stderr) + print(" * raise --max-skip, or pass --outer explicitly", file=sys.stderr) + print(" * the blob may be genuinely encrypted (a keyed cipher) rather than merely", file=sys.stderr) + print(" encoded -- confirm before spending time on it: a keyed blob should look", file=sys.stderr) + print(" random, whereas an encoded one still has structure once the framing is", file=sys.stderr) + print(" removed. See references/runtime-data.md.", file=sys.stderr) + return 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/lib_map.py b/scripts/lib_map.py new file mode 100644 index 0000000..a1ac64e --- /dev/null +++ b/scripts/lib_map.py @@ -0,0 +1,218 @@ +#!/usr/bin/env python3 +"""Report which native libraries are ACTUALLY mapped into a live process. + +`getprop ro.product.cpu.abi` tells you what the device claims to be. It does not +tell you what is executing. Those differ more often than people expect: + + * an x86_64 emulator running an app whose only native libraries are arm64, + because an ARM translation layer is present; + * a hardened app whose real libraries are materialized into its data + directory at runtime, so the paths you see are not the paths inside the APK; + * a fat APK where the package manager picked a different ABI than you assumed, + so you patch a `.so` that never loads. + +This script reads the live mapping table and reports, per library: the path, the +base address, and the architecture of the actual ELF on disk. It classifies each +path so the interesting ones stand out (system / from-APK / runtime-materialized). + +Requires root to read another process's maps. + +Usage: + python lib_map.py --pkg com.example.app + python lib_map.py --pkg com.example.app --app-only + python lib_map.py --pid 1234 --all + python lib_map.py --pkg com.example.app --json + python lib_map.py --pkg com.example.app --grep ssl + +Reading it: + * A library whose path sits under the app's data dir (not under the APK's + lib/ dir) was written at runtime. That is a packer/dropper fingerprint + and it will not be present in a repacked APK -- patch or replace with care. + * If app libraries are 64-bit ARM but the device primary ABI is x86_64, + a translator is in play. Timing and some native checks will differ. + * If the library you patched does not appear here at all, your patch cannot + matter. That is a plan problem, not a patch problem. +""" +from __future__ import annotations + +import argparse +import json +import os +import re +import subprocess +import sys +from collections import defaultdict + +E_MACHINE = { + 0x03: "x86", 0x3E: "x86_64", 0x28: "arm", 0xB7: "aarch64", + 0x08: "mips", 0xF3: "riscv", 0x16: "s390", 0x15: "ppc", +} +TRANSLATOR_MARKERS = ("houdini", "libndk", "native_bridge", "libntv", "armtrans", + "libhoudini", "ndk_translation", "libarm", "exagear", "box64") + + +def run(cmd: list[str], timeout: int = 60) -> str: + try: + p = subprocess.run(cmd, capture_output=True, text=True, errors="replace", timeout=timeout) + return (p.stdout or "") + (p.stderr or "") + except Exception as e: + return "ERROR: %s" % e + + +def classify(path: str, pkg: str | None) -> str: + if path.startswith(("/system/", "/apex/", "/vendor/", "/product/", "/odm/", "/system_ext/")): + return "system" + if re.search(r"/data/app/", path): + return "apk-lib" + if pkg and (("/data/data/%s" % pkg) in path or ("/data/user/0/%s" % pkg) in path): + return "materialized" + if "/data/" in path: + return "data-other" + return "other" + + +def read_elf_arch(sh_fn, path: str) -> str: + """Read the ELF header on the device and name the machine. '?' means the read failed.""" + hdr = sh_fn("dd if='%s' bs=20 count=1 2>/dev/null | od -An -tx1 2>/dev/null" % path) + hexs = re.sub(r"[^0-9a-fA-F]", "", hdr) + if len(hexs) < 40: + return "?" + b = bytes.fromhex(hexs[:40]) + if b[:4] != b"\x7fELF": + return "not-elf" + cls = {1: 32, 2: 64}.get(b[4], 0) + little = b[5] == 1 + mach = int.from_bytes(b[18:20], "little" if little else "big") + if mach in E_MACHINE: + return E_MACHINE[mach] # the machine name already implies the width + return "0x%02x/%db" % (mach, cls or 0) + + +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + tgt = ap.add_mutually_exclusive_group(required=True) + tgt.add_argument("--pkg", help="target package name") + tgt.add_argument("--pid", type=int, help="target pid") + ap.add_argument("--adb", default=os.environ.get("ADB", "adb")) + ap.add_argument("--serial", default=None) + ap.add_argument("--all", action="store_true", help="include system libraries") + ap.add_argument("--app-only", action="store_true", help="only APK libs and runtime-materialized libs") + ap.add_argument("--grep", default=None, help="only paths matching this regex") + ap.add_argument("--no-arch", action="store_true", help="skip reading ELF headers (faster)") + ap.add_argument("--max-libs", type=int, default=120, help="cap how many libraries get an arch probe") + ap.add_argument("--json", action="store_true", dest="as_json") + a = ap.parse_args() + + def sh(cmd: str, timeout: int = 60) -> str: + base = [a.adb] + (["-s", a.serial] if a.serial else []) + ["shell", cmd] + return run(base, timeout) + + def su(cmd: str, timeout: int = 90) -> str: + base = [a.adb] + (["-s", a.serial] if a.serial else []) + ["shell", 'su -c "%s"' % cmd] + return run(base, timeout) + + pid = a.pid + if a.pkg and not pid: + out = sh("pidof %s" % a.pkg).strip() + if not out: + print("no running process for %s -- launch it and let it settle first" % a.pkg, file=sys.stderr) + return 2 + pid = int(out.split()[0]) + + maps = su("cat /proc/%d/maps" % pid) + if not maps.strip() or "No such file" in maps: + print("cannot read /proc/%d/maps (root required, or the process is gone)" % pid, file=sys.stderr) + return 2 + + entries: dict[str, dict] = {} + for line in maps.splitlines(): + m = re.match(r"^([0-9a-f]+)-([0-9a-f]+)\s+(\S+)\s+([0-9a-f]+)\s+\S+\s+\S+\s*(.*)$", line) + if not m: + continue + start, end, perms, off, path = m.groups() + path = path.strip() + if not path or path.startswith("["): + continue + if not path.endswith(".so") and ".so." not in path: + continue + e = entries.setdefault(path, {"path": path, "perms": set(), "base": start, "count": 0}) + e["perms"].add(perms) + e["count"] += 1 + if perms.startswith("r-x") or perms.startswith("r--"): + e["base"] = min(e["base"], start) + + dev_abi = sh("getprop ro.product.cpu.abi").strip() + abilist = sh("getprop ro.product.cpu.abilist").strip() + pkg = a.pkg + if not pkg: + cmdline = su("cat /proc/%d/cmdline" % pid).split("\x00")[0].strip() + pkg = cmdline or None + + rows = [] + probed = 0 + for path, e in sorted(entries.items()): + kind = classify(path, pkg) + if a.app_only and kind not in ("apk-lib", "materialized"): + continue + # --grep is an explicit narrow request, so honour it across all kinds + if not a.all and not a.app_only and not a.grep and kind == "system": + continue + if a.grep and not re.search(a.grep, path): + continue + rows.append({"path": path, "base": e["base"], "kind": kind, + "perms": "/".join(sorted(e["perms"])), "maps": e["count"], "arch": "?"}) + + if not a.no_arch: + for r in rows[:a.max_libs]: + r["arch"] = read_elf_arch(su, r["path"]) + probed += 1 + + # translation markers + markers = sorted({p for p in entries if any(t in p.lower() for t in TRANSLATOR_MARKERS)}) + app_archs = sorted({r["arch"] for r in rows if r["kind"] in ("apk-lib", "materialized") and r["arch"] not in ("?", "not-elf")}) + host_is_x86 = "x86" in dev_abi + translated = bool(markers) or (host_is_x86 and any(x.startswith(("arm", "aarch64")) for x in app_archs)) + + if a.as_json: + print(json.dumps({"pid": pid, "pkg": pkg, "device_abi": dev_abi, "abilist": abilist, + "translation_suspected": translated, "translator_markers": markers, + "libs": rows}, indent=2, ensure_ascii=False)) + return 0 + + print("pid=%s pkg=%s" % (pid, pkg or "?")) + print("device primary abi=%s abilist=[%s]" % (dev_abi, abilist)) + print("%-13s %-9s %-6s %-6s %s" % ("arch", "kind", "perms", "maps", "path")) + print("-" * 100) + for r in rows: + print("%-13s %-9s %-6s %-6d %s" % (r["arch"], r["kind"], r["perms"], r["maps"], r["path"])) + print("-" * 100) + print("mapped libraries shown: %d" % len(rows)) + if not a.no_arch and probed < len(rows): + print("(architecture probed for the first %d; raise --max-libs to cover the rest -- '?' means not probed)" + % probed) + + if markers: + print("\nTRANSLATOR MARKERS PRESENT:") + for mk in markers: + print(" " + mk) + print(" -> native ARM code is being translated. Timing differs from a real ARM device,") + print(" and native integrity checks may behave differently. Verify the final artifact") + print(" on the ABI the user actually runs.") + if app_archs: + print("\napp library architectures: %s" % ", ".join(app_archs)) + if translated: + print("translation suspected: YES") + mat = [r["path"] for r in rows if r["kind"] == "materialized"] + if mat: + print("\nRUNTIME-MATERIALIZED libraries (not inside the APK):") + for p in mat[:20]: + print(" " + p) + print(" -> written at runtime by the app or its shell. A repacked APK will not contain") + print(" them in this location; account for that before assuming your patch is loaded.") + if not rows: + print("\nnothing matched. Widen with --all, or check that the process is the one that draws the UI.") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/preflight.py b/scripts/preflight.py new file mode 100644 index 0000000..aff032f --- /dev/null +++ b/scripts/preflight.py @@ -0,0 +1,250 @@ +#!/usr/bin/env python3 +"""Preflight: prove the environment is sane BEFORE you blame your patch. + +Run this at the start of every experiment block, and again whenever something +fails in a way you did not expect. A large share of "my patch broke the app" +turns out to be device state, a dead device server, a leftover proxy setting, or +a clock skew -- all of which look exactly like a broken artifact. + +It is deliberately read-only: it changes nothing except optional cleanup of a +device-wide proxy setting and port forwards it created itself. + +Usage: + python preflight.py + python preflight.py --serial --pkg com.example.app + python preflight.py --pkg com.example.app --expect-root + python preflight.py --cleanup # remove proxy + frida port forwards it finds + python preflight.py --json # machine-readable summary + +Exit code is 0 when no BLOCKER was found, 1 otherwise. WARN never fails the run. +""" +from __future__ import annotations + +import argparse +import json +import os +import re +import shutil +import subprocess +import sys + +BLOCKER, WARN, OK, INFO = "BLOCKER", "WARN", "OK", "INFO" + + +class Report: + def __init__(self) -> None: + self.rows: list[tuple[str, str, str, str]] = [] + + def add(self, level: str, area: str, detail: str, fix: str = "") -> None: + self.rows.append((level, area, detail, fix)) + + def worst(self) -> str: + for lv in (BLOCKER, WARN, OK, INFO): + if any(r[0] == lv for r in self.rows): + return lv + return OK + + +def run(cmd: list[str], timeout: int = 60) -> tuple[int, str]: + try: + p = subprocess.run(cmd, capture_output=True, text=True, errors="replace", timeout=timeout) + return p.returncode, (p.stdout or "") + (p.stderr or "") + except FileNotFoundError: + return 127, "not found: %s" % cmd[0] + except subprocess.TimeoutExpired: + return 124, "TIMEOUT after %ss: %s" % (timeout, " ".join(cmd[:4])) + except Exception as e: # pragma: no cover - defensive + return 1, "error: %s" % e + + +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument("--adb", default=os.environ.get("ADB", "adb"), help="adb executable (default: $ADB or 'adb')") + ap.add_argument("--serial", default=None, help="device serial; required when more than one is attached") + ap.add_argument("--pkg", default=None, help="target package to check (optional)") + ap.add_argument("--expect-root", action="store_true", help="treat missing root as a BLOCKER") + ap.add_argument("--cleanup", action="store_true", help="remove leftover proxy setting and frida forwards") + ap.add_argument("--json", action="store_true", dest="as_json") + a = ap.parse_args() + + r = Report() + ADB = a.adb + + # ---- host tools ------------------------------------------------------- + for tool, why in ( + ("java", "dexlib2 patcher, apksigner"), + ("python", "all scripts"), + ): + found = shutil.which(tool) + r.add(OK if found else WARN, "host:" + tool, found or "not on PATH", "" if found else "install it if you need " + why) + for tool in ("apksigner", "zipalign", "aapt", "aapt2"): + found = shutil.which(tool) + r.add(OK if found else INFO, "host:" + tool, found or "not on PATH", + "" if found else "only needed for signing/manifest work; build-tools on PATH helps") + + # ---- adb + device selection ------------------------------------------ + if shutil.which(ADB) is None and not os.path.exists(ADB): + r.add(BLOCKER, "adb", "executable not found: %s" % ADB, "pass --adb or set $ADB") + return finish(r, a.as_json) + + rc, out = run([ADB, "devices", "-l"]) + if rc != 0: + r.add(BLOCKER, "adb", "adb devices failed: %s" % out.strip()[:160], "start the adb server; check USB/emulator") + return finish(r, a.as_json) + + devices = [] + for line in out.splitlines()[1:]: + line = line.strip() + if not line: + continue + parts = line.split() + if len(parts) >= 2: + devices.append((parts[0], parts[1], line)) + online = [d for d in devices if d[1] == "device"] + + if not devices: + r.add(BLOCKER, "device", "no device attached", "boot the emulator/connect the device, then re-run") + return finish(r, a.as_json) + if len(online) > 1 and not a.serial: + r.add(BLOCKER, "device", "%d devices online but no --serial given" % len(online), + "pass --serial explicitly, or adb will pick one at random and you will debug the wrong target") + if not online: + bad = ", ".join("%s(%s)" % (d[0], d[1]) for d in devices) + r.add(BLOCKER, "device", "attached but not usable: %s" % bad, + "unauthorized -> accept the USB debugging prompt; offline -> reconnect/reboot the emulator") + return finish(r, a.as_json) + + serial = a.serial or online[0][0] + r.add(OK, "device", "using %s (of %d online)" % (serial, len(online)), "") + + def sh(cmd: str, timeout: int = 60) -> str: + return run([ADB, "-s", serial, "shell", cmd], timeout=timeout)[1] + + def su(cmd: str, timeout: int = 90) -> str: + return run([ADB, "-s", serial, "shell", 'su -c "%s"' % cmd], timeout=timeout)[1] + + # ---- device facts ----------------------------------------------------- + abi = sh("getprop ro.product.cpu.abi").strip() + abilist = sh("getprop ro.product.cpu.abilist").strip() + rel = sh("getprop ro.build.version.release").strip() + sdk = sh("getprop ro.build.version.sdk").strip() + model = sh("getprop ro.product.model").strip() + sec = sh("getenforce").strip() + + r.add(OK, "device:os", "Android %s (sdk %s) %s" % (rel, sdk, model), "") + r.add(OK, "device:abi", "%s abilist=[%s]" % (abi, abilist), "") + r.add(OK if sec.lower() in ("permissive", "") else INFO, "device:selinux", sec or "?", "") + + # translation layer / emulator hints -- these change how native code behaves + trans = [] + for probe in ("ro.dalvik.vm.native.bridge", "ro.enable.native.bridge.exec", "ro.boot.native_bridge"): + v = sh("getprop " + probe).strip() + if v and v != "0": + trans.append("%s=%s" % (probe, v)) + r.add(WARN if trans else OK, "device:translation", + "; ".join(trans) if trans else "no ARM translation declared", + "native arm libraries run through a translator here: timing differs and some native checks misbehave" if trans else "") + if re.search(r"^1$", sh("getprop ro.kernel.qemu").strip()): + r.add(INFO, "device:type", "emulator (ro.kernel.qemu=1)", + "final verification belongs on the real ABI the user will run") + + # ---- root ------------------------------------------------------------- + idout = su("id") + has_root = "uid=0" in idout + lvl = OK if has_root else (BLOCKER if a.expect_root else WARN) + r.add(lvl, "root", "available" if has_root else "NOT available (%s)" % idout.strip()[:60], + "" if has_root else "anything beyond static analysis needs root: data dirs, Frida, file locks") + + # ---- clock (silently breaks TLS work) -------------------------------- + dev_epoch = sh("date +%s").strip() + if dev_epoch.isdigit(): + drift = abs(int(dev_epoch) - int(__import__("time").time())) + r.add(OK if drift < 120 else WARN, "clock", + "device/host drift %ds" % drift, + "" if drift < 120 else "an expired/wrong certificate can be nothing but this clock; fix before TLS triage") + + # ---- leftover state that fakes a failure ----------------------------- + proxy = sh("settings get global http_proxy").strip() + if proxy and proxy not in ("null", ":0"): + r.add(WARN, "state:proxy", "device-wide http_proxy=%s" % proxy, + "a leftover proxy makes every request fail; clear with: settings put global http_proxy :0") + if a.cleanup: + sh("settings put global http_proxy :0") + r.add(INFO, "state:proxy", "cleared (--cleanup)", "") + else: + r.add(OK, "state:proxy", "not set", "") + + rc_f, fwd = run([ADB, "-s", serial, "forward", "--list"]) + if fwd.strip(): + r.add(INFO, "state:forwards", fwd.strip().replace("\n", " | ")[:200], + "stale forwards point frida at the wrong place; 'adb forward --remove-all' resets them") + if a.cleanup: + run([ADB, "-s", serial, "forward", "--remove-all"]) + r.add(INFO, "state:forwards", "removed (--cleanup)", "") + + # ---- frida server reachability --------------------------------------- + srv = su("pgrep -f frida-server || pgrep -f kwork") + if srv.strip().isdigit() or srv.strip().split("\n")[0].strip().isdigit(): + r.add(OK, "frida:server", "device server process alive (pid %s)" % srv.strip().split("\n")[0].strip(), "") + else: + r.add(INFO, "frida:server", "no device server process found", + "only needed for dynamic work; see references/dynamic-frida.md (name it out of obvious paths)") + + # ---- target package --------------------------------------------------- + if a.pkg: + pm = sh("pm list packages | grep -F %s" % a.pkg) + installed = ("package:" + a.pkg) in pm or a.pkg in pm + r.add(OK if installed else INFO, "target", "installed" if installed else "not installed", "") + if installed: + path = sh("pm path %s" % a.pkg).strip() + uid = su("dumpsys package %s | grep -m1 userId=" % a.pkg).strip() + r.add(INFO, "target:apk", path[:160] or "?", "") + r.add(INFO, "target:uid", uid or "?", "re-check after every reinstall: uid increments") + # the native library dir the package manager will use for this device + nat = sh("dumpsys package %s | grep -m3 primaryCpuAbi" % a.pkg).strip() + if nat: + r.add(INFO, "target:abi", nat.replace("\n", " | ")[:160], + "this is the ABI the package manager chose for THIS device") + maps = su("cat /proc/$(pidof %s | awk '{print $1}')/maps 2>/dev/null | head -1" % a.pkg) + if maps.strip(): + r.add(INFO, "target:running", "process is running", "prefer a clean cold start before measuring anything") + else: + r.add(OK, "target:running", "not running", "") + + # ---- free space ------------------------------------------------------- + # Report by device node, not by the mount-point column: a block device can be + # mounted at more than one path (an ARM translation shim does exactly this on + # some emulators), and `df` will then label /data with an unrelated path. + space = "" + for line in sh("df -h /data 2>/dev/null").splitlines()[1:]: + cols = line.split() + if len(cols) >= 5: + space = "%s: %s available of %s (%s used)" % (cols[0], cols[3], cols[1], cols[4]) + break + if space: + r.add(INFO, "device:space", space, "") + + return finish(r, a.as_json) + + +def finish(r: Report, as_json: bool) -> int: + if as_json: + print(json.dumps({"worst": r.worst(), + "rows": [dict(level=l, area=ar, detail=d, fix=f) for l, ar, d, f in r.rows]}, + indent=2, ensure_ascii=False)) + return 1 if r.worst() == BLOCKER else 0 + + width = max(len(x[1]) for x in r.rows) if r.rows else 10 + for level, area, detail, fix in r.rows: + print("%-8s %-*s %s" % ("[" + level + "]", width, area, detail)) + if fix: + print(" " * 9 + "%-*s -> %s" % (width, "", fix)) + print("\nworst: %s" % r.worst()) + if r.worst() == BLOCKER: + print("Fix every BLOCKER before running another experiment. Do not attribute a failure to your patch yet.") + return 1 + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/snap.py b/scripts/snap.py new file mode 100644 index 0000000..202d80e --- /dev/null +++ b/scripts/snap.py @@ -0,0 +1,203 @@ +#!/usr/bin/env python3 +"""Capture what is actually on screen -- and say whether the control tree is usable. + +Why this exists +--------------- +Driving a UI blind (tap a coordinate, wait, tap again) is the single most +expensive habit in device work. A look at the screen resolves in one step what +coordinate guessing cannot resolve in five: the layout moved, a different dialog +is up, a countdown is frozen, a button is disabled, text says why. + +Two independent kinds of evidence, and they are not interchangeable: + + * the **image** -- always available, always truthful about what is rendered. + It is the only evidence for UI drawn by a runtime or a web view, where no + real controls exist. + * the **control tree** (`uiautomator dump`) -- precise, diffable, gives exact + bounds and text. Frequently EMPTY for canvas/webview/cross-platform UI. + +This script grabs both, reports the size and whether the tree had content, so +you know which one is actually informative before you rely on it. It never +touches app state. + +Usage +----- + # one look + python snap.py --out shots --tag before + + # watch a wait: 6 samples, 3 s apart (bounded: hard cap of 20 samples) + python snap.py --out shots --tag waiting --count 6 --interval 3 + + # tree only, no images + python snap.py --out shots --tag tree --tree-only + + # when nothing is on screen, capture and it will say so rather than guess + python snap.py --out shots --tag check --serial + +Reading the output +------------------ + * `bytes=0` after both capture paths -> the device refused; see the note in the + output rather than concluding the screen is blank. + * `tree=EMPTY` -> do not plan around accessibility bounds. Use the image and + reason from pixels, or drive the app through a non-UI path. + * Identical hashes across samples -> nothing is changing. Stop waiting and go + find out why (this is the stall detector). +""" +from __future__ import annotations + +import argparse +import hashlib +import os +import re +import subprocess +import sys +import time + +MAX_SAMPLES = 20 # hard cap so a typo cannot produce an unbounded loop + + +def run(cmd: list[str], timeout: int) -> tuple[int, str, str]: + try: + p = subprocess.run(cmd, capture_output=True, text=True, errors="replace", timeout=timeout) + return p.returncode, p.stdout or "", p.stderr or "" + except subprocess.TimeoutExpired: + return 124, "", "TIMEOUT after %ss: %s" % (timeout, " ".join(cmd[:4])) + except FileNotFoundError: + return 127, "", "not found: %s" % cmd[0] + + +class Dev: + def __init__(self, adb: str, serial: str | None): + self.adb = adb + self.base = [adb] + (["-s", serial] if serial else []) + self.serial = serial + + def sh(self, cmd: str, timeout: int = 60) -> str: + return run(self.base + ["shell", cmd], timeout)[1] + + def su(self, cmd: str, timeout: int = 90) -> str: + return run(self.base + ["shell", 'su -c "%s"' % cmd], timeout)[1] + + def pull(self, remote: str, local: str, timeout: int = 120) -> bool: + rc, _, err = run(self.base + ["pull", remote, local], timeout) + return rc == 0 and os.path.exists(local) and os.path.getsize(local) > 0 + + +def tree_stats(dev: Dev, workdir: str) -> tuple[str, int, int]: + """Return (status, node_count, text_count). status in {OK, EMPTY, FAILED}.""" + xml = "/data/local/tmp/_snap_ui.xml" + dev.sh("rm -f " + xml) + out = dev.su("uiautomator dump %s" % xml, timeout=90) + if "dumped to" not in out and "UI hierchary" not in out and os.sep not in out: + # some builds print nothing useful; fall through to the file check anyway + pass + local = os.path.join(workdir, "_snap_ui.xml") + if not dev.pull(xml, local, timeout=120): + return "FAILED", 0, 0 + try: + data = open(local, encoding="utf-8", errors="replace").read() + except Exception: + return "FAILED", 0, 0 + nodes = len(re.findall(r" tuple[int, str]: + remote = "/data/local/tmp/%s.png" % name + dev.su("rm -f " + remote) + # on-device write then pull: the streaming form returns 0 bytes on some ROMs + dev.su("screencap -p %s" % remote, timeout=90) + local = os.path.join(workdir, name + ".png") + if dev.pull(remote, local, timeout=120): + n = os.path.getsize(local) + if n > 0: + return n, hashlib.sha256(open(local, "rb").read()).hexdigest()[:12] + # fallback: stream to host + rc, out, err = run(dev.base + ["exec-out", "screencap -p"], timeout=120) + if rc == 0: + data = out.encode("latin1", "ignore") if isinstance(out, str) else b"" + if data: + open(local, "wb").write(data) + return len(data), hashlib.sha256(data).hexdigest()[:12] + return 0, "" + + +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument("--adb", default=os.environ.get("ADB", "adb")) + ap.add_argument("--serial", default=None, help="device serial; required when several are attached") + ap.add_argument("--out", default="snapshots", help="output directory (created if missing)") + ap.add_argument("--tag", default="snap", help="label used in filenames") + ap.add_argument("--count", type=int, default=1, help="samples to take (capped at %d)" % MAX_SAMPLES) + ap.add_argument("--interval", type=float, default=2.0, help="seconds between samples") + ap.add_argument("--tree-only", action="store_true", help="skip images; only read the control tree") + ap.add_argument("--image-only", action="store_true", help="skip the control tree") + a = ap.parse_args() + + if a.count > MAX_SAMPLES: + print("count capped at %d (asked for %d)" % (MAX_SAMPLES, a.count), file=sys.stderr) + a.count = MAX_SAMPLES + + workdir = os.path.abspath(a.out) + os.makedirs(workdir, exist_ok=True) + dev = Dev(a.adb, a.serial) + + # cheap reachability gate with a timeout, so we fail fast and informatively + rc, out, err = run(dev.base + ["shell", "echo __ok__"], timeout=30) + if "__ok__" not in out: + print("device not answering: %s%s" % (out.strip()[:120], (" / " + err.strip()[:120]) if err else ""), + file=sys.stderr) + print("check `adb devices`; if the emulator process is alive but nothing is listening on the adb port,", + file=sys.stderr) + print("restart the instance through the vendor console (see references/environment.md).", file=sys.stderr) + return 2 + + tree_status = "skipped" + nodes = texts = 0 + if not a.image_only: + tree_status, nodes, texts = tree_stats(dev, workdir) + print("control tree: %s (nodes=%d, non-empty text=%d)" % (tree_status, nodes, texts)) + if tree_status in ("EMPTY", "FAILED"): + print(" -> the accessibility tree is not usable here. Do NOT plan around bounds from it.") + print(" The image is the primary evidence for this UI. Cross-platform runtimes, web views") + print(" and canvas-drawn UI frequently expose no real controls at all.") + elif tree_status == "SPARSE": + print(" -> tree exists but carries almost no text; treat it as unreliable and read the image.") + + prev_hash = None + same_run = 0 + for i in range(a.count): + tag = a.tag if a.count == 1 else "%s_%02d" % (a.tag, i) + if a.tree_only: + print("%-14s tree-only sample" % tag) + else: + size, digest = capture_image(dev, workdir, tag) + if size == 0: + print("%-14s bytes=0 -> device refused the capture. Not evidence that the screen is blank." % tag) + else: + note = "" + if digest == prev_hash: + same_run += 1 + note = " (identical to previous -> nothing changed)" + else: + same_run = 0 + prev_hash = digest + print("%-14s bytes=%-8d sha=%s%s" % (tag, size, digest, note)) + print("%-14s %s" % ("", os.path.join(workdir, tag + ".png"))) + if same_run >= 2: + print(" -> %d identical samples in a row. This is a stall, not a slow operation." % (same_run + 1)) + print(" Stop waiting: inspect the image, or check the app is still alive and in the foreground.") + if i < a.count - 1: + time.sleep(max(0.0, a.interval)) + + print("\nNow LOOK at the image(s) before deciding the next action.") + print("Driving blind and waiting is how rounds get wasted; the screen usually states the reason.") + return 0 + + +if __name__ == "__main__": + sys.exit(main())