Crash reporting, and reading a stack trace from the field

Kavita opens Mandi Bhav in the yard at Lasalgaon, taps a price, and the app closes. She tries once more, it closes again, and she goes back to asking the commission agent. You will never see her phone, never read its Logcat, and she will not write to you. Everything in lessons 1 to 4 assumed the bug happens where you can watch it. This lesson is about the ones that do not: what Android and Google Play record for you without any extra code, what a crash-reporting service adds and costs, how to read a stack trace that R8 has scrambled, and what Mandi Bhav now records about its own crashes and ANRs, with no account anywhere.

Reading a stack trace, properly

Lesson 1's crash, from a debug build:

FATAL EXCEPTION: main
Process: com.riztechacademy.mandibhav.m13, PID: 6076
java.util.IllegalFormatConversionException: d != java.lang.Double
	at java.util.Formatter$FormatSpecifier.failConversion(Formatter.java:4608)
	...
	at java.lang.String.format(String.java:4078)
	at com.riztechacademy.mandibhav.ui.PriceWeekKt.percentChange(PriceWeek.kt:40)
	at com.riztechacademy.mandibhav.ui.PriceDetailScreenKt.PriceDetailContent$lambda$0$0$1(PriceDetailScreen.kt:230)
	at androidx.compose.runtime.internal.ComposableLambdaImpl.invoke(ComposableLambda.kt:131)
	...

The method for any trace is the same: read the exception line, then find the first frame in your own package, then read one or two frames below it to see who called it. 124 frames, two of yours, and those two are the ones to read.

What you get from the field without writing anything

Android vitals, in the Google Play Console. When a user has agreed to share usage and diagnostics data, Android itself reports every crash and ANR of every app installed from Play, with the stack trace, the device model and the Android version. No library, no code, no extra permission. It counts the crashes that happen before any library in your app has started, and, on phones older than Android 12, ANRs that a library inside the app cannot see. Play also uses it to judge you:

Cross those, measured over 28 days, and Play may show your app to fewer people and warn on its store listing. The rates are per daily active user: one crash in a day is a crashed user for that day, however many times they opened the app.

What it costs: a Play Console account, which module 14 sets up, and the app installed from Play. Before that, during development and with testers who install your APK directly, vitals sees nothing. You also see only users who opted in, and only once there are enough of them to report anonymously.

Where module 9's promise lands. Module 9 said ANRs reach Android vitals and that module 14 reads them. This lesson covers what an ANR report contains and how to read it (below), because you can produce one and read it today on the emulator. Module 14 shows where those reports appear in the Play Console once Mandi Bhav is published there.

A crash-reporting service

Firebase Crashlytics is the common choice on Android; Sentry and others work the same way. You add a Gradle plugin and a library, create a project in their console, and the library installs its own handler that records each crash and sends it the next time the app starts.

What it adds over vitals: reports within minutes rather than the next day; testers' and sideloaded installs as well as Play's; non-fatal reports, where you catch an exception, carry on, and still want to know; custom keys and log lines ("market = Lasalgaon, last refresh failed with Timeout") attached to the next crash; and alerts.

What it costs: an account and a project of your own, which you must create (this course does not, and Mandi Bhav's code does not include one, so it builds for everyone without anybody's keys); a library in the release APK that starts with the app; a configuration file in the project; and data leaving the phone, which you must declare in the Play Console's data safety form (module 14) and mention in your privacy policy. Follow the Crashlytics documentation linked below with your own Firebase project if you want it; the reading skills in the rest of this lesson are the same whichever tool shows you the trace.

What not to do: write your own Thread.setDefaultUncaughtExceptionHandler to "send crashes somewhere". It replaces Android's own handler, which writes the trace to Logcat and shows the "keeps stopping" dialog, unless you are careful to call the previous one; it fights with every library that installs one; it runs when the process is already broken; and it cannot see ANRs or native crashes at all. Android already records how every process ended, as the end of this lesson shows.

The scrambled trace

Mandi Bhav's release build is not shrunk yet; module 14 turns on R8, which removes unused code and renames everything left to short names. To see what the field will look like then, a scratch copy built release with isMinifyEnabled = true (signed with the debug key so it installs). The APK went from 10.9 MB to 2.8 MB, it ran, prices and all, and lesson 1's crash came back like this:

java.util.IllegalFormatConversionException: d != java.lang.Double
	at java.util.Formatter$FormatSpecifier.failConversion(Formatter.java:4608)
	...
	at java.lang.String.format(String.java:4078)
	at nm1.g(r8-map-id-f10adc2cb4e17766be78e259e8e27955f9efe2e3df269e1c0bf5fd36958c1fa3:587)
	at xv.f(r8-map-id-f10adc2cb4e17766be78e259e8e27955f9efe2e3df269e1c0bf5fd36958c1fa3:42)
	at xv.g(r8-map-id-f10adc2cb4e17766be78e259e8e27955f9efe2e3df269e1c0bf5fd36958c1fa3:9)
	at za1.p(r8-map-id-f10adc2cb4e17766be78e259e8e27955f9efe2e3df269e1c0bf5fd36958c1fa3:62)
	...

Java's own classes keep their names, because they are on the phone, not in your APK. Everything of yours and of your libraries is now nm1.g, xv.f, and where a file name was, R8 has written the id of the build's mapping file. Every minified build writes one, app/build/outputs/mapping/release/mapping.txt (44 MB for this scratch build), which says what every short name was. Its header names the same id:

# compiler: R8
# compiler_version: 9.5.29
# min_api: 26
# pg_map_id: f10adc2cb4e17766be78e259e8e27955f9efe2e3df269e1c0bf5fd36958c1fa3

The SDK's retrace tool, in cmdline-tools/latest/bin, turns one back into the other:

retrace app/build/outputs/mapping/release/mapping.txt crash.txt

java.util.IllegalFormatConversionException: d != java.lang.Double
	...
	at java.lang.String.format(String.java:4078)
	at com.riztechacademy.mandibhav.ui.PriceWeekKt.percentChange(PriceWeek.kt:40)
	at com.riztechacademy.mandibhav.ui.PriceDetailScreenKt.PriceDetailContent$lambda$0$0$1(PriceDetailScreen.kt:230)
	at androidx.compose.runtime.internal.ComposableLambdaImpl.invoke(ComposableLambda.kt:131)
	...

Exactly the debug trace's lines, file names and line numbers, even though R8 had folded percentChange into its caller: the single obfuscated frame nm1.g:587 came back as two. Play Console and Crashlytics do the same for you when you give them the mapping file; retrace is how you do it yourself, for a trace a tester pasted into a message.

The trap: the wrong mapping file

The mapping belongs to one build. I changed a few lines above percentChange in the scratch copy, rebuilt, and retraced the old crash with the new mapping:

	at com.riztechacademy.mandibhav.ui.PriceWeekKt.percentChange(PriceWeek.kt:46)

No error, no warning: a plausible answer that points six lines away from the bug, in a file that in the crashed build had nothing there. With a bigger change between builds, the class names stop matching too and you get confident nonsense. So:

• Keep the mapping file of every build you release, with the version code, for as long as anyone may run that version. build/ is deleted by clean; archive it, and module 14 uploads it to Play with each release.
• Check the id before you retrace. The r8-map-id-… in the trace must equal the pg_map_id in the mapping. Here, the crash said f10adc2c…; the new mapping said 389dfe33….

ANRs: the trace you cannot get from Logcat

An ANR (module 9) does not kill the app with an exception, so there is no FATAL EXCEPTION in anybody's Logcat. What Android does is dump every thread's stack at the moment it decided the app was not responding. The scratch copy blocked the main thread for ten seconds in the Share button, with a tap one second in:

I WindowManager: ANR in Window{37c9428 u0 com.riztechacademy.mandibhav.m13/com.riztechacademy.mandibhav.MainActivity}. Reason:Input dispatching timed out (… is not responding. Waited 5001ms for MotionEvent).
E ActivityManager: ANR in com.riztechacademy.mandibhav.m13 (com.riztechacademy.mandibhav.m13/com.riztechacademy.mandibhav.MainActivity)

No dialog appeared on this emulator, and after ten seconds the app carried on as if nothing had happened. On Kavita's phone she would have seen "isn't responding" and probably closed it. Either way, the dump exists.

Asking Android how the last process ended

Since Android 11 (API 30), an app can ask the system how its previous processes ended, with ActivityManager.getHistoricalProcessExitReasons. Each answer is an ApplicationExitInfo: a reason, a time, a description, and for ANRs and native crashes the dump itself. On the next launch, the scratch copy printed what it found. The ANR's dump began:

Subject: Input dispatching timed out (… is not responding. Waited 5001ms for MotionEvent).
…
"main" prio=5 tid=1 Sleeping
  | group="main" sCount=1 ucsCount=0 flags=1 obj=0x73c8cf68 self=0xb40000701aa4f010
  | sysTid=7011 nice=-10 cgrp=top-app sched=0/0 handle=0x71ed445090
  at java.lang.Thread.sleep(Native method)
  - sleeping on <0x0fa629e9> (a java.lang.Object)
  at java.lang.Thread.sleep(Thread.java:451)
  - locked <0x0fa629e9> (a java.lang.Object)
  at java.lang.Thread.sleep(Thread.java:356)
  at mc2.l(r8-map-id-e2ec9451664dfc2049b79333c4fbafeef5b9a0a0ff6dd5e88c6e1f27bf06d8ec:19)
  at t6.a(r8-map-id-e2ec9451664dfc2049b79333c4fbafeef5b9a0a0ff6dd5e88c6e1f27bf06d8ec:468)

53,547 characters in all, every thread in the app. The one to read is "main", and its state is the first clue: Sleeping (waiting on purpose), Blocked (waiting for a lock another thread holds: look for that thread), Native (inside system code, often I/O) or Runnable (busy computing). Retraced with that build's mapping:

  at java.lang.Thread.sleep(Thread.java:356)
  at com.riztechacademy.mandibhav.ui.PriceDetailScreenKt.LatestPriceRoute$lambda$2$0(PriceDetailScreen.kt:107)
  at com.riztechacademy.mandibhav.ui.PriceDetailScreenKt.PriceDetailScreen$lambda$1$2$0$0(PriceDetailScreen.kt:135)
  at androidx.compose.foundation.AbstractClickableNode.performClick(AbstractClickableNode.java:1701)

The Share button's click handler, PriceDetailScreen.kt:107, sleeping. That is exactly the report Android vitals shows for an ANR in the field, and you read it the same way.

The other records were as instructive. For lesson 1's crash, the record said only this:

reason=4 status=0 importance=100 desc=crash

REASON_CRASH, and no stack: Android keeps Java crash traces in Logcat and in vitals, not here. And the ANR'd process, which recovered and was later stopped from adb, was recorded as reason=10 … desc=[FORCE STOP], with the ANR's dump attached to that record. The reason says how a process ended; the dump says what went wrong on the way. Code that looks only for REASON_ANR would have missed this one.

What Mandi Bhav now does with it

diagnostics/ProcessExits.kt turns the framework's record into plain values and plain text, so the logic is tested on the JVM:

data class ProcessExit(
    val reason: Int,
    val timestamp: Instant,
    val description: String?,
    val trace: String?,
)

/** The reasons a developer should look into; the rest are normal ends of a process. */
@RequiresApi(30)
fun isProblem(exit: ProcessExit): Boolean =
    exit.trace != null || exit.reason in setOf(
        ApplicationExitInfo.REASON_ANR,
        ApplicationExitInfo.REASON_CRASH,
        ApplicationExitInfo.REASON_CRASH_NATIVE,
        ApplicationExitInfo.REASON_EXCESSIVE_RESOURCE_USAGE,
        ApplicationExitInfo.REASON_INITIALIZATION_FAILURE,
    )

exit.trace != null is the line the emulator taught. mainThreadOf(trace) keeps only the main thread's lines, and describe writes a summary; ProcessExitsTest covers a crash, an ANR, the ANR attached to a force-stop, the normal ends that are not worth a word, and a dump with no main thread. MandiBhavApplication calls it on every start of a debug build on Android 11 or later, off the main thread, and logs under the tag LastExit:

if (BuildConfig.DEBUG && Build.VERSION.SDK_INT >= Build.VERSION_CODES.R) logLastExit()

/** Off the main thread: reading an ANR's dump means reading a file. */
@RequiresApi(Build.VERSION_CODES.R)
private fun logLastExit() {
    applicationScope.launch {
        val last = readProcessExits(this@MandiBhavApplication, max = 1).firstOrNull() ?: return@launch
        if (isProblem(last)) describe(last, ZoneId.systemDefault()).forEach { Log.w("LastExit", it) }
    }
}

Crashing the debug app from adb and opening it again:

adb shell am crash com.riztechacademy.mandibhav

W LastExit: Last process ended: crash at 9 Nov 03:05:06 (crash)

Why debug only: it is for you and your testers, the crash or ANR that happened while nobody was watching Logcat, told to you the next time you open the app. In release it would be a log line nobody reads, and anything more, such as sending it somewhere, is a crash-reporting service with all the costs above. Mandi Bhav sends nothing anywhere.

Check your work

Where is the real cause in a trace with several Caused by sections? Usually in the last one; the outer exceptions wrapped it.

Why might Android vitals show a crash that Crashlytics missed? Vitals comes from Android itself, so it includes crashes before any library initialised and, before Android 12, ANRs a library could not see; it also counts per daily user rather than per session.

What did the minified crash's first Mandi Bhav frame look like, and what did retrace make of it? nm1.g(r8-map-id-f10adc2c…:587), which became two frames: PriceWeekKt.percentChange(PriceWeek.kt:40) and its caller at PriceDetailScreen.kt:230.

What happened when the trace was retraced with the next build's mapping? It gave PriceWeek.kt:46, six lines off, with no warning; the trace's map id and the mapping's pg_map_id did not match.

What did ApplicationExitInfo record for lesson 1's crash, and for the ANR? For the crash, REASON_CRASH with the description "crash" and no stack. For the ANR, a full thread dump, attached to the record of the process's later force-stop, whose main thread retraced to the Share button's handler at PriceDetailScreen.kt:107.

Why does isProblem check trace != null and not only the reason? Because an ANR the app survived is recorded with the reason the process eventually ended for, such as a force-stop, while its dump is attached.

Practice

1. Add ProcessExits.kt, its test and the LastExit log to Mandi Bhav. Crash the debug build with adb shell am crash, open it again, and read the LastExit line.
2. Read a trace from your own scratch app with a Caused by: make parseDeepLink throw IllegalStateException("bad link", e) from inside its catch, send a broken link, and find the original exception in the trace.
3. In a scratch copy, turn on isMinifyEnabled for release, sign it with the debug key, and reproduce lesson 1's crash. Save the trace from Logcat and the mapping file, and retrace it.
4. Break it on purpose: change any line above the crash, rebuild, and retrace the old trace with the new mapping. Compare the map ids.
5. Block the main thread in a scratch copy as above, tap while it is blocked, then open the app again and read what LastExit printed. Which thread state did the dump show, and why?
6. Harder: Crashlytics counts per session and vitals per daily user. Work out, for a farmer who opens Mandi Bhav four times a day and has one crash a day, what each would report, and decide which number you would put in front of your team, and why. Then read the Crashlytics getting-started guide and list, in order, everything you would have to add to Mandi Bhav to use it, with what each item costs.

Official documentation

• Crashes — Reading a stack trace, native crashes, and how vitals reports crashes.
• Android vitals — The core vitals and their thresholds, how the data is collected, and why it differs from SDK counts.
• retrace — The command-line tool, its options, and where mapping files come from.
• ApplicationExitInfo — Every exit reason, and which ones carry a trace.
• Get started with Firebase Crashlytics — Setting it up in your own Firebase project, if you choose to.

Next: module 14, Shipping to Google Play.

---

That is module 13. You can now:

• read Logcat beyond the basics, stop the app with line, conditional, logging and exception breakpoints, evaluate expressions in a paused app, and read the semantics and recomposition counts in the Layout Inspector;
• write unit tests with fakes rather than mocks, use runTest's virtual clock and the two test dispatchers, test flows and SavedStateHandle, and prove each test can fail by breaking the code it protects;
• test Compose screens through their semantics, on the JVM under Robolectric and on a device, replace a Hilt binding for an instrumented test, and know which checks do nothing under Robolectric;
• find jank with gfxinfo and a Perfetto trace, a leak with meminfo and LeakCanary, and a stuck wake lock with batterystats, and measure start-up with Macrobenchmark and a Baseline Profile, honestly;
• read a crash or an ANR from a phone you will never see, from Android vitals, a crash-reporting service, ApplicationExitInfo and retrace, and know what each costs.

Mandi Bhav now has 204 unit tests and 10 instrumented ones, a Baseline Profile, LeakCanary in debug builds, and a debug log of how its last process ended. Its release build is still unshrunk and unsigned. Module 14, Shipping to Google Play, turns on R8 and keeps the mapping files this lesson said to keep, signs the app with a key you must not lose, creates the Play Console listing, and takes Mandi Bhav through internal, closed and open testing, where Android vitals starts reporting the crashes and ANRs this lesson taught you to read.