diff --git a/.github/release-notes.md b/.github/release-notes.md new file mode 100644 index 0000000..68f4e34 --- /dev/null +++ b/.github/release-notes.md @@ -0,0 +1,23 @@ +Gearslip 0.3 goes wireless, takes voice commands, and works with the car's rotary knob. + +## New +- **Wireless Android Auto.** Pair the phone with the car once, and leave the cable at home. +- **Voice control, on the phone.** Say "navigate to a gas station", "play" a song or artist, or open an app by name. Gearslip turns speech into text itself, so it needs no Google account. "Navigate to" picks the nearest place and stops at the map app's start screen. +- **The rotary knob.** The car's knob moves between buttons, so cars without a touchscreen work too. +- **Music over Bluetooth or USB.** Bluetooth plays as the phone normally would. USB has no delay but asks to share audio each drive. Gearslip offers to pair with the car the first time. +- **Directions on the car.** The car speaks each turn from your map app, and shows the directions on its own display where it can. +- **Updates in the app.** Gearslip finds new versions and installs them, with an optional beta channel. +- **Button styles** for the car screen, picked in the intro or the car's settings. + +## Fixed +- More cars play music over USB, including ones that refused Gearslip's audio before. +- Cars that give no screen size get a sharper picture, up to 1080p, and taps land where you press on cars that crop the edges. +- A new destination mid-route replaces the old route, and switching map apps ends the old app's route. +- Tidal's like and shuffle buttons show their real state. +- MapQuest finds your location. + +## Changed +- The app is half the size: 30 MB instead of 65. +- On GrapheneOS, Gearslip no longer needs exploit protection compatibility mode. +- The car warns you when the phone's location is off. +- Support Gearslip through Liberapay or Monero, in Settings. diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 0636864..42bd647 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -45,4 +45,6 @@ jobs: uses: softprops/action-gh-release@v2 with: files: ${{ steps.apk.outputs.path }} + # The hand-written notes go first; GitHub adds the full changelog link under them. + body_path: .github/release-notes.md generate_release_notes: true diff --git a/.gitignore b/.gitignore index a50757c..576794a 100644 --- a/.gitignore +++ b/.gitignore @@ -30,4 +30,11 @@ CLAUDE.md # Other archive/ .wrangler/ -stats-worker/ \ No newline at end of file +stats-worker/ + +# Certificates and keys never go in the repo; the app downloads or imports them on the phone. +*.p12 +*.pfx +*.pem +*.crt +*.key diff --git a/CARS.md b/CARS.md new file mode 100644 index 0000000..3a8c992 --- /dev/null +++ b/CARS.md @@ -0,0 +1,49 @@ +# Cars + +Drivers who share usage notes tell us which cars Gearslip works in. This list comes from +those notes, and from our own testing. A car shows up here once Gearslip has reached its +screen at least once. + +The year is the one the car reports. Some cars report the year of the head unit instead, +and some don't say. + +## Works + +| Car | Year | Head unit | +|---|---|---| +| Audi | | MIB2+ (Alpine) | +| Audi | | Harman | +| Chevrolet, with myChevrolet | 2019, 2020, 2021 | myChevrolet | +| Chevrolet, with MyLink | 2017 | MyLink | +| Dacia | | Media Nav (LG ULC 4.1) | +| Dacia | | LG ULC 4.5 | +| Ford | 2018 | SYNC 3.0 (Panasonic) | +| Ford Fiesta | 2019 | SYNC 3.2 (Panasonic) | +| Ford Fusion | 2018 | SYNC 3.1 (Panasonic) | +| Jeep Cherokee | 2019 | Uconnect 8.4 (Delphi) | +| MG | 2019 | Desay SV | +| Ram 1500 | 2018 | Uconnect 8.4 (Panasonic) | +| Škoda | 2018 | MIB Standard2 (TechniSat) | +| Subaru Outback | 2020 | Denso | +| Toyota RAV4 | 2020 | Pioneer T1 | +| Volkswagen | 2018 | MIB Standard2 (Delphi) | +| Volkswagen | 2024 | ICAS3 | +| Volvo XC90 | 2022 | SPA Plus (Mitsubishi Electric) | +| Wuling | 2021 | Desay SV | +| [LIVI](https://github.com/f-io/LIVI), the open-source head unit | | | + +## Not working yet + +| Car | Year | What happens | +|---|---|---| +| Honda | 2016 | The connection drops before Gearslip reaches the screen. | +| Honda | 2019 | The connection drops before Gearslip reaches the screen. | +| Mitsubishi Attrage | 2020 | The connection drops before Gearslip reaches the screen. | + +Some cars reject Gearslip's certificate before they say what they are, so they can't appear +here by name. Most of these are from around 2020 or later. + +## Add your car + +Turn on usage notes in Gearslip's setup or Settings, and drive once. If your car is missing +or listed wrongly, open an issue. diff --git a/README.md b/README.md index 1cce473..04cd9fe 100644 --- a/README.md +++ b/README.md @@ -15,16 +15,17 @@ Questions, ideas? ## Which cars work Most cars work. Starting around 2020, some cars added newer security standards that reject -Gearslip's certificate, and Gearslip can't run in those cars. +Gearslip's certificate, and Gearslip can't run in those cars. [CARS.md](CARS.md) lists the +cars drivers have confirmed. ## What Gearslip is Gearslip is an Android Auto client for the phone. It speaks the Android Auto protocol -directly to a car head unit over USB, with no Google software in the path. Gearslip acts -as a Car App Library host: an app that ships a car interface hands Gearslip a template, -and Gearslip renders that template natively on the car screen. Nothing is mirrored from -the phone. For apps without a car interface, Gearslip offers a separate screen-sharing -mode. +directly to a car head unit, over USB or wirelessly, with no Google software in the path. +Gearslip acts as a Car App Library host: an app that ships a car interface hands Gearslip +a template, and Gearslip renders that template natively on the car screen. Nothing is +mirrored from the phone. For apps without a car interface, Gearslip offers a separate +screen-sharing mode. | | | |---|---| @@ -38,7 +39,7 @@ mode. ## Features - **Phone calls.** Dial from a keypad, or search your contacts, then confirm before the - call goes out. + call goes out. When a call comes in, answer or decline it on the car screen. - **Navigation.** Gearslip hosts real Car App Library map apps on the car screen. Organic Maps, OsmAnd, and Flitsmeister work today. Vela runs, with rough edges. @@ -49,6 +50,9 @@ mode. doesn't ship a car interface of its own. - **Weather and the day ahead.** A dashboard screen shows the week's calendar next to the current and coming weather. +- **Voice.** Tap the mic, or press the talk button on the wheel, and say "navigate to the + nearest gas station", "play Daft Punk", or "call Mom". Say "help" for the full list. + Speech turns into text on the phone, and nothing you say leaves it. - **Messages.** New messages appear as a small card on the car screen. Gearslip reads each one aloud, and you reply by voice or with a quick reply. Music pauses while Gearslip speaks or listens. Speech runs on the phone, so it needs no Google account. @@ -61,8 +65,9 @@ mode. ## Status -- **The connection is wired only, for now.** Plug the phone into the head unit's USB port; - Gearslip does not yet offer a wireless option. +- **Wired or wireless.** Plug the phone into the car's USB port, or, in a car with wireless + Android Auto, tap Connect wirelessly and pick the car. Pair the phone with the car over + Bluetooth first. - **Setup walks you through it.** First launch asks for the Android Auto certificate, then one permission at a time, each with a plain reason and an Allow or Skip. A help screen inside the car answers anything setup didn't cover. @@ -76,14 +81,19 @@ mode. Gearslip's own player. - **Calls use the car's Bluetooth,** as they do in Android Auto. Pair the phone with the car to hear calls through its speakers. -- **Audio reaches the car by capturing playback, not by mirroring the screen.** Android - shows a screen-share consent dialog for this, even though Gearslip never reads the - screen; it is the only way an ordinary app can capture another app's audio. The dialog - appears once per drive. +- **Music reaches the car over Bluetooth or USB.** Setup asks which. Bluetooth is the + default: Android never asks for permission, and your map app's voice reaches the car + too. USB sounds a little clearer, with no delay, but Android shows a screen-share + consent dialog once per drive. Gearslip never reads the screen; capturing playback is + the only way an ordinary app can pass another app's audio along. - **Gearslip needs no adb, root, or Shizuku.** Every feature relies on an ordinary - Android permission or an on-device consent dialog. The setup guide offers two optional - adb commands: one removes the audio dialog for good, and the other keeps notifications - visible while audio plays. + Android permission or an on-device consent dialog. For USB audio, setup offers two + optional adb commands: one removes the audio dialog for good, and the other keeps + notifications visible while audio plays. +- **Updates come from GitHub.** Once a day, Gearslip checks this repository's latest + release. When a new version is out, it downloads the APK, checks it against GitHub's + SHA-256, and installs it through Android's own installer. Turn off automatic checks in + Settings. ## Usage notes @@ -92,10 +102,13 @@ turn them on, in setup or in Settings. - **When they go out.** When you open Gearslip, at most once a day, and after each drive. Nothing runs in the background. -- **What they carry.** The Gearslip version, plus whatever you tick: the Android version - and whether the phone runs GrapheneOS, the phone model, whether a car connected that - day, and each car's head unit, model, year, screen size, and connection result. - Settings shows the exact note. +- **What they carry.** The Gearslip version and which boxes you ticked, plus whatever + those boxes cover: the Android version and whether the phone runs GrapheneOS, the phone + model, whether a car connected that day, and each car's head unit, model, year, screen + size, and connection result, with whether that drive used the cable or wireless and + whether music went over USB or Bluetooth. If you tick it, how each drive went: how far + the connection got, how long it lasted, and how it ended. Settings shows the exact note. Untick a phone detail, and + the server drops the copy it kept. - **What they leave out.** Your name, accounts, location, contacts, messages, and anything you type or play. - **How phones are counted.** A random number counts each phone once. The server keeps diff --git a/docs/icon/archive/gearslip-icon-rounded.png b/docs/icon/archive/gearslip-icon-rounded.png deleted file mode 100644 index ad868c9..0000000 Binary files a/docs/icon/archive/gearslip-icon-rounded.png and /dev/null differ diff --git a/docs/icon/archive/gearslip-icon-rounded.svg b/docs/icon/archive/gearslip-icon-rounded.svg deleted file mode 100644 index 95c7437..0000000 --- a/docs/icon/archive/gearslip-icon-rounded.svg +++ /dev/null @@ -1,14 +0,0 @@ - - - - - - - - - - - - - - \ No newline at end of file diff --git a/docs/icon/on-device-verification.png b/docs/icon/on-device-verification.png deleted file mode 100644 index fcc2b87..0000000 Binary files a/docs/icon/on-device-verification.png and /dev/null differ diff --git a/gearslip/build.gradle.kts b/gearslip/build.gradle.kts index 0f8635e..6bf18a7 100644 --- a/gearslip/build.gradle.kts +++ b/gearslip/build.gradle.kts @@ -24,6 +24,12 @@ val keystoreProperties = Properties().apply { ).firstOrNull { it.exists() }?.inputStream()?.use { load(it) } } +// Keeps build output out of the module folder: /build/gearslip/... instead of +// /gearslip/build/..., so everything this project produces lands under one +// root-level build/ directory. Set before android {}, which works out the path to R8's default +// rules from it. +layout.buildDirectory.set(rootProject.layout.buildDirectory.dir("gearslip")) + android { namespace = "app.seb3thehacker.gearslip" compileSdk = 37 @@ -32,8 +38,9 @@ android { applicationId = "app.seb3thehacker.gearslip" minSdk = 31 targetSdk = 37 - versionCode = 8 - versionName = "0.2.00-alpha" + versionCode = 9 + // -PgearslipVersionName=0.1.99-test makes a build that thinks it's older, to try the updater. + versionName = project.findProperty("gearslipVersionName")?.toString() ?: "0.3.00-alpha" manifestPlaceholders["appLabel"] = "Gearslip" // Where opted-in phones send their daily usage note: the stats-worker/ Worker's address, // set as gearslipStatsUrl in gradle.properties. Left empty, nothing is ever sent. @@ -56,14 +63,23 @@ android { buildTypes { release { signingConfig = signingConfigs.findByName("release") + isMinifyEnabled = true + isShrinkResources = true + proguardFiles(getDefaultProguardFile("proguard-android-optimize.txt"), "proguard-rules.pro") } - // Its own package, so a dev build installs beside the release one instead of over it. + // Its own package, so a debug build installs beside the release one instead of over it. // Signed with the release key when it's here: both builds declare the car template // permission, and Android refuses a second app declaring it under a different key. debug { applicationIdSuffix = ".dev" signingConfigs.findByName("release")?.let { signingConfig = it } - manifestPlaceholders["appLabel"] = "Gearslip Dev" + manifestPlaceholders["appLabel"] = "Gearslip Debug" + // -Pr8 shrinks a debug build like a release one, to test R8 in the debug install. + if (project.hasProperty("r8")) { + isMinifyEnabled = true + isShrinkResources = true + proguardFiles(getDefaultProguardFile("proguard-android-optimize.txt"), "proguard-rules.pro") + } } } @@ -71,32 +87,16 @@ android { buildConfig = true } - // Keeps build output out of the module folder: /build/gearslip/... instead of - // /gearslip/build/..., so everything this project produces lands under one - // root-level build/ directory. - layout.buildDirectory.set(rootProject.layout.buildDirectory.dir("gearslip")) - compileOptions { sourceCompatibility = JavaVersion.VERSION_21 targetCompatibility = JavaVersion.VERSION_21 } defaultConfig { - // Speech-to-text ships for 64-bit ARM only: every phone Gearslip runs on, and one copy - // of the native library instead of four. + // Speech-to-text (sherpa-onnx, in jniLibs) ships for 64-bit ARM only: every phone + // Gearslip runs on, and one copy of the native library instead of four. ndk { abiFilters += "arm64-v8a" } } - ndkVersion = "27.1.12297006" - externalNativeBuild { - cmake { - path = file("src/main/cpp/CMakeLists.txt") - version = "3.22.1" - } - } - androidResources { - // Model weights barely compress, and stored plainly they stream straight out of the APK. - noCompress += "bin" - } testOptions { // GearslipLog calls android.util.Log, which is a stub on the local JVM. @@ -129,6 +129,7 @@ dependencies { implementation(platform(libs.androidx.compose.bom)) implementation(libs.androidx.compose.ui) implementation(libs.androidx.compose.material3) + implementation(libs.commons.compress) implementation(libs.androidx.compose.material.icons.core) implementation(libs.androidx.activity.compose) implementation(libs.androidx.lifecycle.runtime.compose) diff --git a/gearslip/proguard-rules.pro b/gearslip/proguard-rules.pro new file mode 100644 index 0000000..b833500 --- /dev/null +++ b/gearslip/proguard-rules.pro @@ -0,0 +1,20 @@ +# R8 shrinks the release build; these keep what it can't see being used. + +# Names stay readable, so stack traces in shared logs need no mapping file. Shrinking does +# the real work on size. +-dontobfuscate + +# Gearslip hosts other apps' car screens: their templates arrive as Bundles, and the car +# library rebuilds them by class name and reflection. Any model class an app might send has to +# survive, used by Gearslip's code or not. +-keep class androidx.car.app.** { *; } + +# Speech-to-text: the native library calls back into these classes by name. +-keep class com.k2fsa.sherpa.onnx.** { *; } + +# commons-compress refers to optional codecs Gearslip doesn't ship. +-dontwarn org.tukaani.xz.** +-dontwarn com.github.luben.zstd.** +-dontwarn org.brotli.dec.** +-dontwarn org.objectweb.asm.** +-dontwarn org.osgi.** diff --git a/gearslip/src/main/AndroidManifest.xml b/gearslip/src/main/AndroidManifest.xml index 4dc1ede..2aa3ab1 100644 --- a/gearslip/src/main/AndroidManifest.xml +++ b/gearslip/src/main/AndroidManifest.xml @@ -4,6 +4,7 @@ + + + + + + + + + + + @@ -74,8 +91,10 @@ LAUNCHER query below to find, so the setup guide's Android Auto check needs this to see whether the package is installed at all. --> - + + diff --git a/gearslip/src/main/cpp/CMakeLists.txt b/gearslip/src/main/cpp/CMakeLists.txt deleted file mode 100644 index f4ab8b5..0000000 --- a/gearslip/src/main/cpp/CMakeLists.txt +++ /dev/null @@ -1,36 +0,0 @@ -# Speech-to-text for voice replies and the assistant: whisper.cpp, pinned to one commit and -# fetched at build time, so nothing outside this folder (cloned-repos included) is needed. -cmake_minimum_required(VERSION 3.22) -project(gearslip_speech CXX C) - -set(CMAKE_CXX_STANDARD 17) - -# Optimised in debug builds too. Unoptimised, Whisper runs several times slower, and a dev build -# would make voice look far worse than what ships. -foreach(lang C CXX) - set(CMAKE_${lang}_FLAGS_DEBUG "-O3 -DNDEBUG") -endforeach() - -include(FetchContent) -FetchContent_Declare( - whisper - URL https://github.com/ggml-org/whisper.cpp/archive/6e4ab854f67f743900934a703d5603419384c961.tar.gz - URL_HASH SHA256=55b585aec5e1bc03e2a5cea61da97c6406fa833996cf7daeed704972273cc039 -) - -set(BUILD_SHARED_LIBS OFF CACHE BOOL "" FORCE) -set(WHISPER_BUILD_EXAMPLES OFF CACHE BOOL "" FORCE) -set(WHISPER_BUILD_TESTS OFF CACHE BOOL "" FORCE) -set(WHISPER_BUILD_SERVER OFF CACHE BOOL "" FORCE) -set(GGML_OPENMP OFF CACHE BOOL "" FORCE) -set(GGML_NATIVE OFF CACHE BOOL "" FORCE) -# ARMv8.2 with half floats and dot products: every phone that runs Android 12 or later in -# practice, and what makes Whisper fast enough to answer in under a second. -set(GGML_CPU_ARM_ARCH "armv8.2-a+dotprod+fp16" CACHE STRING "" FORCE) -FetchContent_MakeAvailable(whisper) - -add_library(gearslip_speech SHARED speech_jni.cpp) -target_compile_options(gearslip_speech PRIVATE -O3) -find_library(LOG_LIB log) -target_link_libraries(gearslip_speech whisper android ${LOG_LIB}) -target_link_options(gearslip_speech PRIVATE -Wl,--gc-sections -Wl,-z,max-page-size=16384) diff --git a/gearslip/src/main/cpp/speech_jni.cpp b/gearslip/src/main/cpp/speech_jni.cpp deleted file mode 100644 index bbd79c3..0000000 --- a/gearslip/src/main/cpp/speech_jni.cpp +++ /dev/null @@ -1,86 +0,0 @@ -// The bridge between Kotlin's SpeechEngine and whisper.cpp: load a model, turn one recorded -// phrase into text, free the model. Recording and deciding when a phrase ends stay in Kotlin. - -#include -#include -#include -#include -#include -#include -#include -#include "whisper.h" - -#define TAG "GearslipSpeech" -#define LOGI(...) __android_log_print(ANDROID_LOG_INFO, TAG, __VA_ARGS__) - -static whisper_context_params context_params() { - whisper_context_params p = whisper_context_default_params(); - p.use_gpu = false; - return p; -} - -// Streams a model straight out of the APK, so the bundled one is never copied to storage. -static size_t asset_read(void *ctx, void *output, size_t size) { - int n = AAsset_read(static_cast(ctx), output, size); - return n > 0 ? static_cast(n) : 0; -} -static bool asset_eof(void *ctx) { return AAsset_getRemainingLength64(static_cast(ctx)) <= 0; } -static void asset_close(void *ctx) { AAsset_close(static_cast(ctx)); } - -extern "C" JNIEXPORT jlong JNICALL -Java_app_seb3thehacker_gearslip_speech_SpeechEngine_nativeInitAsset(JNIEnv *env, jobject, jobject assets, jstring jpath) { - AAssetManager *manager = AAssetManager_fromJava(env, assets); - const char *path = env->GetStringUTFChars(jpath, nullptr); - AAsset *asset = AAssetManager_open(manager, path, AASSET_MODE_STREAMING); - env->ReleaseStringUTFChars(jpath, path); - if (!asset) return 0; - whisper_model_loader loader = {asset, asset_read, asset_eof, asset_close}; - return reinterpret_cast(whisper_init_with_params(&loader, context_params())); -} - -// Bytes, not a jstring: NewStringUTF wants modified UTF-8 and aborts on some of what Whisper can -// write (four-byte characters), so Kotlin decodes the plain UTF-8 instead. -extern "C" JNIEXPORT jbyteArray JNICALL -Java_app_seb3thehacker_gearslip_speech_SpeechEngine_nativeTranscribe( - JNIEnv *env, jobject, jlong handle, jfloatArray jsamples, jstring jlang, jint threads) { - auto *ctx = reinterpret_cast(handle); - if (!ctx) return env->NewByteArray(0); - jsize n = env->GetArrayLength(jsamples); - jfloat *samples = env->GetFloatArrayElements(jsamples, nullptr); - const char *lang = env->GetStringUTFChars(jlang, nullptr); - - whisper_full_params p = whisper_full_default_params(WHISPER_SAMPLING_GREEDY); - p.n_threads = threads; - p.language = lang; - p.translate = false; - p.no_context = true; - p.no_timestamps = true; - p.single_segment = true; - p.print_progress = false; - p.print_realtime = false; - p.print_special = false; - p.print_timestamps = false; - p.suppress_blank = true; - // The ACFT models are trained for a context cut to the audio's own length (1500 frames is - // 30 s, so 50 per second), which is what makes a three-second phrase fast. A little extra - // keeps the last word from being clipped. - p.audio_ctx = std::min(1500, static_cast(std::ceil(n / 320.0)) + 32); - - int rc = whisper_full(ctx, p, samples, n); - env->ReleaseStringUTFChars(jlang, lang); - env->ReleaseFloatArrayElements(jsamples, samples, JNI_ABORT); - if (rc != 0) { - LOGI("whisper_full failed: %d", rc); - return env->NewByteArray(0); - } - std::string text; - for (int i = 0; i < whisper_full_n_segments(ctx); i++) text += whisper_full_get_segment_text(ctx, i); - jbyteArray out = env->NewByteArray(static_cast(text.size())); - env->SetByteArrayRegion(out, 0, static_cast(text.size()), reinterpret_cast(text.data())); - return out; -} - -extern "C" JNIEXPORT void JNICALL -Java_app_seb3thehacker_gearslip_speech_SpeechEngine_nativeFree(JNIEnv *, jobject, jlong handle) { - if (handle) whisper_free(reinterpret_cast(handle)); -} diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/AppSettings.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/AppSettings.kt index 7c9cf73..bf4608d 100644 --- a/gearslip/src/main/java/app/seb3thehacker/gearslip/AppSettings.kt +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/AppSettings.kt @@ -9,7 +9,6 @@ object AppSettings { private const val PREFS = "gearslip_settings" private const val KEY_STARTUP_URL = "startup_url" - private const val KEY_SEEN_COMPAT_WARNING = "seen_compat_warning" private const val KEY_SKIPPED_CERT_SETUP = "skipped_cert_setup" private const val KEY_SEEN_PERMISSIONS_SETUP = "seen_permissions_setup" private const val KEY_GEARSLIP_ENABLED = "gearslip_enabled" @@ -32,14 +31,6 @@ object AppSettings { prefs(context).edit().putString(KEY_STARTUP_URL, value.trim()).apply() } - /** Whether the newer-car compatibility warning has already been shown and dismissed once. */ - fun hasSeenCompatWarning(context: Context): Boolean = - prefs(context).getBoolean(KEY_SEEN_COMPAT_WARNING, false) - - fun setSeenCompatWarning(context: Context) { - prefs(context).edit().putBoolean(KEY_SEEN_COMPAT_WARNING, true).apply() - } - /** Whether the certificate setup screen has been deliberately skipped. */ fun hasSkippedCertSetup(context: Context): Boolean = prefs(context).getBoolean(KEY_SKIPPED_CERT_SETUP, false) diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/Changelog.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/Changelog.kt index 346d9e8..205fd1b 100644 --- a/gearslip/src/main/java/app/seb3thehacker/gearslip/Changelog.kt +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/Changelog.kt @@ -10,6 +10,26 @@ object Changelog { class Release(val versionCode: Int, val versionName: String, val changes: List) val releases = listOf( + Release( + 9, "0.3.00-alpha", + listOf( + "Wireless Android Auto: pair the phone with the car once, and leave the cable at home.", + "Voice control is built in. Say \"navigate to a gas station\", \"play\" a song or artist, or open an app by name.", + "\"Navigate to\" picks the nearest place and stops at the map app's start screen, so you only tap Start.", + "Gearslip turns speech into text on the phone, with no Google account. Download the voice models once in setup.", + "The car's rotary knob moves between buttons, so cars without a touchscreen work too.", + "Choose how music reaches the car: Bluetooth, or USB with no delay. Gearslip offers to pair with the car the first time.", + "The car speaks each turn from your map app, and shows the directions on its own display where it can.", + "The car shows what's playing, and music apps get their own like, shuffle and repeat buttons.", + "Pick how the car's buttons look. Change it any time in the car's settings.", + "A new destination mid-route replaces the old route, and switching map apps ends the old app's route.", + "The car warns you when the phone's location is off.", + "Gearslip finds new versions and installs them for you.", + "More cars play music over USB, show a sharper picture, and put taps where you press.", + "On GrapheneOS, Gearslip no longer needs exploit protection compatibility mode.", + "The app is half the size it was.", + ), + ), Release( 8, "0.2.00-alpha", listOf( diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/GearslipActivity.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/GearslipActivity.kt index 8fe7da3..2f8ee50 100644 --- a/gearslip/src/main/java/app/seb3thehacker/gearslip/GearslipActivity.kt +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/GearslipActivity.kt @@ -8,13 +8,17 @@ import android.hardware.usb.UsbAccessory import android.hardware.usb.UsbManager import android.media.projection.MediaProjection import android.media.projection.MediaProjectionManager +import android.net.Uri import android.os.Bundle import android.os.ParcelFileDescriptor +import android.provider.Settings import android.view.MotionEvent import android.view.Surface import android.view.WindowManager import androidx.activity.ComponentActivity import androidx.activity.compose.setContent +import androidx.compose.runtime.collectAsState +import androidx.compose.runtime.getValue import androidx.activity.enableEdgeToEdge import androidx.activity.result.contract.ActivityResultContracts.StartActivityForResult import android.content.res.Configuration @@ -45,6 +49,8 @@ class GearslipActivity : ComponentActivity(), Projection { private var descriptor: ParcelFileDescriptor? = null + /** The wireless session's socket, when the car came over Wi-Fi instead of USB. */ + private var wirelessSocket: java.net.Socket? = null private var runner: GearslipRunner? = null private var screenProjector: ScreenProjector? = null private var worker: Thread? = null @@ -72,7 +78,13 @@ class GearslipActivity : ComponentActivity(), Projection { private val audioPermission = registerForActivityResult( androidx.activity.result.contract.ActivityResultContracts.RequestPermission(), ) { granted -> - if (granted) askAudioCapture() else { + if (granted) askAudioCapture() + else if (!shouldShowRequestPermissionRationale(android.Manifest.permission.RECORD_AUDIO)) { + // Refused twice: Android now says no without showing anything, so the phone must. + GearslipLog.w("audio: record permission blocked - Android no longer asks") + CarAudio.micBlocked() + micBlockedNotice.value = true + } else { GearslipLog.w("audio: record permission declined") CarAudio.denied() } @@ -85,14 +97,22 @@ class GearslipActivity : ComponentActivity(), Projection { } else { GearslipLog.w("audio: capture consent declined") CarAudio.denied() + if (!CarSettings.audioChoiceAsked) offerBluetooth.value = true } } + /** Raised when the microphone permission is blocked; see [MicBlockedDialog]. */ + private val micBlockedNotice = kotlinx.coroutines.flow.MutableStateFlow(false) + + /** Raised once, after the first "no" to the capture dialog; see [AudioChoiceDialog]. */ + private val offerBluetooth = kotlinx.coroutines.flow.MutableStateFlow(false) + private val detachReceiver = object : BroadcastReceiver() { override fun onReceive(context: Context?, intent: Intent?) { if (intent?.action == UsbManager.ACTION_USB_ACCESSORY_DETACHED) { GearslipLog.w("accessory detached") - closeAccessory() + // A wireless session has no accessory to lose; leave it running. + if (wirelessSocket == null) closeAccessory() } } } @@ -115,6 +135,9 @@ class GearslipActivity : ComponentActivity(), Projection { GearslipLog.init(this) GearslipLog.installCrashHandler() GearslipLog.i("Gearslip ready") + app.seb3thehacker.gearslip.wireless.WirelessLink.onConnected = { socket -> + runOnUiThread { startWireless(socket) } + } CarSettings.init(this) AppSettings.applyGearslipEnabled(this) // A capture that died without tidying up would leave notifications exposed to every @@ -127,7 +150,33 @@ class GearslipActivity : ComponentActivity(), Projection { setContent { GearslipTheme { - GearslipApp(onDisconnect = ::closeAccessory, onRequestCallScreening = ::requestCallScreeningRole) + GearslipApp(onDisconnect = { closeAccessory() }, onRequestCallScreening = ::requestCallScreeningRole) + val offer by offerBluetooth.collectAsState() + if (offer) { + fun close() { + CarSettings.markAudioChoiceAsked() + offerBluetooth.value = false + } + app.seb3thehacker.gearslip.ui.AudioChoiceDialog( + onUseBluetooth = { + CarSettings.setPipeAudio(false) + close() + }, + onKeepAsking = ::close, + ) + } + val micNotice by micBlockedNotice.collectAsState() + if (micNotice) { + app.seb3thehacker.gearslip.ui.MicBlockedDialog( + onOpenSettings = { + micBlockedNotice.value = false + startActivity( + Intent(Settings.ACTION_APPLICATION_DETAILS_SETTINGS, Uri.fromParts("package", packageName, null)), + ) + }, + onDismiss = { micBlockedNotice.value = false }, + ) + } } } @@ -152,6 +201,11 @@ class GearslipActivity : ComponentActivity(), Projection { super.onStart() Prefetch.warm(this) UsageStats.onAppOpened(this) + app.seb3thehacker.gearslip.update.UpdateChecker.onAppOpened(this) + // Back from the app's settings with the microphone turned on: try car audio again. + if (checkSelfPermission(android.Manifest.permission.RECORD_AUDIO) == android.content.pm.PackageManager.PERMISSION_GRANTED) { + CarAudio.micAllowed() + } } override fun onNewIntent(intent: Intent) { @@ -161,6 +215,7 @@ class GearslipActivity : ComponentActivity(), Projection { override fun onDestroy() { super.onDestroy() + app.seb3thehacker.gearslip.wireless.WirelessLink.onConnected = null runCatching { unregisterReceiver(detachReceiver) } PhoneMirror.onStartRequested = null PhoneMirror.onTokenReady = null @@ -296,16 +351,44 @@ class GearslipActivity : ComponentActivity(), Projection { worker = Thread { runner.run() }.apply { name = "gearslip"; start() } } + /** A wireless session: the same runner, on the socket [WirelessLink] opened to the car. */ + private fun startWireless(socket: java.net.Socket) { + closeAccessory(stopWireless = false) + if (!AppSettings.gearslipEnabled(this)) { + GearslipLog.i("wireless: Gearslip is switched off - not connecting") + runCatching { socket.close() } + return + } + GearslipLog.i("wireless: starting a session over Wi-Fi to ${socket.inetAddress?.hostAddress}:${socket.port}") + wirelessSocket = socket + val runner = GearslipRunner( + input = socket.getInputStream(), + output = socket.getOutputStream(), + identityProvider = { CertProvider.load(this) }, + projection = this, + vehicleProfileFor = { info -> VehicleProfiles.find(VehicleProfiles.load(this), info) }, + wireless = true, + ) + this.runner = runner + worker = Thread { runner.run() }.apply { name = "gearslip"; start() } + } + private fun describe(accessory: UsbAccessory) { GearslipLog.i("accessory: manufacturer=${accessory.manufacturer} model=${accessory.model}") GearslipLog.i(" version=${accessory.version} description=${accessory.description}") } - private fun closeAccessory() { + private fun closeAccessory() = closeAccessory(stopWireless = true) + + /** Ends the session, wired or wireless. [stopWireless] also drops the car's Bluetooth and Wi-Fi. */ + private fun closeAccessory(stopWireless: Boolean) { runner?.stop() SessionReport.print() runCatching { descriptor?.close() } descriptor = null + runCatching { wirelessSocket?.close() } + wirelessSocket = null + if (stopWireless) app.seb3thehacker.gearslip.wireless.WirelessLink.stop(this) runner = null } @@ -337,6 +420,8 @@ class GearslipActivity : ComponentActivity(), Projection { } override fun onProjectionStopped() { + // Before the car UI goes, while the media connection is still up. + app.seb3thehacker.gearslip.car.CarServices.pausePhoneMedia() runOnUiThread { PhoneMirror.requestStop() app.seb3thehacker.gearslip.car.CarKeys.focusSink = null diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/GearslipRunner.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/GearslipRunner.kt index 9dbb6aa..63c3c34 100644 --- a/gearslip/src/main/java/app/seb3thehacker/gearslip/GearslipRunner.kt +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/GearslipRunner.kt @@ -3,6 +3,12 @@ package app.seb3thehacker.gearslip import android.os.Build import app.seb3thehacker.gearslip.audio.AudioLink import app.seb3thehacker.gearslip.audio.CarMic +import app.seb3thehacker.gearslip.media.CarMediaStatus +import app.seb3thehacker.gearslip.car.BluetoothAudioCheck +import app.seb3thehacker.gearslip.car.LocationCheck +import app.seb3thehacker.gearslip.car.CarBluetooth +import app.seb3thehacker.gearslip.car.ClusterNav +import app.seb3thehacker.gearslip.car.DirectionsVoice import app.seb3thehacker.gearslip.car.CarAssistant import app.seb3thehacker.gearslip.car.CarEnvironment import app.seb3thehacker.gearslip.car.CarKeys @@ -33,6 +39,8 @@ class GearslipRunner( private val identityProvider: () -> CertProvider.Identity, private val projection: Projection? = null, private val vehicleProfileFor: (ServiceDiscovery.HeadUnitInfo) -> VehicleProfile? = { null }, + /** Over Wi-Fi rather than USB: the video is sent smaller so the link keeps up. */ + private val wireless: Boolean = false, ) { private val log = GearslipLog.tagged("PROTO") @@ -43,10 +51,15 @@ class GearslipRunner( private lateinit var identityTask: FutureTask /** Sends every message, in order, so no other thread ever blocks on the USB link. See [send]. */ - private val writer = java.util.concurrent.Executors.newSingleThreadExecutor { Thread(it, "gearslip-writer") } + private val writer = java.util.concurrent.Executors.newSingleThreadExecutor { + // Sound goes out through here too, so it gets audio's priority: late writes are skips. + Thread({ android.os.Process.setThreadPriority(android.os.Process.THREAD_PRIORITY_AUDIO); it.run() }, "gearslip-writer") + } /** Waits for the certificate only when TLS is first needed; see [run]. */ private val tls: PhoneTls by lazy { identityTask.get() } + private var authenticationFailed = false + @Volatile private var stopped = false @Volatile private var state = State.WAIT_VERSION @Volatile private var lastProgress = System.currentTimeMillis() @@ -68,8 +81,12 @@ class GearslipRunner( @Volatile private var inputChannelId: Int = -1 @Volatile private var inputKeycodes: List = emptyList() @Volatile private var sensorChannelId: Int = -1 + @Volatile private var mediaStatus: CarMediaStatus? = null + @Volatile private var clusterNav: ClusterNav? = null private var sensorTypes: List = emptyList() @Volatile private var audioLink: AudioLink? = null + @Volatile private var guidanceLink: AudioLink? = null + @Volatile private var vendorChannelId = -1 private var audioMessagesSeen = 0 private var touchWidth = 0 private var touchHeight = 0 @@ -82,7 +99,9 @@ class GearslipRunner( @Volatile private var lastKeyframeSentAt = 0L fun run() { - SessionReport.begin() + // This thread reads the car's acks; audio waits on them, so it can't be left at the back. + android.os.Process.setThreadPriority(android.os.Process.THREAD_PRIORITY_AUDIO) + SessionReport.begin(wireless) log.i("--- session starting ---") SessionStatus.connecting("Connected to the head unit") log.i("device=${Build.MANUFACTURER} ${Build.MODEL} android=${Build.VERSION.RELEASE} (sdk ${Build.VERSION.SDK_INT})") @@ -107,6 +126,7 @@ class GearslipRunner( try { while (running) { val read = input.read(chunk) + if (stopped) return if (read < 0) { log.w("head unit closed the connection (EOF) while in state $state") reportDisconnect() @@ -120,16 +140,19 @@ class GearslipRunner( } } } catch (t: Throwable) { + if (stopped) return log.e("transport failed in state $state", t) SessionReport.fail(SessionReport.Category.USB, "transport failed", t, state.name) reportDisconnect() } finally { running = false + writer.shutdownNow() log.flush() } } fun stop() { + stopped = true running = false writer.shutdownNow() CarFocus.request = null @@ -139,8 +162,17 @@ class GearslipRunner( videoSource = null audioLink?.close() audioLink = null + guidanceLink?.close() + guidanceLink = null CarSensors.clear() CarMic.current = null + CarBluetooth.onCarDisconnected() + BluetoothAudioCheck.onCarDisconnected() + LocationCheck.onCarDisconnected() + mediaStatus?.stop() + mediaStatus = null + clusterNav?.stop() + clusterNav = null lastSensorNight = null CarEnvironment.setSensorNight(null) // back to the clock until a car says otherwise projection?.onProjectionStopped() @@ -148,6 +180,8 @@ class GearslipRunner( } private fun handleFrame(frame: Frames.Frame) { + // A rejected handshake cannot consume later messages as if authentication succeeded. + if (authenticationFailed || stopped) return val payload = if (frame.encrypted) { if (!tls.handshakeComplete) { log.w("encrypted frame arrived before the handshake finished - ignoring") @@ -176,6 +210,16 @@ class GearslipRunner( onInputMessage(messageId, body) } else if (frame.channel == sensorChannelId) { onSensorMessage(messageId, body) + } else if (frame.channel == clusterNav?.channelId) { + clusterNav?.onMessage(messageId, body) + } else if (frame.channel == mediaStatus?.channelId) { + mediaStatus?.onMessage(messageId, body) + } else if (frame.channel == guidanceLink?.channelId) { + guidanceLink?.onMessage(messageId, body) + } else if (frame.channel == CarBluetooth.channelId) { + CarBluetooth.onMessage(messageId, body) + } else if (frame.channel == vendorChannelId) { + onVendorMessage(messageId, body) } else if (frame.channel == CarMic.current?.channelId) { CarMic.current?.onMessage(messageId, body) } else if (frame.channel == audioLink?.channelId) { @@ -225,7 +269,7 @@ class GearslipRunner( } log.w("<- ByeByeRequest: reason=$reason ($name) in state $state") log.hex(" byebye raw", body) - SessionReport.fail(SessionReport.Category.BYEBYE, "ByeByeRequest reason=$reason ($name)", state.name) + SessionReport.fail(SessionReport.Category.BYEBYE, "ByeByeRequest reason=$reason ($name)", state.name, "bye $reason") send(MSG_BYEBYE_RESPONSE, ByteArray(0), encrypted = true) log.i("-> ByeByeResponse") } @@ -240,11 +284,10 @@ class GearslipRunner( log.i("<- VersionRequest: head unit speaks $major.$minor") SessionReport.protocolVersion(major, minor) - // PROTOCOL-DOWNGRADE EXPERIMENT. The 2018 Uconnect speaks 1.3 and ACCEPTS the JVC cert; - // the 2025 unit speaks 4.3 and rejects it at AuthComplete (-3). If the 4.3 identity check - // is tied to the *negotiated* protocol version rather than baked into the firmware, - // claiming an older version makes the newer unit skip the check entirely. CLAIM_VERSION - // = null restores the previous echo-the-head-unit behaviour. + // Always 1.3, on purpose. Every car that accepts Gearslip was tested this way, 1.5 units + // included. Echoing the car's own version was tried too: a 4.3 unit rejected the + // certificate either way, so the version doesn't decide acceptance. The log says so, since + // a car offering 1.5 and getting 1.3 back looks like a bug to anyone reading it. val claim = CLAIM_VERSION ?: (major to minor) val response = byteArrayOf( ((claim.first shr 8) and 0xFF).toByte(), (claim.first and 0xFF).toByte(), @@ -254,7 +297,10 @@ class GearslipRunner( send(MSG_VERSION_RESPONSE, response, encrypted = false) log.i( "-> VersionResponse: ${claim.first}.${claim.second} status=0 (STATUS_SUCCESS)" + - if (CLAIM_VERSION != null) " [FORCED; head unit offered $major.$minor]" else "", + if (CLAIM_VERSION != null && claim != (major to minor)) { + " - intended: Gearslip always answers ${claim.first}.${claim.second}, which cars " + + "offering $major.$minor accept too" + } else "", ) state = State.TLS_HANDSHAKE SessionStatus.connecting("Securing the link") @@ -262,10 +308,14 @@ class GearslipRunner( } private fun onHandshake(body: ByteArray) { + if (state != State.TLS_HANDSHAKE && state != State.WAIT_AUTH) return log.i("<- EncapsulatedSSL (${body.size} bytes)") val replies = try { - tls.pumpHandshake(body) + val preparedTls = tls + if (stopped) return + preparedTls.pumpHandshake(body) } catch (t: Throwable) { + if (stopped) return log.e("TLS handshake threw", t) SessionReport.fail(SessionReport.Category.TLS, "handshake threw", t, state.name) log.verdict( @@ -278,9 +328,11 @@ class GearslipRunner( "The head unit aborted the secure handshake. It validates the phone's certificate.", ) state = State.DONE + authenticationFailed = true SessionReport.print() return } + if (stopped) return replies.forEach { send(MSG_ENCAPSULATED_SSL, it, encrypted = false) log.i("-> EncapsulatedSSL (${it.size} bytes)") @@ -293,6 +345,7 @@ class GearslipRunner( private fun onAuthComplete(body: ByteArray) { val status = Protobuf.readInt32Field(body, 1) + if (state != State.WAIT_AUTH && !(state == State.TLS_HANDSHAKE && status in listOf(-2, -3))) return log.i("<- AuthComplete: status=$status") status?.let { SessionReport.auth(it) } @@ -304,7 +357,7 @@ class GearslipRunner( sendServiceDiscoveryRequest() } -2 -> { - SessionReport.fail(SessionReport.Category.CERTIFICATE, "AuthComplete status -2 (certificate error)", state.name) + SessionReport.fail(SessionReport.Category.CERTIFICATE, "AuthComplete status -2 (certificate error)", state.name, "auth -2") log.verdict( "NOT VIABLE (STATUS_CERTIFICATE_ERROR)", "The head unit explicitly rejected the phone's self-signed certificate. " + @@ -316,9 +369,10 @@ class GearslipRunner( "The head unit refused the phone's certificate (certificate error).", ) state = State.DONE + authenticationFailed = true } -3 -> { - SessionReport.fail(SessionReport.Category.AUTH, "AuthComplete status -3 (authentication failure)", state.name) + SessionReport.fail(SessionReport.Category.AUTH, "AuthComplete status -3 (authentication failure)", state.name, "auth -3") log.verdict( "NOT VIABLE (STATUS_AUTHENTICATION_FAILURE)", "Authentication rejected. Broader than a pure certificate error, but the " + @@ -329,9 +383,10 @@ class GearslipRunner( "Authentication failed. The head unit does not trust the certificate in use.", ) state = State.DONE + authenticationFailed = true } else -> { - SessionReport.fail(SessionReport.Category.AUTH, "AuthComplete status $status (unexpected)", state.name) + SessionReport.fail(SessionReport.Category.AUTH, "AuthComplete status $status (unexpected)", state.name, "auth $status") log.verdict( "INCONCLUSIVE (AuthComplete status=$status)", "Unexpected status. Check MessageStatus.proto in aasdk for the meaning " + @@ -339,6 +394,7 @@ class GearslipRunner( ) SessionStatus.failed("Unexpected response", "The head unit answered authentication with status $status.") state = State.DONE + authenticationFailed = true } } } @@ -399,7 +455,7 @@ class GearslipRunner( val video = ServiceDiscovery.findVideoService(serviceDiscoveryResponse) if (video == null) { log.w("no video service in the discovery response - cannot start Phase A") - SessionReport.fail(SessionReport.Category.VIDEO, "head unit advertised no video service", state.name) + SessionReport.fail(SessionReport.Category.VIDEO, "head unit advertised no video service", state.name, "no video service") return } videoChannelId = video.serviceId @@ -415,8 +471,13 @@ class GearslipRunner( openChannel(video.serviceId) startAudioChannel(serviceDiscoveryResponse) + startGuidanceChannel(serviceDiscoveryResponse) startSensorChannel(serviceDiscoveryResponse) startMicChannel(serviceDiscoveryResponse) + startMediaStatusChannel(serviceDiscoveryResponse) + startClusterNavChannel(serviceDiscoveryResponse) + startBluetooth(serviceDiscoveryResponse) + startVendorChannel(serviceDiscoveryResponse) val input = ServiceDiscovery.findInputService(serviceDiscoveryResponse, video.displayId) if (input == null) { @@ -434,7 +495,7 @@ class GearslipRunner( } } - /** Media audio rides its own sink; guidance, system and telephony sinks are left closed. */ + /** Media audio rides its own sink; spoken directions take the guidance sink, and system and telephony stay closed. */ private fun startAudioChannel(serviceDiscoveryResponse: ByteArray) { val sinks = ServiceDiscovery.findAudioServices(serviceDiscoveryResponse) sinks.forEach { sink -> @@ -451,7 +512,7 @@ class GearslipRunner( return } if (media.codecType != ServiceDiscovery.CODEC_PCM) { - log.w("the car offers no PCM media sink, only ${media.codecName}; it may play our PCM as silence") + log.w("the car offers no PCM media sink, only ${media.codecName}; music goes as AAC") } val first = media.configs.firstOrNull() SessionReport.audio( @@ -462,10 +523,34 @@ class GearslipRunner( media, sendOnChannel = { id, body -> send(id, body, encrypted = true, channel = media.serviceId) }, sendControl = { id, body -> send(id, body, encrypted = true) }, + // The driver picked the radio or another source on the car: pause the phone's music + // as Android Auto does, instead of letting it play on to no one. + onTakenBack = { CarServices.pausePhoneMedia("the car switched to another source") }, ) openChannel(media.serviceId) } + /** The car's directions channel, where [DirectionsVoice] reads out each turn. */ + private fun startGuidanceChannel(serviceDiscoveryResponse: ByteArray) { + val guidance = ServiceDiscovery.findAudioServices(serviceDiscoveryResponse) + .filter { it.streamType == ServiceDiscovery.STREAM_GUIDANCE } + .firstOrNull { it.codecType == ServiceDiscovery.CODEC_PCM } ?: run { + log.i("no PCM directions sink - turns won't be spoken") + return + } + guidanceLink = AudioLink( + guidance, + sendOnChannel = { id, body -> send(id, body, encrypted = true, channel = guidance.serviceId) }, + sendControl = { id, body -> send(id, body, encrypted = true) }, + tag = "guidance", + sessionId = GUIDANCE_SESSION_ID, + asksFocus = false, + onReady = { DirectionsVoice.attach(it) }, + onClose = { DirectionsVoice.detach(it) }, + ) + openChannel(guidance.serviceId) + } + /** The car's microphone, for voice: opened now, recording only while something listens. */ private fun startMicChannel(serviceDiscoveryResponse: ByteArray) { val mic = ServiceDiscovery.findMicService(serviceDiscoveryResponse) ?: run { @@ -474,10 +559,75 @@ class GearslipRunner( } val config = mic.configs.first() log.i("car microphone: channel=${mic.serviceId} ${mic.codecName} ${config.sampleRate}Hz/${config.bits}bit/x${config.channels}") - CarMic.current = CarMic(mic) { id, body -> send(id, body, encrypted = true, channel = mic.serviceId) } + CarMic.current = CarMic( + mic, + sendOnChannel = { id, body -> send(id, body, encrypted = true, channel = mic.serviceId) }, + // VoiceSessionNotification { status = 1 START, 2 END }, on the control channel. + voiceSession = { start -> + send(MSG_VOICE_SESSION_NOTIFICATION, Protobuf.varintField(1, if (start) 1L else 2L), encrypted = true) + }, + ) openChannel(mic.serviceId) } + /** What's playing, for the car's own screens: the cluster, the media source page. */ + private fun startMediaStatusChannel(serviceDiscoveryResponse: ByteArray) { + val channel = ServiceDiscovery.findMediaStatusChannel(serviceDiscoveryResponse) ?: run { + log.i("no media status channel - the car's own screens won't show what's playing") + return + } + log.i("media status channel: $channel") + mediaStatus = CarMediaStatus(channel) { id, body -> send(id, body, encrypted = true, channel = channel) } + openChannel(channel) + } + + /** Bluetooth pairing, for calls: the channel only opens if the driver asks to pair. */ + /** + * GM's own channel ("com.gm.vip"), opened in GM cars only. Issue #18: a 2021 Bolt ends every + * session about 25 s in, and of the channels it offers, this is the one a Dacia that stays + * connected doesn't have. What GM sends on it is private, so Gearslip only opens it and logs + * what arrives, by message id and size. + */ + private fun startVendorChannel(serviceDiscoveryResponse: ByteArray) { + val vendors = ServiceDiscovery.findVendorServices(serviceDiscoveryResponse) + vendors.forEach { log.i("vendor channel ${it.serviceId}: ${it.name}") } + val gm = vendors.firstOrNull { it.name.startsWith("com.gm.") } ?: return + vendorChannelId = gm.serviceId + log.i("-> ChannelOpenRequest on GM's channel ${gm.serviceId}") + openChannel(gm.serviceId) + } + + private fun onVendorMessage(messageId: Int, body: ByteArray) { + if (messageId == MSG_CHANNEL_OPEN_RESPONSE) { + log.i("<- GM channel open: status=${Protobuf.readInt32Field(body, 1) ?: 0}") + } else { + log.i("<- GM channel message id=$messageId (${body.size} bytes)") + } + } + + private fun startBluetooth(serviceDiscoveryResponse: ByteArray) { + val bluetooth = ServiceDiscovery.findBluetoothService(serviceDiscoveryResponse) ?: run { + log.i("no Bluetooth service - the car can't be paired from here") + return + } + CarBluetooth.onCarConnected( + bluetooth, + open = { openChannel(bluetooth.serviceId) }, + send = { id, body -> send(id, body, encrypted = true, channel = bluetooth.serviceId) }, + ) + } + + /** Turn-by-turn for the instrument cluster: the next turn, its road, and how far. */ + private fun startClusterNavChannel(serviceDiscoveryResponse: ByteArray) { + val channel = ServiceDiscovery.findNavStatusChannel(serviceDiscoveryResponse) ?: run { + log.i("no navigation status channel - the cluster won't show turns") + return + } + log.i("navigation status channel: $channel") + clusterNav = ClusterNav(channel) { id, body -> send(id, body, encrypted = true, channel = channel) } + openChannel(channel) + } + /** * Every sensor type the head unit advertised, subscribed to at once - there is no screen * to pick and choose yet, this is meant to show everything reachable. [CarSensors] answers @@ -599,8 +749,8 @@ class GearslipRunner( * right-hand third of the screen away entirely - taps on a right-edge button landed on * the middle one. The head unit maps its own panel onto our video before reporting. * - * So: inject as-is, clamped to the video bounds. The advertised touchscreen size is kept - * only for diagnostics. + * So: inject them unscaled, shifted past the cropped margin (see below) and clamped to the + * video. The advertised touchscreen size is kept only for diagnostics. */ private fun onInputMessage(messageId: Int, body: ByteArray) { when (messageId) { @@ -631,13 +781,20 @@ class GearslipRunner( val action = Wire.varint(touchFields, 3)?.toInt() ?: 0 val pointers = Wire.allBytes(touchFields, 1) if (pointers.isEmpty()) return - val size = videoConfigs.getOrNull(selectedConfigIndex ?: 0)?.pixelSize() + val config = videoConfigs.getOrNull(selectedConfigIndex ?: 0) + val size = config?.pixelSize() + // The car crops its margin off the frame and measures taps from the corner of what + // it shows, so they come in short by half the margin: the 2024 VW (margin 240) sent + // y=6, which is inside the cropped band, and the Uconnect's never passed 351 on a + // 480-high frame with a 130 margin. Without this every tap landed 120 px too high + // and the VW couldn't press past its first screen. + val shift = config?.marginInsets() ?: Insets.NONE val points = pointers.mapIndexedNotNull { index, pointer -> val fields = Wire.fields(pointer) val rawX = Wire.varint(fields, 1)?.toInt() ?: return@mapIndexedNotNull null val rawY = Wire.varint(fields, 2)?.toInt() ?: return@mapIndexedNotNull null val id = Wire.varint(fields, 3)?.toInt() ?: index - TouchPoint(id, clamp(rawX, size?.first), clamp(rawY, size?.second)) + TouchPoint(id, clamp(rawX + shift.left, size?.first), clamp(rawY + shift.top, size?.second)) } if (points.isEmpty()) return val actionIndex = (Wire.varint(touchFields, 2)?.toInt() ?: 0).coerceIn(0, points.size - 1) @@ -795,13 +952,17 @@ class GearslipRunner( /** * The offered config that best fills the car's screen: the smallest whose visible width * reaches most of the touchscreen's, else the largest there is. Always taking the smallest - * put 800x480 on a 1540x720 Audi screen and stretched it. With no touchscreen size to go - * by, the smallest is still the safe choice. + * put 800x480 on a 1540x720 Audi screen and stretched it. A car that gives no touchscreen + * size gets the largest up to 1080p: a car only offers sizes it can show, and the 2024 VW + * (touchscreen 0x0) blurred 800x480 across a 1080p screen. */ private fun fitToScreen(offered: List): Int? { val known = offered.filter { videoConfigs.getOrNull(it)?.pixelSize() != null } fun pixels(idx: Int) = videoConfigs[idx].pixelCount() - if (touchWidth <= 0) return known.minByOrNull(::pixels) + if (touchWidth <= 0) { + return known.filter { pixels(it) <= 1920 * 1080 }.maxByOrNull(::pixels) + ?: known.minByOrNull(::pixels) + } val wide = known.filter { (videoConfigs[it].visibleSize()?.first ?: 0) >= touchWidth * 8 / 10 } return wide.minByOrNull(::pixels) ?: known.maxByOrNull(::pixels) } @@ -836,8 +997,10 @@ class GearslipRunner( videoSource = VideoSource( width = width, height = height, - frameRate = if (config.frameRate == 1) 60 else 30, + // 60fps only over a cable: over Wi-Fi it doubles the data for little the driver sees. + frameRate = if (config.frameRate == 1 && !wireless) 60 else 30, mode = if (useProjection) VideoSource.Mode.SURFACE else VideoSource.Mode.TEST_CARD, + wireless = wireless, onCodecConfig = { csd -> lastCodecConfig = csd send(MSG_MEDIA_CODEC_CONFIG, csd, encrypted = true, channel = videoChannelId) @@ -854,6 +1017,8 @@ class GearslipRunner( if (projection != null && surface != null) { projection.onSurfaceReady(surface, width, height, config.density) SessionStatus.projecting() + BluetoothAudioCheck.onProjecting() + LocationCheck.onProjecting() } }.onFailure { e -> log.e("encoder start failed", e) } } @@ -1065,10 +1230,10 @@ class GearslipRunner( private fun reportDisconnect() { when (state) { - State.WAIT_VERSION -> SessionReport.fail(SessionReport.Category.NO_HANDSHAKE, "no VersionRequest arrived", state.name) - State.TLS_HANDSHAKE -> SessionReport.fail(SessionReport.Category.TLS, "head unit dropped the link during TLS", state.name) - State.WAIT_AUTH -> SessionReport.fail(SessionReport.Category.AUTH, "dropped after TLS, before AuthComplete", state.name) - State.WAIT_SDR -> SessionReport.fail(SessionReport.Category.SERVICE_DISCOVERY, "accepted, then dropped before service discovery", state.name) + State.WAIT_VERSION -> SessionReport.fail(SessionReport.Category.NO_HANDSHAKE, "no VersionRequest arrived", state.name, "closed") + State.TLS_HANDSHAKE -> SessionReport.fail(SessionReport.Category.TLS, "head unit dropped the link during TLS", state.name, "closed") + State.WAIT_AUTH -> SessionReport.fail(SessionReport.Category.AUTH, "dropped after TLS, before AuthComplete", state.name, "closed") + State.WAIT_SDR -> SessionReport.fail(SessionReport.Category.SERVICE_DISCOVERY, "accepted, then dropped before service discovery", state.name, "closed") State.DONE -> {} } when (state) { @@ -1155,9 +1320,9 @@ class GearslipRunner( private companion object { /** - * Force the protocol version Gearslip claims in VersionResponse, as (major, minor), or - * null to echo the head unit. 1.3 is what the 2018 Uconnect (which accepts the JVC cert) - * speaks. This is the 2025 identity-check downgrade experiment: see onVersionRequest. + * The protocol version Gearslip answers every car with, as (major, minor), or null to echo + * the head unit's. Kept at 1.3 because every working car was tested with it; see + * onVersionRequest. */ val CLAIM_VERSION: Pair? = 1 to 3 @@ -1172,6 +1337,7 @@ class GearslipRunner( const val MSG_PING_RESPONSE = 12 const val MSG_BYEBYE_REQUEST = 15 const val MSG_BYEBYE_RESPONSE = 16 + const val MSG_VOICE_SESSION_NOTIFICATION = 17 const val MSG_CHANNEL_OPEN_REQUEST = 7 const val MSG_CHANNEL_OPEN_RESPONSE = 8 @@ -1212,6 +1378,7 @@ class GearslipRunner( const val ACK_STALL_MS = 2_000L const val AUDIO_TRACE_MESSAGES = 6 + const val GUIDANCE_SESSION_ID = 1 const val PERIODIC_KEYFRAME_MS = 5_000L } } diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/Protobuf.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/Protobuf.kt index e253147..c651284 100644 --- a/gearslip/src/main/java/app/seb3thehacker/gearslip/Protobuf.kt +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/Protobuf.kt @@ -31,6 +31,9 @@ object Protobuf { return varint(((field shl 3) or 2).toLong()) + varint(bytes.size.toLong()) + bytes } + fun bytesField(field: Int, bytes: ByteArray): ByteArray = + varint(((field shl 3) or 2).toLong()) + varint(bytes.size.toLong()) + bytes + fun varintField(field: Int, value: Long): ByteArray = varint((field shl 3).toLong()) + varint(value) diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/ScreenProjector.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/ScreenProjector.kt index 7a29fc6..c9ffee5 100644 --- a/gearslip/src/main/java/app/seb3thehacker/gearslip/ScreenProjector.kt +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/ScreenProjector.kt @@ -198,6 +198,12 @@ class ScreenProjector(private val context: Context) { */ fun dispatchKey(keyCode: Int, down: Boolean) { val target = root ?: return + // Keys handed straight to the view skip the step where Android leaves touch mode, and in + // touch mode Compose's buttons refuse focus, so the knob had nothing to move between. + if (target.isInTouchMode) { + target.requestFocusFromTouch() + GearslipLog.i("keys: left touch mode for the knob/D-pad (now in touch mode: ${target.isInTouchMode})") + } // Nothing on the display takes keys until something in it holds focus. if (target.findFocus() == null) target.requestFocus() val shift = keyCode == app.seb3thehacker.gearslip.car.CarKeys.KEYCODE_SHIFT_TAB diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/ServiceDiscovery.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/ServiceDiscovery.kt index 3cf57e6..b9f5266 100644 --- a/gearslip/src/main/java/app/seb3thehacker/gearslip/ServiceDiscovery.kt +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/ServiceDiscovery.kt @@ -110,6 +110,21 @@ object ServiceDiscovery { return media.firstOrNull { it.codecType == CODEC_PCM } ?: media.firstOrNull() } + /** + * The channel the car takes "now playing" on (MediaPlaybackStatusService, Service field 9), + * which feeds its cluster and media screens. It carries no settings. Null when the car has none. + */ + fun findMediaStatusChannel(response: ByteArray): Int? = findChannelWith(response, 9) + + /** The instrument cluster's turn-by-turn channel (NavigationStatusService, Service field 8). */ + fun findNavStatusChannel(response: ByteArray): Int? = findChannelWith(response, 8) + + private fun findChannelWith(response: ByteArray, serviceField: Int): Int? = + Wire.allBytes(Wire.fields(response), 1).firstNotNullOfOrNull { channel -> + val service = Wire.fields(channel) + if (service.any { it.number == serviceField }) Wire.varint(service, 1)?.toInt() else null + } + /** * The car's microphone, from MediaSourceService (Service field 5): * MediaSourceService { MediaCodecType available_type = 1; AudioConfiguration audio_config = 2; @@ -138,8 +153,45 @@ object ServiceDiscovery { return null } + /** + * The car's Bluetooth, from BluetoothService (Service field 6): + * BluetoothService { string car_address = 1; repeated BluetoothPairingMethod methods = 2 [packed]; } + * Null when the car offers no pairing over the link. + */ + class BluetoothService(val serviceId: Int, val carAddress: String, val pairingMethods: List) + + /** A carmaker's own channel (service field 12), for its apps on the phone: GM's is "com.gm.vip". */ + class VendorService(val serviceId: Int, val name: String) + + fun findVendorServices(response: ByteArray): List = + Wire.allBytes(Wire.fields(response), 1).mapNotNull { channel -> + val service = Wire.fields(channel) + val id = Wire.varint(service, 1)?.toInt() ?: return@mapNotNull null + val fields = Wire.fields(Wire.bytes(service, 12) ?: return@mapNotNull null) + val name = Wire.bytes(fields, 1)?.let { String(it, Charsets.UTF_8) } ?: return@mapNotNull null + VendorService(id, name) + } + + fun findBluetoothService(response: ByteArray): BluetoothService? { + for (channel in Wire.allBytes(Wire.fields(response), 1)) { + val service = Wire.fields(channel) + val id = Wire.varint(service, 1)?.toInt() ?: continue + val fields = Wire.fields(Wire.bytes(service, 6) ?: continue) + val address = Wire.bytes(fields, 1)?.let { String(it, Charsets.UTF_8) } ?: continue + // Packed as declared, though a car may send each method as its own varint. + val methods = fields.filter { it.number == 2 }.flatMap { f -> + f.bytes?.let(::packedVarints) ?: listOf(f.varint.toInt()) + } + return BluetoothService(id, address, methods) + } + return null + } + + const val STREAM_GUIDANCE = 1 const val STREAM_MEDIA = 3 const val CODEC_PCM = 1 + const val CODEC_AAC_LC = 2 + const val CODEC_AAC_LC_ADTS = 4 private val STREAMS = mapOf(0 to "NONE", 1 to "GUIDANCE", 2 to "SYSTEM_AUDIO", 3 to "MEDIA", 4 to "TELEPHONY") diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/SessionReport.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/SessionReport.kt index fe2a01a..5d0fae2 100644 --- a/gearslip/src/main/java/app/seb3thehacker/gearslip/SessionReport.kt +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/SessionReport.kt @@ -29,12 +29,30 @@ object SessionReport { UNKNOWN("unclassified"), } + /** + * How far a session got, in the order a connection is built. The usage note's "Where + * connections fail" share sends the furthest one reached, so a failure says where it broke. + */ + enum class Stage(val key: String) { + USB("usb"), // the link opened; the car may have said nothing + VERSIONS("versions"), // the car said which protocol it speaks + TLS("tls"), // the secure link is up + ACCEPTED("accepted"), // the car accepted the certificate + CHANNELS("channels"), // the car listed its channels and said who it is + VIDEO("video"), // the car's screen is set up for Gearslip + STREAMING("streaming"), // video frames are reaching the car + } + private var appVersion = "?" private var appCode = 0L @Volatile private var startedAt = 0L @Volatile var category = Category.NONE; private set @Volatile private var failure = "" + @Volatile var stage = Stage.USB; private set + /** What the car or the phone said when it broke: a TLS alert, an auth status, a ByeBye reason, an errno. */ + @Volatile private var failCode = "" + @Volatile private var failedAfter = 0L @Volatile private var protocolVersion = "" @Volatile private var certSource = "" @Volatile private var certIssuer = "" @@ -58,10 +76,17 @@ object SessionReport { appCode = info?.longVersionCode ?: 0 } - fun begin() { + /** How this drive reached the car, and how its music did: "usb" or "wireless", "usb" or "bluetooth". */ + @Volatile var link = ""; private set + @Volatile var audioMode = ""; private set + + fun begin(wireless: Boolean) { startedAt = System.currentTimeMillis() + link = if (wireless) "wireless" else "usb" + audioMode = if (app.seb3thehacker.gearslip.car.CarSettings.pipeAudio.value) "usb" else "bluetooth" category = Category.NONE failure = ""; protocolVersion = ""; certSource = ""; certIssuer = "" + stage = Stage.USB; failCode = ""; failedAfter = 0 tlsProtocol = ""; tlsCipher = ""; authStatus = null headUnitInfo = null carScreen = ""; carDpi = 0 @@ -69,34 +94,87 @@ object SessionReport { printed = false } - fun protocolVersion(major: Int, minor: Int) { protocolVersion = "$major.$minor" } + /** Moves the session's furthest stage on; never back. */ + private fun reached(next: Stage) { if (next > stage) stage = next } + + fun protocolVersion(major: Int, minor: Int) { protocolVersion = "$major.$minor"; reached(Stage.VERSIONS) } fun certificate(source: String, issuer: String) { certSource = source; certIssuer = issuer } - fun tls(protocol: String, cipher: String) { tlsProtocol = protocol; tlsCipher = cipher } - fun auth(status: Int) { authStatus = status } + fun tls(protocol: String, cipher: String) { tlsProtocol = protocol; tlsCipher = cipher; reached(Stage.TLS) } + fun auth(status: Int) { authStatus = status; if (status == 0) reached(Stage.ACCEPTED) } fun headUnit(info: String) { headUnit = info } /** Who the car said it is, for the opt-in usage note; null until service discovery. */ @Volatile var headUnitInfo: ServiceDiscovery.HeadUnitInfo? = null + set(value) { field = value; if (value != null) reached(Stage.CHANNELS) } /** The car screen's usable size once its margins are cropped, and its density; blank before video starts. */ @Volatile var carScreen = ""; private set @Volatile var carDpi = 0; private set - fun carScreen(size: String, dpi: Int) { carScreen = size; carDpi = dpi } + fun carScreen(size: String, dpi: Int) { carScreen = size; carDpi = dpi; reached(Stage.VIDEO) } fun profile(name: String?) { profile = name ?: "none matched" } fun video(summary: String) { video = summary } fun audio(summary: String) { audio = summary } - fun stream(summary: String) { lastStream = summary } + fun stream(summary: String) { lastStream = summary; reached(Stage.STREAMING) } - /** Records the first failure only: later ones are usually fallout from it. */ - fun fail(category: Category, detail: String, state: String = "") { + /** + * Records the first failure only: later ones are usually fallout from it. [code] is the short + * thing the car or the phone said, like "auth -3" or "bye 1", for the usage note. + */ + fun fail(category: Category, detail: String, state: String = "", code: String = "") { if (this.category != Category.NONE) return this.category = category failure = detail lastState = state + failCode = code + failedAfter = if (startedAt == 0L) 0 else (System.currentTimeMillis() - startedAt) / 1000 } fun fail(category: Category, detail: String, t: Throwable, state: String = "") = - fail(category, "$detail (${GearslipLog.rootCause(t)})", state) + fail(category, "$detail (${GearslipLog.rootCause(t)})", state, codeOf(t)) + + /** + * A short code from an exception, never its whole message: the TLS alert's name, the errno + * of a USB failure, or else the exception's class. + */ + private fun codeOf(t: Throwable): String { + var root = t + while (root.cause != null && root.cause !== root) root = root.cause!! + val text = "${t.message} ${root.message}" + Regex("(?i)alert[ _:]*\\(?([a-z_ ]{3,40}?)\\)?(?:\\s|$|[,;.])").find(text)?.let { + return "alert ${it.groupValues[1].trim().lowercase().replace(' ', '_')}" + } + Regex("\\b(E[A-Z]{2,12})\\b").find(text)?.let { return it.groupValues[1] } + return root.javaClass.simpleName.take(40) + } + + /** + * One session for the usage note's "Where connections fail" share: how far it got, how and + * when it ended, and with which kind of certificate. Counts and codes only, nothing personal. + */ + fun sessionNote(): org.json.JSONObject { + val seconds = if (startedAt == 0L) 0 else (System.currentTimeMillis() - startedAt) / 1000 + val source = certSource.lowercase() + return org.json.JSONObject() + .put("stage", stage.key) + .put("ended", category.name.lowercase()) + .put("code", failCode) + .put("failed_at", if (category == Category.NONE) 0 else rounded(failedAfter)) + .put("seconds", rounded(seconds)) + .put("cert", when { + "imported" in source -> "imported" + "downloaded" in source -> "downloaded" + "self" in source -> "self-signed" + source.isEmpty() -> "" + else -> "other" + }) + } + + /** Seconds rounded so a drive's length can't be matched to anything: 5 s, then minutes, then 5 minutes. */ + private fun rounded(s: Long): Long = when { + s < 60 -> (s + 2) / 5 * 5 + s < 3600 -> (s + 30) / 60 * 60 + else -> (s + 150) / 300 * 300 + } /** The report as plain text, for the log file and for anything that gets sent. */ fun render(): String { @@ -145,7 +223,14 @@ object SessionReport { category == Category.NONE -> "ended before video" else -> category.label } - runCatching { UsageStats.recordSession(context, headUnitInfo, protocolVersion, outcome, carScreen, carDpi) } + runCatching { + UsageStats.recordSession(context, headUnitInfo, protocolVersion, outcome, carScreen, carDpi, sessionNote(), link, audioMode) + } + if (outcome == "connected") runCatching { + app.seb3thehacker.gearslip.stats.WorkedPrompt.onDriveWorked(context, headUnitInfo, protocolVersion, carScreen, carDpi) + app.seb3thehacker.gearslip.ui.DonatePrompt.onDriveWorked(context) + } + runCatching { app.seb3thehacker.gearslip.update.UpdateChecker.onDriveEnded(context) } } GearslipLog.flush() } diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/SessionStatus.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/SessionStatus.kt index 9f80f59..e62f285 100644 --- a/gearslip/src/main/java/app/seb3thehacker/gearslip/SessionStatus.kt +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/SessionStatus.kt @@ -20,7 +20,7 @@ object SessionStatus { Snapshot( Phase.IDLE, "Not connected", - "Plug the phone into the car's USB port. Gearslip opens by itself when the head unit connects.", + "Plug into the car's USB port, or connect wirelessly. Gearslip opens by itself when the car answers.", ), ) diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/VideoSource.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/VideoSource.kt index 8a7aa81..685ab75 100644 --- a/gearslip/src/main/java/app/seb3thehacker/gearslip/VideoSource.kt +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/VideoSource.kt @@ -23,6 +23,8 @@ class VideoSource( private val height: Int, private val frameRate: Int, private val mode: Mode = Mode.TEST_CARD, + /** Over Wi-Fi: a fraction of the cable bitrate, so the link keeps up and sound isn't starved. */ + private val wireless: Boolean = false, private val onCodecConfig: (ByteArray) -> Unit, private val onFrame: (data: ByteArray, presentationTimeUs: Long, keyFrame: Boolean) -> Unit, ) { @@ -45,7 +47,10 @@ class VideoSource( private var frameIndex = 0L fun start() { - val bitRate = (width.toLong() * height * frameRate * BITS_PER_PIXEL).toInt().coerceIn(MIN_BIT_RATE, MAX_BIT_RATE) + val level = levelFor(width, height, frameRate) + val bitRate = defaultBitRate(width, height, frameRate, level).let { + if (wireless) (it * WIRELESS_SHARE).toInt().coerceAtMost(WIRELESS_MAX_BIT_RATE) else it + } fun format(tuned: Boolean) = MediaFormat.createVideoFormat(MIME, width, height).apply { setInteger( MediaFormat.KEY_COLOR_FORMAT, @@ -74,7 +79,7 @@ class VideoSource( setFloat(MediaFormat.KEY_MAX_FPS_TO_ENCODER, frameRate.toFloat()) } setInteger(MediaFormat.KEY_PROFILE, MediaCodecInfo.CodecProfileLevel.AVCProfileBaseline) - setInteger(MediaFormat.KEY_LEVEL, MediaCodecInfo.CodecProfileLevel.AVCLevel31) + setInteger(MediaFormat.KEY_LEVEL, level.codecLevel) if (tuned) { // Hints only: a car screen wants each frame now, not the best frame a moment later. setInteger(MediaFormat.KEY_BITRATE_MODE, MediaCodecInfo.EncoderCapabilities.BITRATE_MODE_VBR) @@ -98,7 +103,7 @@ class VideoSource( codec = encoder running = true - log.i("encoder started: ${width}x$height @ ${frameRate}fps, ${bitRate / 1000} kbps, H.264 Baseline, mode=$mode") + log.i("encoder started: ${width}x$height @ ${frameRate}fps, ${bitRate / 1000} kbps${if (wireless) " (wireless)" else ""}, H.264 Baseline level ${level.label}, mode=$mode") worker = Thread { runCatching { pump(encoder) } @@ -223,10 +228,48 @@ class VideoSource( const val TIMEOUT_US = 10_000L const val SURFACE_TIMEOUT_US = 100_000L - /** Enough for sharp text and map lines; 800x480 at 30fps comes out at 4 Mbps. */ - const val BITS_PER_PIXEL = 0.3 - const val MIN_BIT_RATE = 4_000_000 - const val MAX_BIT_RATE = 12_000_000 + /** + * 800x480 at 30fps gets 8 Mbps: about the most that size can show, enough that a moving + * video in the browser stays sharp. Bigger screens get more, but less than in proportion, + * since large frames compress better: bitrate grows with pixels^0.75. 60fps gets 1.5x, + * because frames that close together are mostly alike. + */ + const val BASE_BIT_RATE = 8_000_000.0 + const val BASE_PIXELS = 800.0 * 480 + /** No car's decoder is asked for more than this, whatever its level allows. */ + const val MAX_BIT_RATE = 25_000_000 + + /** + * The H.264 levels a car screen can need, smallest first: macroblocks a second each + * decodes, and the Baseline bitrate it must handle. The encoder declares the smallest + * that fits, and the bitrate stays within it, so a car decoder isn't asked for more than + * the level it was told about. + */ + enum class Level(val label: String, val codecLevel: Int, val macroblocksPerSecond: Int, val maxBitRate: Int) { + L31("3.1", MediaCodecInfo.CodecProfileLevel.AVCLevel31, 108_000, 14_000_000), + L32("3.2", MediaCodecInfo.CodecProfileLevel.AVCLevel32, 216_000, 20_000_000), + L4("4", MediaCodecInfo.CodecProfileLevel.AVCLevel4, 245_760, 20_000_000), + L42("4.2", MediaCodecInfo.CodecProfileLevel.AVCLevel42, 522_240, 50_000_000), + } + + fun levelFor(width: Int, height: Int, frameRate: Int): Level { + val perSecond = ((width + 15) / 16) * ((height + 15) / 16) * frameRate + return Level.entries.firstOrNull { perSecond <= it.macroblocksPerSecond } ?: Level.L42 + } + + fun defaultBitRate(width: Int, height: Int, frameRate: Int, level: Level): Int { + val size = Math.pow(width.toDouble() * height / BASE_PIXELS, 0.75) + val rate = if (frameRate > 30) 1.5 else 1.0 + return (BASE_BIT_RATE * size * rate).toInt().coerceAtMost(minOf(level.maxBitRate, MAX_BIT_RATE)) + } + /** + * A car's Wi-Fi is shared, noisy and far slower than its nominal rate, and the sound rides + * the same link behind the video. Real Android Auto sends a few Mbps wirelessly; at the + * cable rate (15-20 Mbps for 720p) a laptop hotspot fell seconds behind and the music + * cut in and out. 720p30 comes out near 5.4 Mbps. + */ + const val WIRELESS_SHARE = 0.35 + const val WIRELESS_MAX_BIT_RATE = 8_000_000 const val I_FRAME_INTERVAL_S = 5 private const val REPEAT_AFTER_US = 100_000L diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/audio/AacEncoder.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/audio/AacEncoder.kt new file mode 100644 index 0000000..f80ae92 --- /dev/null +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/audio/AacEncoder.kt @@ -0,0 +1,94 @@ +package app.seb3thehacker.gearslip.audio + +import android.media.MediaCodec +import android.media.MediaCodecInfo +import android.media.MediaFormat + +/** + * Turns captured PCM into AAC-LC for a car whose music channel takes nothing else. Cars that + * offer PCM never get here: AAC costs a little delay and quality, and every car that works today + * is a PCM car. + * + * Android's encoder hands back raw access units, which is what AUDIO_AAC_LC expects. For + * AUDIO_AAC_LC_ADTS each unit gets the 7-byte ADTS header in front, so the car can find the + * frame edges itself. The encoder's first output is its configuration, not audio, and is skipped. + */ +internal class AacEncoder(private val sampleRate: Int, private val channels: Int, private val adts: Boolean) { + + private val codec: MediaCodec = MediaCodec.createEncoderByType(MediaFormat.MIMETYPE_AUDIO_AAC).apply { + val format = MediaFormat.createAudioFormat(MediaFormat.MIMETYPE_AUDIO_AAC, sampleRate, channels).apply { + setInteger(MediaFormat.KEY_AAC_PROFILE, MediaCodecInfo.CodecProfileLevel.AACObjectLC) + setInteger(MediaFormat.KEY_BIT_RATE, if (channels == 1) 96_000 else 192_000) + setInteger(MediaFormat.KEY_MAX_INPUT_SIZE, 16 * 1024) + } + configure(format, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE) + start() + } + private val info = MediaCodec.BufferInfo() + private var samplesIn = 0L + + /** Encodes [size] bytes of 16-bit PCM and returns the AAC frames that came out, oldest first. */ + fun encode(pcm: ByteArray, size: Int): List { + var offset = 0 + val out = ArrayList() + while (offset < size) { + val index = codec.dequeueInputBuffer(INPUT_WAIT_US) + if (index < 0) { + drain(out) + continue + } + val buffer = codec.getInputBuffer(index) ?: continue + val n = minOf(buffer.remaining(), size - offset) + buffer.put(pcm, offset, n) + val presentationUs = samplesIn * 1_000_000L / sampleRate + codec.queueInputBuffer(index, 0, n, presentationUs, 0) + samplesIn += n / (2 * channels) + offset += n + drain(out) + } + return out + } + + private fun drain(out: MutableList) { + while (true) { + val index = codec.dequeueOutputBuffer(info, 0) + if (index < 0) return + if (info.flags and MediaCodec.BUFFER_FLAG_CODEC_CONFIG == 0 && info.size > 0) { + val buffer = codec.getOutputBuffer(index) + if (buffer != null) { + val frame = ByteArray(info.size) + buffer.position(info.offset) + buffer.get(frame) + out.add(if (adts) adtsHeader(frame.size) + frame else frame) + } + } + codec.releaseOutputBuffer(index, false) + } + } + + /** ADTS for one AAC-LC frame: sync word, no CRC, and the frame's length including this header. */ + private fun adtsHeader(payload: Int): ByteArray { + val length = payload + 7 + val freq = FREQUENCIES.indexOf(sampleRate).let { if (it < 0) 3 else it } + val profile = 1 // AAC-LC is object type 2, written as type minus one + return byteArrayOf( + 0xFF.toByte(), + 0xF1.toByte(), + ((profile shl 6) or (freq shl 2) or (channels shr 2)).toByte(), + (((channels and 3) shl 6) or (length shr 11)).toByte(), + ((length shr 3) and 0xFF).toByte(), + (((length and 7) shl 5) or 0x1F).toByte(), + 0xFC.toByte(), + ) + } + + fun release() { + runCatching { codec.stop() } + runCatching { codec.release() } + } + + private companion object { + const val INPUT_WAIT_US = 10_000L + val FREQUENCIES = listOf(96000, 88200, 64000, 48000, 44100, 32000, 24000, 22050, 16000, 12000, 11025, 8000, 7350) + } +} diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/audio/AudioCaptureService.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/audio/AudioCaptureService.kt index cbdb536..6948015 100644 --- a/gearslip/src/main/java/app/seb3thehacker/gearslip/audio/AudioCaptureService.kt +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/audio/AudioCaptureService.kt @@ -16,6 +16,7 @@ import android.media.AudioRecord import android.media.projection.MediaProjection import android.media.projection.MediaProjectionManager import android.os.IBinder +import android.os.Process import app.seb3thehacker.gearslip.GearslipLog import kotlin.math.abs @@ -147,6 +148,9 @@ class AudioCaptureService : Service() { reading = true thread = Thread({ + // A busy phone (a map app drawing, the video encoder) would otherwise starve this + // loop, and the car hears every late read as a skip. + Process.setThreadPriority(Process.THREAD_PRIORITY_URGENT_AUDIO) val buffer = ByteArray(CHUNK_BYTES) var peak = 0 var heardAt = System.currentTimeMillis() diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/audio/AudioLink.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/audio/AudioLink.kt index 46fdeb9..e398b6d 100644 --- a/gearslip/src/main/java/app/seb3thehacker/gearslip/audio/AudioLink.kt +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/audio/AudioLink.kt @@ -22,6 +22,15 @@ class AudioLink( private val service: ServiceDiscovery.AudioService, private val sendOnChannel: (messageId: Int, body: ByteArray) -> Unit, private val sendControl: (messageId: Int, body: ByteArray) -> Unit, + /** Log name: "audio" for the media sink, "guidance" for the directions sink. */ + private val tag: String = "audio", + private val sessionId: Int = SESSION_ID, + /** Media asks the car for focus; directions just start, and the car ducks music under them. */ + private val asksFocus: Boolean = true, + private val onReady: (AudioLink) -> Unit = { CarAudio.attach(it) }, + private val onClose: (AudioLink) -> Unit = { CarAudio.detach(it) }, + /** The car took the sound back after giving it, as when the driver switches to the radio. */ + private val onTakenBack: () -> Unit = {}, ) : CarAudio.Link { private var configIndex = 0 @@ -33,11 +42,15 @@ class AudioLink( // The Uconnect asks for 4 on its video channel; assume the same until its Config says otherwise. @Volatile private var maxUnacked = 4 @Volatile private var lastAckAt = 0L + private var stalls = 0 private var chunks = 0 private var bytes = 0L private var dropped = 0 @Volatile private var configTimer: Thread? = null private val ackLock = Object() + /** Only for a sink that takes AAC and nothing else; PCM sinks get the capture as it is. */ + @Volatile private var aac: AacEncoder? = null + private val aacSink get() = service.codecType == ServiceDiscovery.CODEC_AAC_LC || service.codecType == ServiceDiscovery.CODEC_AAC_LC_ADTS private val config get() = service.configs.getOrNull(configIndex) ?: service.configs.firstOrNull() override val sampleRate get() = config?.sampleRate ?: 48_000 @@ -46,28 +59,29 @@ class AudioLink( val channelId get() = service.serviceId fun opened(status: Int) { - GearslipLog.i("audio: <- ChannelOpenResponse (media audio): status=$status") + GearslipLog.i("$tag: <- ChannelOpenResponse: status=$status") if (status != 0) { - GearslipLog.w("audio: the head unit refused the media audio channel (status $status)") + GearslipLog.w("$tag: the head unit refused the $tag channel (status $status)") return } - // AVChannelSetupRequest { required uint32 config_index = 1 }. This field is an index into - // the sink's own audio_configs, not a codec id - sending the codec type asks for a config - // the sink does not have, and it answers with a status that never reaches READY. - sendOnChannel(MSG_SETUP, Protobuf.varintField(1, 0L)) - GearslipLog.i("audio: -> Setup(config_index=0) on channel ${service.serviceId}, codec ${service.codecName}") + // Setup { required MediaCodecType type = 1 }: the codec the car advertised for this sink + // (aasdk's current protos, and LIVI reads it as the codec). Gearslip used to send 0 here, + // read from an older proto as a config index; 0 is no codec at all, and the 2024 VW + // answered UNEXPECTED_MESSAGE (id 255), never acked a chunk and played nothing. + sendOnChannel(MSG_SETUP, Protobuf.varintField(1, service.codecType.toLong())) + GearslipLog.i("$tag: -> Setup(codec=${service.codecName}) on channel ${service.serviceId}") armConfigFallback() } /** - * The 2018 Uconnect answers our Setup with a two-byte message we cannot identify (id 255, no - * body) rather than the AVChannelSetupResponse the video channel gets, so the sink never - * reaches READY and no audio is ever sent. + * The 2018 Uconnect, a Dacia ULC 4.5 and the 2024 VW answered the old Setup (codec 0) with + * UNEXPECTED_MESSAGE (id 255, no body) rather than a Config, so the sink never reached READY. + * [MSG_SETUP_ANSWER] still takes that as the go-ahead, in case a car rejects the real codec too. * - * Service discovery already told us everything that message would have: the sink is PCM at - * 48000Hz stereo and it has exactly one configuration. So rather than wait forever for a - * message that is not coming, assume the advertised configuration and carry on. If that guess - * is wrong the head unit simply will not ack, which the write path already reports. + * Service discovery already told us everything a Config would have: the sink's rate, channels + * and its one configuration. So for a car that answers nothing at all, wait a moment and then + * assume the advertised configuration. If that guess is wrong the head unit simply will not + * ack, which the write path already reports. */ private fun armConfigFallback() { val timer = Thread({ @@ -76,19 +90,25 @@ class AudioLink( } catch (e: InterruptedException) { return@Thread } - if (ready) return@Thread - GearslipLog.w( - "audio: no usable Config after ${CONFIG_WAIT_MS}ms - falling back to the sink the " + - "head unit advertised (${sampleRate}Hz x$channels, config_index=0, max_unacked=$maxUnacked)", - ) - configIndex = 0 - ready = true - CarAudio.attach(this) + useAdvertisedSink("no usable Config after ${CONFIG_WAIT_MS}ms") }, "gearslip-audio-config").apply { isDaemon = true } configTimer = timer timer.start() } + @Synchronized + private fun useAdvertisedSink(why: String) { + if (ready) return + configTimer?.interrupt() + GearslipLog.i( + "$tag: $why - using the sink the head unit advertised " + + "(${sampleRate}Hz x$channels, config_index=0, max_unacked=$maxUnacked)", + ) + configIndex = 0 + ready = true + onReady(this) + } + fun onMessage(messageId: Int, body: ByteArray) { when (messageId) { MSG_CHANNEL_OPEN_RESPONSE -> opened(Protobuf.readInt32Field(body, 1) ?: 0) @@ -98,17 +118,17 @@ class AudioLink( val status = Wire.varint(fields, 1)?.toInt() val window = Wire.varint(fields, 2)?.toInt() val indices = fields.filter { it.number == 3 && it.wireType == 0 }.map { it.varint.toInt() } - GearslipLog.i("audio: <- Config status=$status max_unacked=$window indices=$indices") + GearslipLog.i("$tag: <- Config status=$status max_unacked=$window indices=$indices") if (window != null) maxUnacked = maxOf(1, window) configIndex = indices.firstOrNull() ?: 0 if (status == STATUS_READY) { configTimer?.interrupt() ready = true - GearslipLog.i("audio: sink ready - ${sampleRate}Hz x$channels ${config?.bits}bit") - CarAudio.attach(this) + GearslipLog.i("$tag: sink ready - ${sampleRate}Hz x$channels ${config?.bits}bit") + onReady(this) } else { GearslipLog.w( - "audio: the sink is not ready (status=$status, 1=WAIT 2=READY) - " + + "$tag: the sink is not ready (status=$status, 1=WAIT 2=READY) - " + "nothing will be sent until it is", ) } @@ -123,9 +143,11 @@ class AudioLink( } } - // Whatever the head unit answers Setup with, we need its bytes to identify it. - else -> { - GearslipLog.i("audio: <- unhandled message id=$messageId (${body.size} bytes)") + // Whatever else the head unit answers Setup with, we need its bytes to identify it. + else -> if (messageId == MSG_SETUP_ANSWER && body.isEmpty()) { + useAdvertisedSink("<- empty Setup answer (id $messageId)") + } else { + GearslipLog.i("$tag: <- unhandled message id=$messageId (${body.size} bytes)") GearslipLog.hex(" audio msg", body, limit = 64) } } @@ -138,17 +160,21 @@ class AudioLink( */ fun onFocus(state: Int) { val name = FOCUS_STATES[state] ?: "unknown($state)" - GearslipLog.i("audio: <- AudioFocusResponse state=$name") - focusGranted = state == FOCUS_STATE_GAIN || state == FOCUS_STATE_GAIN_TRANSIENT + GearslipLog.i("$tag: <- AudioFocusResponse state=$name") + val had = focusGranted + focusGranted = state == FOCUS_STATE_GAIN || state == FOCUS_STATE_GAIN_TRANSIENT || + state == FOCUS_STATE_GAIN_MEDIA_ONLY if (focusGranted) startStream() else stop() + // A ducking request leaves the music playing; a loss means another source has the speakers. + if (had && (state == FOCUS_STATE_LOSS || state == FOCUS_STATE_LOSS_TRANSIENT)) onTakenBack() } override fun start() { if (!ready || started) return - if (!focusRequested) { + if (asksFocus && !focusRequested) { focusRequested = true sendControl(MSG_FOCUS_REQUEST, Protobuf.varintField(1, FOCUS_GAIN.toLong())) - GearslipLog.i("audio: -> AudioFocusRequest(GAIN); waiting for the head unit to hand the sink over") + GearslipLog.i("$tag: -> AudioFocusRequest(GAIN); waiting for the head unit to hand the sink over") } // Some head units grant the sink silently and never answer. Do not let that be the end of // it: start anyway, and let the acks say whether anything is being taken. @@ -159,12 +185,18 @@ class AudioLink( if (!ready || started) return sendOnChannel( MSG_START, - Protobuf.varintField(1, SESSION_ID.toLong()) + Protobuf.varintField(2, configIndex.toLong()), + Protobuf.varintField(1, sessionId.toLong()) + Protobuf.varintField(2, configIndex.toLong()), ) inFlight = 0 lastAckAt = System.currentTimeMillis() + if (aacSink && aac == null) { + aac = runCatching { AacEncoder(sampleRate, channels, adts = service.codecType == ServiceDiscovery.CODEC_AAC_LC_ADTS) } + .onSuccess { GearslipLog.i("$tag: the car takes ${service.codecName} only - encoding to AAC") } + .onFailure { GearslipLog.e("$tag: couldn't start the AAC encoder", it) } + .getOrNull() + } started = true - GearslipLog.i("audio: -> Start(session=$SESSION_ID, config=$configIndex) - streaming ${sampleRate}Hz x$channels") + GearslipLog.i("$tag: -> Start(session=$sessionId, config=$configIndex) - streaming ${sampleRate}Hz x$channels") } override fun stop() { @@ -172,18 +204,39 @@ class AudioLink( started = false synchronized(ackLock) { ackLock.notifyAll() } runCatching { - sendOnChannel(MSG_STOP, Protobuf.varintField(1, SESSION_ID.toLong())) + sendOnChannel(MSG_STOP, Protobuf.varintField(1, sessionId.toLong())) if (focusRequested) sendControl(MSG_FOCUS_REQUEST, Protobuf.varintField(1, FOCUS_RELEASE.toLong())) } focusRequested = false focusGranted = false - GearslipLog.i("audio: -> Stop, AudioFocusRequest(RELEASE) after $chunks chunks / $bytes bytes") + aac?.release() + aac = null + GearslipLog.i("$tag: -> Stop, AudioFocusRequest(RELEASE) after $chunks chunks / $bytes bytes") chunks = 0 bytes = 0 } override fun write(pcm: ByteArray, size: Int) { if (!started) return + val encoder = aac + if (encoder == null) { + send(pcm.copyOf(size)) + return + } + val frames = try { + encoder.encode(pcm, size) + } catch (e: Exception) { + // A codec that dies mid-drive leaves the car silent, as it was before AAC; log it once. + GearslipLog.e("$tag: AAC encoding failed - no more audio this drive", e) + aac = null + started = false + return + } + frames.forEach(::send) + } + + /** One Data message, once the car has room for it. */ + private fun send(payload: ByteArray) { // Wait for the car to catch up rather than throw the chunk away: a dropped chunk is a gap // the driver hears, while a short wait is soaked up by the capture buffer behind us. synchronized(ackLock) { @@ -191,7 +244,11 @@ class AudioLink( val waited = System.currentTimeMillis() - lastAckAt if (waited > STALL_MS) { // A lost ack must not silence the stream for good; after a stall assume the window cleared. - GearslipLog.w("audio: no ack for ${STALL_MS}ms with $inFlight in flight - reopening the window") + // A car that never acks (the 2024 VW, the Bolt) hits this every chunk: say so + // once, then now and then, so the rest of the log stays readable. + if (stalls++ % 500 == 0) { + GearslipLog.w("$tag: no ack for ${STALL_MS}ms with $inFlight in flight - reopening the window ($stalls times)") + } inFlight = 0 break } @@ -206,20 +263,20 @@ class AudioLink( val micros = System.nanoTime() / 1_000 for (i in 0 until 8) timestamp[i] = ((micros shr ((7 - i) * 8)) and 0xFF).toByte() try { - sendOnChannel(MSG_DATA, timestamp + pcm.copyOf(size)) + sendOnChannel(MSG_DATA, timestamp + payload) } catch (e: java.io.IOException) { - GearslipLog.e("audio: the link died mid-stream", e) + GearslipLog.e("$tag: the link died mid-stream", e) started = false return } synchronized(ackLock) { inFlight++ } chunks++ - bytes += size + bytes += payload.size // The first chunk proves the whole path; after that, an occasional line is enough to show // the car is still taking audio and to catch a window that has quietly closed. if (chunks == 1 || chunks % CHUNK_LOG_EVERY == 0) { GearslipLog.i( - "audio: -> $chunks chunks / $bytes bytes, inFlight=$inFlight/$maxUnacked, " + + "$tag: -> $chunks chunks / $bytes bytes, inFlight=$inFlight/$maxUnacked, " + "dropped=$dropped, focus=${if (focusGranted) "granted" else "not granted"}", ) } @@ -230,7 +287,7 @@ class AudioLink( configTimer = null stop() ready = false - CarAudio.detach(this) + onClose(this) } companion object { @@ -243,18 +300,24 @@ class AudioLink( private const val MSG_STOP = 32770 private const val MSG_CONFIG = 32771 private const val MSG_ACK = 32772 + private const val MSG_SETUP_ANSWER = 255 private const val STATUS_READY = 2 private const val FOCUS_GAIN = 1 private const val FOCUS_RELEASE = 4 private const val FOCUS_STATE_GAIN = 1 private const val FOCUS_STATE_GAIN_TRANSIENT = 2 + private const val FOCUS_STATE_GAIN_MEDIA_ONLY = 6 + private const val FOCUS_STATE_LOSS = 3 + private const val FOCUS_STATE_LOSS_TRANSIENT = 5 private const val SESSION_ID = 2 private const val STALL_MS = 2_000L private const val CHUNK_LOG_EVERY = 200 // about every four seconds - private const val CONFIG_WAIT_MS = 2_500L + private const val CONFIG_WAIT_MS = 1_000L + // AudioFocusStateNotification.AudioFocusState in aasdk. private val FOCUS_STATES = mapOf( - 1 to "GAIN", 2 to "GAIN_TRANSIENT", 3 to "LOSS", 4 to "LOSS_TRANSIENT", + 1 to "GAIN", 2 to "GAIN_TRANSIENT", 3 to "LOSS", 4 to "LOSS_TRANSIENT_CAN_DUCK", + 5 to "LOSS_TRANSIENT", 6 to "GAIN_MEDIA_ONLY", 7 to "GAIN_TRANSIENT_GUIDANCE_ONLY", ) } } diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/audio/CarAudio.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/audio/CarAudio.kt index 13a0073..21c6cf2 100644 --- a/gearslip/src/main/java/app/seb3thehacker/gearslip/audio/CarAudio.kt +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/audio/CarAudio.kt @@ -100,6 +100,26 @@ object CarAudio { _capture.value = Capture.DENIED } + private val _micBlocked = MutableStateFlow(false) + + /** The microphone permission is off for good: Android refuses without asking, so say why. */ + val micBlocked: StateFlow = _micBlocked.asStateFlow() + + fun micBlocked() { + _micBlocked.value = true + denied() + } + + /** The driver turned the permission back on mid-drive: ask for capture again. */ + fun micAllowed() { + if (!_micBlocked.value) return + _micBlocked.value = false + if (_capture.value == Capture.DENIED) { + _capture.value = Capture.OFF + if (wanted) request() + } + } + internal fun captureStarted() { _capture.value = Capture.ON link?.start() diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/audio/CarMic.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/audio/CarMic.kt index 1236601..c3ee465 100644 --- a/gearslip/src/main/java/app/seb3thehacker/gearslip/audio/CarMic.kt +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/audio/CarMic.kt @@ -12,17 +12,21 @@ import java.util.concurrent.TimeUnit * * -> CHANNEL_OPEN_REQUEST <- CHANNEL_OPEN_RESPONSE * -> Setup { config_index } <- Config + * -> VoiceSessionNotification { START } (control channel) * -> MicrophoneRequest { open } <- MicrophoneResponse { status, session_id } * <- Data { 8-byte timestamp, PCM }... -> Ack { session_id, ack } * -> MicrophoneRequest { open = false } + * -> VoiceSessionNotification { END } * - * The channel opens with the session, but the car only records between [start] and [stop]. + * The channel opens with the session, but the car only records between [start] and [stop], and + * only inside a voice session: the 2024 VW answered a bare MicrophoneRequest with status -7. * Speech wants 16 kHz mono; a car that records at a multiple of that is averaged down to it, and * one that doesn't isn't used, so voice falls back to the phone's microphone. */ class CarMic( private val service: ServiceDiscovery.AudioService, private val sendOnChannel: (messageId: Int, body: ByteArray) -> Unit, + private val voiceSession: (start: Boolean) -> Unit, ) { val channelId get() = service.serviceId @@ -43,7 +47,8 @@ class CarMic( val status = Protobuf.readInt32Field(body, 1) ?: 0 GearslipLog.i("mic: <- ChannelOpenResponse: status=$status") if (status != 0) return - sendOnChannel(MSG_SETUP, Protobuf.varintField(1, 0L)) + // Setup { type = the codec }, as for audio: 0 got UNEXPECTED_MESSAGE from the VW. + sendOnChannel(MSG_SETUP, Protobuf.varintField(1, service.codecType.toLong())) ready = usable if (!usable) { GearslipLog.w( @@ -97,6 +102,7 @@ class CarMic( chunks.clear() packets = 0 listening = true + voiceSession(true) sendOnChannel(MSG_MIC_REQUEST, Protobuf.varintField(1, 1L) + Protobuf.varintField(4, 1L)) GearslipLog.i("mic: -> MicrophoneRequest(open)") return true @@ -109,7 +115,8 @@ class CarMic( if (!listening) return listening = false sendOnChannel(MSG_MIC_REQUEST, Protobuf.varintField(1, 0L)) - GearslipLog.i("mic: -> MicrophoneRequest(close) after $packets packets") + voiceSession(false) + GearslipLog.i("mic: -> MicrophoneRequest(close) after $packets packets, voice session ended") } companion object { diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/car/AssistantCommands.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/car/AssistantCommands.kt index 12cd633..49d233f 100644 --- a/gearslip/src/main/java/app/seb3thehacker/gearslip/car/AssistantCommands.kt +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/car/AssistantCommands.kt @@ -1,10 +1,14 @@ package app.seb3thehacker.gearslip.car import android.content.Context +import app.seb3thehacker.gearslip.media.CarMedia +import app.seb3thehacker.gearslip.speech.VoiceFocus import java.text.DateFormat import java.util.Date import kotlinx.coroutines.Dispatchers +import kotlinx.coroutines.delay import kotlinx.coroutines.withContext +import kotlinx.coroutines.withTimeoutOrNull /** * Turns what [CarAssistant] heard into an action plus something to say back. @@ -17,9 +21,16 @@ import kotlinx.coroutines.withContext object AssistantCommands { suspend fun handle(context: Context, navigator: CarNavigator, frame: CarEnvironment.Frame, heardRaw: String): String { - val heard = heardRaw.lowercase().trim() + val heard = natural(heardRaw) + VoiceHelp.hide() + if (heard in HELP_WORDS || heard.startsWith("what can i say") || heard.startsWith("what can you do")) { + VoiceHelp.show() + return "Here's what you can say." + } + if (heard.matches(Regex("(fuck|f\\*+) (you|off)( too)?"))) return "Fuck you too." navigateCommand(context, navigator, frame, heard)?.let { return it } + playSearchCommand(context, navigator, heard)?.let { return it } mediaCommand(heard)?.let { return it } navCommand(navigator, heard)?.let { return it } infoCommand(heard)?.let { return it } @@ -29,6 +40,47 @@ object AssistantCommands { return "I don't know how to do that yet." } + /** + * What was said, minus the padding people wrap a command in: "hey", "can you", "I want to", + * "please", "thanks". Each command below then only has to know its own words. + */ + private fun natural(raw: String): String { + var t = raw.lowercase().replace(Regex("[.,!?]"), " ").replace(Regex("\\s+"), " ").trim() + var changed = true + while (changed) { + changed = false + for (lead in LEAD_FILLERS) if (t.startsWith("$lead ")) { t = t.removePrefix("$lead ").trim(); changed = true } + for (tail in TAIL_FILLERS) if (t.endsWith(" $tail")) { t = t.removeSuffix(" $tail").trim(); changed = true } + } + return t + } + + /** The part after the first of [triggers] that [text] starts with, or null. */ + private fun after(text: String, triggers: List): String? = + triggers.firstOrNull { text.startsWith(it) }?.let { text.removePrefix(it).trim() } + + private suspend fun allEntries(context: Context): List { + val vehicleData = CarSettings.experimentalFeaturesEnabled.value && CarSettings.carSensorsEnabled.value + val builtIn = BuiltInApps.entries.filter { it.builtIn !== BuiltInApps.VehicleData || vehicleData } + return builtIn + (LauncherCache.entries ?: withContext(Dispatchers.IO) { LauncherCache.load(context) }) + } + + /** + * Best label match: exact, then a label starting with [name], then one containing it. Spaces + * and punctuation don't count, since the recognizer may hear "MapQuest" as "map quest". + */ + private fun bestMatch(items: List, name: String, label: (T) -> String): T? { + val n = squash(name.removePrefix("the ").removePrefix("my ").removeSuffix(" app")) + if (n.isEmpty()) return null + items.firstOrNull { squash(label(it)) == n }?.let { return it } + // Loose matches need a real word: "pizza in a" mustn't pick "Organic Maps" for its "a". + if (n.length < 3) return null + return items.firstOrNull { squash(label(it)).startsWith(n) } + ?: items.firstOrNull { squash(label(it)).contains(n) } + } + + private fun squash(s: String) = s.lowercase().filter { it.isLetterOrDigit() } + /** * "navigate to X [with|using|on|via Y]" - Y picks the nav app by (partial) label; without it, * whichever nav app was used last, or the only one installed, or the first one found. Starts @@ -42,29 +94,49 @@ object AssistantCommands { frame: CarEnvironment.Frame, heard: String, ): String? { - val trigger = NAVIGATE_TRIGGERS.firstOrNull { heard.startsWith(it) } ?: return null - var destination = heard.removePrefix(trigger).trim() + // "use Organic Maps to get to X", "open OsmAnd and navigate to X": the app comes first. + var text = heard + var leadingApp: String? = null + Regex("^(?:use|using|with|open|in|on) (.+?) (?:to |and )(.+)$").find(text)?.let { m -> + if (after(m.groupValues[2], NAVIGATE_TRIGGERS) != null) { + leadingApp = m.groupValues[1] + text = m.groupValues[2] + } + } + var destination = after(text, NAVIGATE_TRIGGERS) ?: return null if (destination.isBlank()) return "Where to?" - var appName: String? = null - for (sep in listOf(" with ", " using ", " on ", " via ")) { - val at = destination.lastIndexOf(sep) - if (at > 0) { - appName = destination.substring(at + sep.length).trim() + val entries = allEntries(context) + // "go to Spotify" means the app, and "go to settings" the screen; "go to Target" is a place. + val asApp = destination.removePrefix("the ").removeSuffix(" app") + if (text.startsWith("go to ") && + (asApp in SCREEN_WORDS || asApp in APP_NICKNAMES || entries.any { it.label.lowercase() == asApp }) + ) { + return null + } + val navApps = entries.filter { it.template?.isNavigation == true } + if (navApps.isEmpty()) return "You don't have any navigation apps installed." + + // "X with Organic Maps", "X on OsmAnd": split only when the end names a nav app, so a + // place like "Starbucks on Main Street" stays whole. + var chosenApp = leadingApp?.let { name -> bestMatch(navApps, name) { it.label } ?: return "I couldn't find a navigation app called $name." } + if (chosenApp == null) { + for (sep in listOf(" with ", " using ", " on ", " via ", " in ", " through ")) { + val at = destination.lastIndexOf(sep) + if (at <= 0) continue + val app = bestMatch(navApps, destination.substring(at + sep.length)) { it.label } ?: continue + chosenApp = app destination = destination.substring(0, at).trim() break } } + // "the nearest gas station" -> "gas station": the map app already searches around the car + // and AutoRoute picks the nearest result, and Organic Maps took "nearest" as part of the name. + val (place, nearby) = plainPlace(destination) + destination = place if (destination.isBlank()) return "Where to?" - val entries = LauncherCache.entries ?: withContext(Dispatchers.IO) { LauncherCache.load(context) } - val navApps = entries.filter { it.template?.isNavigation == true } - if (navApps.isEmpty()) return "You don't have any navigation apps installed." - - val requestedApp = appName - val chosen = when { - requestedApp != null -> navApps.firstOrNull { it.label.lowercase().contains(requestedApp) } - ?: return "I couldn't find a navigation app called $requestedApp." + val chosen = chosenApp ?: when { navApps.size == 1 -> navApps.first() else -> navApps.firstOrNull { it.componentId == CarSettings.lastNav.value } ?: navApps.first() } @@ -72,29 +144,104 @@ object AssistantCommands { CarServices.connectNav(template, frame, destination) navigator.home() - return "Navigating to $destination with ${chosen.label}." + return if (nearby) "Finding the nearest $destination with ${chosen.label}." else "Navigating to $destination with ${chosen.label}." + } + + /** [destination] without "a", "the nearest", "near me" and the like, and whether any of that said "close by". */ + private fun plainPlace(destination: String): Pair { + var t = destination + var nearby = false + var changed = true + while (changed) { + changed = false + for (lead in PLACE_LEADS) if (t.startsWith(lead)) { + t = t.removePrefix(lead).trim(); changed = true + if (lead.trim() !in ARTICLES) nearby = true + } + for (tail in PLACE_TAILS) if (t.endsWith(tail)) { t = t.removeSuffix(tail).trim(); changed = true; nearby = true } + } + return t to nearby + } + + /** + * "play X" with something after it: a song, artist, album or playlist, asked of the music app + * that's open through its own search ([CarMedia.search]). A bare "play", or "play music", + * falls through to resume. + */ + private suspend fun playSearchCommand(context: Context, navigator: CarNavigator, heard: String): String? { + var query = after(heard, PLAY_TRIGGERS) ?: return null + query = query.removePrefix("some ").trim() + if (query.isEmpty() || query in RESUME_WORDS) return null + val media = CarServices.media + + // "play Daft Punk on Metrolist": switch to that app first, if it's one we know. + for (sep in listOf(" on ", " in ", " from ", " with ", " using ")) { + val at = query.lastIndexOf(sep) + if (at <= 0) continue + val apps = withContext(Dispatchers.IO) { app.seb3thehacker.gearslip.media.MediaCatalog.installed(context) } + val wanted = bestMatch(apps, query.substring(at + sep.length)) { it.label } ?: continue + query = query.substring(0, at).trim() + if (query.isEmpty() || query in RESUME_WORDS) { + CarServices.openMedia(wanted, autoplay = true) + navigator.media() + return "Opening ${wanted.label}." + } + if (CarServices.mediaApp.value?.component != wanted.component) { + CarServices.openMedia(wanted) + navigator.media() + // Give it a moment to connect before asking it to search. + withTimeoutOrNull(APP_CONNECT_MS) { + while (media.appLabel == null || (!media.canSearchLibrary && !media.now.value.canSearch)) delay(200) + } + } + break + } + + // No music app open, or its connection is gone (the app was closed): use the one played + // last, the way "play X" works on Android Auto. + val phase = media.phase.value + if (media.appLabel == null || phase == CarMedia.Phase.IDLE || phase == CarMedia.Phase.REJECTED) { + if (!CarServices.reopenLastMedia()) return "Open a music app first, then ask again." + navigator.media() + } + withTimeoutOrNull(APP_CONNECT_MS) { + while (media.phase.value != CarMedia.Phase.READY || media.appLabel == null || + (!media.canSearchLibrary && !media.now.value.canSearch) + ) delay(200) + } + val app = media.appLabel ?: return "Your music app didn't open in time. Try again." + if (!media.canSearchLibrary && !media.now.value.canSearch) { + return "$app ignores voice searches. Pick something on its screen instead." + } + media.playBestMatch(query) + return "Looking for $query on $app." } private fun mediaCommand(heard: String): String? { val media = CarServices.media return when { - "pause" in heard || heard.startsWith("stop") -> { - if (media.now.value.playing) media.togglePlay() + "pause" in heard || heard.startsWith("stop") || heard == "quiet" || heard == "shut up" -> { + media.pause() + VoiceFocus.afterRelease { media.pause() } "Paused." } - "resume" in heard || (heard.startsWith("play") && "playing" !in heard) -> { + "resume" in heard || "unpause" in heard || heard in RESUME_PHRASES || + (heard.startsWith("play") && "playing" !in heard) -> { if (!media.now.value.playing) media.togglePlay() "Playing." } - "next" in heard || "skip" in heard -> { + "next" in heard || "skip" in heard || heard.startsWith("change the song") -> { media.next() "Skipping." } - "previous" in heard || "last song" in heard || "back song" in heard -> { + "previous" in heard || "last song" in heard || "back song" in heard || "go back a song" in heard || + "play that again" in heard || "replay" in heard -> { media.previous() "Playing the previous track." } - "what" in heard && "playing" in heard -> { + ("what" in heard && "playing" in heard) || "what song" in heard || "who sings" in heard || + "who's singing" in heard || "who is singing" in heard || "what's this song" in heard || + "name of this song" in heard -> { val now = media.now.value if (now.isActive) listOf(now.title, now.artist).filter { it.isNotBlank() }.joinToString(" by ") else "Nothing's playing." @@ -104,11 +251,19 @@ object AssistantCommands { } private fun navCommand(navigator: CarNavigator, heard: String): String? = when { - "go home" in heard || heard == "home" -> { + heard == "go back" || heard == "back" -> { + navigator.back() + "Going back." + } + heard in EXIT_WORDS -> { + BuiltInApps.Exit.open(navigator) + "Back to the car's screen." + } + "go home" in heard || heard == "home" || heard == "home screen" || "go to the home screen" in heard -> { navigator.home() "Heading home." } - "open apps" in heard || heard == "apps" || "app list" in heard -> { + "open apps" in heard || heard == "apps" || "app list" in heard || "all apps" in heard || "my apps" in heard || heard == "go to apps" -> { navigator.apps() "Here are your apps." } @@ -128,9 +283,10 @@ object AssistantCommands { } private fun infoCommand(heard: String): String? = when { - "what time" in heard || "what's the time" in heard -> + "what time" in heard || "what's the time" in heard || heard == "time" || "the time" in heard -> "It's ${DateFormat.getTimeInstance(DateFormat.SHORT).format(Date())}." - "weather" in heard -> { + "weather" in heard || "going to rain" in heard || "gonna rain" in heard || "how hot" in heard || + "how cold" in heard || "temperature outside" in heard -> { val data = (Weather.state.value as? WeatherState.Ready)?.data if (data != null) "${data.temp}${data.tempUnit} and ${Weather.describe(data.code)} in ${data.place}." else "I don't have the weather yet." @@ -139,12 +295,11 @@ object AssistantCommands { } private suspend fun callCommand(context: Context, heard: String): String? { - val name = when { - heard.startsWith("call ") -> heard.removePrefix("call ") - heard.startsWith("phone ") -> heard.removePrefix("phone ") - heard.startsWith("dial ") -> heard.removePrefix("dial ") - else -> return null - }.trim() + val name = ( + after(heard, CALL_TRIGGERS) + ?: Regex("^give (.+) a (?:call|ring)$").find(heard)?.groupValues?.get(1) + ?: return null + ).replace(Regex(" (?:on|at) (?:their |his |her )?(?:mobile|cell|phone|home|work)$"), "").trim() if (name.isBlank()) return "Who should I call?" if (!hasCallPermissions(context)) return "Gearslip needs the phone and contacts permissions first, from the phone screen." @@ -162,21 +317,69 @@ object AssistantCommands { frame: CarEnvironment.Frame, heard: String, ): String? { - val name = when { - heard.startsWith("open ") -> heard.removePrefix("open ") - heard.startsWith("launch ") -> heard.removePrefix("launch ") - heard.startsWith("start ") -> heard.removePrefix("start ") - else -> return null - }.trim() + val name = (after(heard, OPEN_TRIGGERS) ?: return null) + .removePrefix("the ").removePrefix("my ").removeSuffix(" app").trim() if (name.isBlank()) return null + val wanted = APP_NICKNAMES[name] ?: name - val entries = LauncherCache.entries ?: withContext(Dispatchers.IO) { LauncherCache.load(context) } - val match = entries.firstOrNull { it.label.lowercase().contains(name) } + // Gearslip's own apps come first in the list, so "phone" opens its dialer, not the phone's. + val match = bestMatch(allEntries(context), wanted) { it.label } ?: return "I couldn't find an app called $name." launchEntry(match, navigator, frame) return "Opening ${match.label}." } - private val NAVIGATE_TRIGGERS = listOf("navigate to ", "directions to ", "take me to ", "drive to ") + private val EXIT_WORDS = setOf("exit", "exit gearslip", "close gearslip", "quit", "quit gearslip", "open exit") + + /** Other names people use for Gearslip's own apps. */ + private val APP_NICKNAMES = mapOf( + "web" to "browser", "internet" to "browser", "the web" to "browser", "web browser" to "browser", + "dialer" to "phone", "the dialer" to "phone", "keypad" to "phone", + "screen share" to "screen sharing", "screen mirroring" to "screen sharing", "mirroring" to "screen sharing", + ) + + private val NAVIGATE_TRIGGERS = listOf( + "navigate to ", "navigate me to ", "directions to ", "get directions to ", "give me directions to ", + "take me to ", "drive to ", "drive me to ", "get me to ", "go to ", "head to ", "route to ", + "how do i get to ", "show me the way to ", "find a route to ", "navigation to ", "get to ", + // Last, so "navigate to" wins: the recognizer sometimes drops the "to". + "navigate me ", "navigate ", "find me the nearest ", "find the nearest ", "find the closest ", + ) + + private val ARTICLES = setOf("a", "an", "the", "some") + // Longest first, so "the nearest " goes before "the ". + private val PLACE_LEADS = listOf( + "the nearest ", "the closest ", "a nearby ", "the nearby ", "nearest ", "closest ", "nearby ", + "a ", "an ", "some ", + ) + private val PLACE_TAILS = listOf(" near me", " nearby", " near here", " around here", " close by", " closest to me") + + private val PLAY_TRIGGERS = listOf("play me ", "play ", "put on ", "listen to ", "i want to hear ", "shuffle ") + + private val CALL_TRIGGERS = listOf("call ", "phone ", "dial ", "ring ", "make a call to ", "place a call to ", "start a call with ") + + private val OPEN_TRIGGERS = listOf("open ", "launch ", "start ", "go to ", "switch to ", "show me ", "bring up ", "pull up ") + + /** Screens "go to X" means rather than a place. */ + private val SCREEN_WORDS = setOf("settings", "home", "the home screen", "home screen", "apps", "my apps", "all apps") + + /** Padding stripped from the front of a command, longest first where they overlap. */ + private val LEAD_FILLERS = listOf( + "hey gearslip", "ok gearslip", "okay gearslip", "gearslip", "hey", "ok", "okay", "um", "uh", "so", + "please", "can you please", "can you", "could you", "would you", "will you", "can i", "could i", + "i want you to", "i want to", "i'd like to", "i would like to", "i wanna", "i need to", "i need you to", + "let's", "lets", "go ahead and", "just", "now", "quickly", + ) + + /** Longest first, so "right now" goes whole rather than leaving "right". */ + private val TAIL_FILLERS = listOf("thank you very much", "thank you", "right now", "for me", "please", "thanks", "now") + + private const val APP_CONNECT_MS = 5_000L + + private val HELP_WORDS = setOf("help", "commands", "voice commands", "list commands", "show commands") + + private val RESUME_WORDS = setOf("music", "the music", "some music", "something", "it", "again", "that", "my music") + + private val RESUME_PHRASES = setOf("keep playing", "continue", "keep going", "start the music", "music on", "turn the music back on") } diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/car/AssistantOverlay.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/car/AssistantOverlay.kt index 8c7a161..35f5814 100644 --- a/gearslip/src/main/java/app/seb3thehacker/gearslip/car/AssistantOverlay.kt +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/car/AssistantOverlay.kt @@ -4,7 +4,21 @@ import app.seb3thehacker.gearslip.car.theme.* import android.content.Intent import android.net.Uri import android.provider.Settings +import androidx.compose.animation.core.FastOutSlowInEasing +import androidx.compose.animation.core.RepeatMode +import androidx.compose.animation.core.animateFloat +import androidx.compose.animation.core.infiniteRepeatable +import androidx.compose.animation.core.rememberInfiniteTransition +import androidx.compose.animation.core.tween import androidx.compose.foundation.layout.Arrangement +import androidx.compose.foundation.layout.Box +import androidx.compose.ui.draw.drawBehind +import androidx.compose.ui.geometry.Offset +import androidx.compose.ui.graphics.Brush +import androidx.compose.ui.graphics.Color +import androidx.compose.ui.graphics.drawscope.scale +import androidx.compose.ui.text.style.TextAlign +import androidx.compose.ui.text.style.TextOverflow import androidx.compose.foundation.layout.Column import androidx.compose.foundation.layout.Row import androidx.compose.foundation.layout.fillMaxWidth @@ -30,47 +44,66 @@ import androidx.compose.ui.text.font.FontWeight import androidx.compose.ui.unit.dp /** - * The band that appears while [CarAssistant] is listening, thinking or speaking - a thin strip - * across the top, same idea as [CallOverlay]'s active-call banner, so it never blocks the screen - * the driver was already looking at. + * What the assistant hears and says, laid over the bottom of whatever screen is up: a dark shade + * so white words read over a bright map, and a glow along the bottom edge in the phone's own + * colours that breathes while listening. The shade passes touches through; only the buttons take them. */ @Composable fun AssistantOverlay(modifier: Modifier = Modifier) { val state by CarAssistant.state.collectAsState() if (state.phase == CarAssistant.Phase.IDLE) return val context = LocalContext.current + val listening = state.phase == CarAssistant.Phase.LISTENING && state.ready + val glow = MaterialTheme.colorScheme.primary + val glow2 = MaterialTheme.colorScheme.tertiary + val breathe by rememberInfiniteTransition(label = "glow").animateFloat( + 0.55f, 1f, infiniteRepeatable(tween(900, easing = FastOutSlowInEasing), RepeatMode.Reverse), label = "glow", + ) + val strength = when (state.phase) { + CarAssistant.Phase.LISTENING -> if (state.ready) breathe else 0.3f + CarAssistant.Phase.ERROR -> 0.3f + else -> 0.6f + } - Surface( - color = MaterialTheme.colorScheme.primaryContainer, - contentColor = MaterialTheme.colorScheme.onPrimaryContainer, - modifier = modifier.fillMaxWidth(), - ) { - Row( - Modifier.padding(horizontal = 16.dp, vertical = 10.dp), - verticalAlignment = Alignment.CenterVertically, - horizontalArrangement = Arrangement.spacedBy(12.dp), - ) { - PulsingMic(state.phase == CarAssistant.Phase.LISTENING) - Column(Modifier.weight(1f)) { - Text( - when (state.phase) { - CarAssistant.Phase.LISTENING -> "Listening…" - CarAssistant.Phase.THINKING -> "…" - CarAssistant.Phase.SPEAKING -> state.reply - CarAssistant.Phase.ERROR -> state.reply - CarAssistant.Phase.IDLE -> "" - }, - style = MaterialTheme.typography.titleSmall, - fontWeight = FontWeight.SemiBold, - maxLines = 2, - ) - if (state.heard.isNotBlank() && state.phase != CarAssistant.Phase.SPEAKING) { - Text( - "\"${state.heard}\"", - style = MaterialTheme.typography.bodySmall, + Box( + modifier + .fillMaxWidth() + .drawBehind { + drawRect(Brush.verticalGradient(0f to Color.Transparent, 0.45f to Color.Black.copy(alpha = 0.6f), 1f to Color.Black.copy(alpha = 0.85f))) + // A wide, flat oval of light rising from the bottom edge. + scale(1f, 0.22f, Offset(size.width / 2, size.height)) { + drawCircle( + Brush.radialGradient( + 0f to glow.copy(alpha = 0.9f * strength), 0.5f to glow2.copy(alpha = 0.35f * strength), 1f to Color.Transparent, + center = Offset(size.width / 2, size.height), radius = size.width * 0.55f, + ), + radius = size.width * 0.55f, center = Offset(size.width / 2, size.height), ) } } + .padding(start = 16.dp, end = 16.dp, top = 56.dp, bottom = 20.dp), + ) { + Row(verticalAlignment = Alignment.CenterVertically, horizontalArrangement = Arrangement.spacedBy(16.dp)) { + PulsingMic(listening) + Column(Modifier.weight(1f), horizontalAlignment = Alignment.CenterHorizontally) { + val (small, big) = when (state.phase) { + CarAssistant.Phase.LISTENING -> "" to state.heard.ifBlank { if (state.ready) "Listening…" else "One moment…" } + CarAssistant.Phase.THINKING -> "" to state.heard.ifBlank { "…" } + else -> state.heard to state.reply + } + if (small.isNotBlank()) { + Text(small, color = Color.White.copy(alpha = 0.7f), style = MaterialTheme.typography.bodyLarge, maxLines = 1, overflow = TextOverflow.Ellipsis) + } + Text( + big, + color = if (big == "Listening…" || big == "One moment…" || big == "…") Color.White.copy(alpha = 0.7f) else Color.White, + style = MaterialTheme.typography.headlineSmall, + fontWeight = FontWeight.SemiBold, + textAlign = TextAlign.Center, + maxLines = 2, + overflow = TextOverflow.Ellipsis, + ) + } if (state.missingPermission) { GsButton(tone = GsTone.Tonal, onClick = { context.startActivity( diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/car/BluetoothAudioCheck.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/car/BluetoothAudioCheck.kt new file mode 100644 index 0000000..8d84f41 --- /dev/null +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/car/BluetoothAudioCheck.kt @@ -0,0 +1,146 @@ +package app.seb3thehacker.gearslip.car + +import android.Manifest +import android.annotation.SuppressLint +import android.bluetooth.BluetoothA2dp +import android.bluetooth.BluetoothAdapter +import android.bluetooth.BluetoothManager +import android.bluetooth.BluetoothProfile +import android.content.BroadcastReceiver +import android.content.Context +import android.content.Intent +import android.content.IntentFilter +import android.content.pm.PackageManager +import android.os.Handler +import android.os.Looper +import app.seb3thehacker.gearslip.GearslipLog +import kotlinx.coroutines.flow.MutableStateFlow +import kotlinx.coroutines.flow.StateFlow + +/** + * With Bluetooth audio chosen, music only reaches the car if the phone is connected to it over + * Bluetooth. Twenty seconds after the car shows Gearslip, long enough for a car to connect on its + * own, this checks. If the phone isn't connected, the car and the phone's Home screen both say so, + * until it connects or the drive ends. + * + * When the car named its Bluetooth address, only that device counts. Otherwise any audio + * connection does, since there's no telling which device is the car. + */ +object BluetoothAudioCheck { + + private val _missing = MutableStateFlow(false) + /** True while Bluetooth audio is chosen and the phone isn't connected to the car for it. */ + val missing: StateFlow = _missing + + private val _carDismissed = MutableStateFlow(false) + /** The driver tapped OK on the car's card; it stays away for the rest of this drive. */ + val carDismissed: StateFlow = _carDismissed + + private val log = GearslipLog.tagged("bluetooth") + private val main = Handler(Looper.getMainLooper()) + private var appContext: Context? = null + private var armed = false + private var receiver: BroadcastReceiver? = null + + fun init(context: Context) { + appContext = context.applicationContext + } + + /** The car shows Gearslip. Called again whenever the video restarts, so it arms once per drive. */ + fun onProjecting() = main.post { + if (armed) return@post + armed = true + main.postDelayed(::start, WAIT_MS) + } + + fun onCarDisconnected() = main.post { + armed = false + main.removeCallbacksAndMessages(null) + stopWatching() + _missing.value = false + _carDismissed.value = false + } + + fun dismissOnCar() { + _carDismissed.value = true + } + + /** "Don't show again" on the car: turns the warning off in Settings, for every car. */ + fun neverAgain() { + CarSettings.setBluetoothWarning(false) + log.i("Bluetooth audio warning turned off") + _carDismissed.value = true + _missing.value = false + } + + private fun start() { + if (!armed) return + check() + // Connecting later clears the warning, and losing the connection brings it back. + val context = appContext ?: return + val r = object : BroadcastReceiver() { + override fun onReceive(context: Context, intent: Intent) { + main.postDelayed(::check, SETTLE_MS) + } + } + context.registerReceiver( + r, + IntentFilter(BluetoothA2dp.ACTION_CONNECTION_STATE_CHANGED).apply { + addAction(BluetoothAdapter.ACTION_STATE_CHANGED) + }, + Context.RECEIVER_EXPORTED, + ) + receiver = r + } + + @SuppressLint("MissingPermission") + private fun check() { + if (!armed) return + val context = appContext ?: return + if (CarSettings.pipeAudio.value) { + set(false, "USB audio is on") + return + } + if (!CarSettings.bluetoothWarning.value) { + set(false, "the warning is turned off") + return + } + if (context.checkSelfPermission(Manifest.permission.BLUETOOTH_CONNECT) != PackageManager.PERMISSION_GRANTED) { + log.i("can't check the audio connection: the Bluetooth permission isn't granted") + return + } + val adapter = context.getSystemService(BluetoothManager::class.java)?.adapter ?: return + if (!adapter.isEnabled) { + set(true, "Bluetooth is off") + return + } + val car = CarBluetooth.carAddress?.uppercase() + adapter.getProfileProxy(context, object : BluetoothProfile.ServiceListener { + override fun onServiceConnected(profile: Int, proxy: BluetoothProfile) { + val connected = runCatching { proxy.connectedDevices }.getOrDefault(emptyList()) + adapter.closeProfileProxy(profile, proxy) + val ok = if (car != null) connected.any { it.address.uppercase() == car } else connected.isNotEmpty() + main.post { + set(!ok, if (ok) "connected to the car" else "not connected to the car") + } + } + + override fun onServiceDisconnected(profile: Int) {} + }, BluetoothProfile.A2DP) + } + + private fun set(missing: Boolean, why: String) { + if (!armed || _missing.value == missing) return + _missing.value = missing + log.i("Bluetooth audio: $why") + } + + private fun stopWatching() { + val r = receiver ?: return + receiver = null + runCatching { appContext?.unregisterReceiver(r) } + } + + private const val WAIT_MS = 20_000L + private const val SETTLE_MS = 1_000L +} diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/car/BluetoothPairOverlay.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/car/BluetoothPairOverlay.kt new file mode 100644 index 0000000..ee35562 --- /dev/null +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/car/BluetoothPairOverlay.kt @@ -0,0 +1,118 @@ +package app.seb3thehacker.gearslip.car + +import app.seb3thehacker.gearslip.car.theme.* +import androidx.compose.foundation.background +import androidx.compose.foundation.clickable +import androidx.compose.foundation.interaction.MutableInteractionSource +import androidx.compose.foundation.layout.Arrangement +import androidx.compose.foundation.layout.Box +import androidx.compose.foundation.layout.Column +import androidx.compose.foundation.layout.Row +import androidx.compose.foundation.layout.fillMaxSize +import androidx.compose.foundation.layout.fillMaxWidth +import androidx.compose.foundation.layout.height +import androidx.compose.foundation.layout.padding +import androidx.compose.material3.MaterialTheme +import androidx.compose.material3.Surface +import androidx.compose.material3.Text +import androidx.compose.runtime.Composable +import androidx.compose.runtime.collectAsState +import androidx.compose.runtime.getValue +import androidx.compose.runtime.remember +import androidx.compose.ui.Alignment +import androidx.compose.ui.Modifier +import androidx.compose.ui.graphics.Color +import androidx.compose.ui.text.font.FontWeight +import androidx.compose.ui.text.style.TextAlign +import androidx.compose.ui.unit.dp + +/** + * Asks once per car whether to pair it over Bluetooth, then waits while the phone shows its + * pairing dialog. See [CarBluetooth]. With no pairing to do, says when Bluetooth audio isn't + * connected (see [BluetoothAudioCheck]), or else when the phone's location is off (see + * [LocationCheck]). Unlike the call prompt, a tap outside the card does nothing: + * the driver picks Pair or Not now. + */ +@Composable +internal fun BluetoothPairOverlay() { + val prompt by CarBluetooth.prompt.collectAsState() + val bluetoothAudio = !CarSettings.pipeAudio.collectAsState().value + when (prompt) { + CarBluetooth.Prompt.NONE -> { + val missing by BluetoothAudioCheck.missing.collectAsState() + val dismissed by BluetoothAudioCheck.carDismissed.collectAsState() + val warn by CarSettings.bluetoothWarning.collectAsState() + val locationOff by LocationCheck.off.collectAsState() + val locationDismissed by LocationCheck.carDismissed.collectAsState() + if (missing && bluetoothAudio && warn && !dismissed) PairCard( + title = "Bluetooth isn't connected", + body = "Music plays over Bluetooth, but your phone isn't connected to the car. " + + "Connect it from the car's Bluetooth menu or your phone's.", + secondary = "Don't show again" to BluetoothAudioCheck::neverAgain, + primary = "OK" to BluetoothAudioCheck::dismissOnCar, + ) else if (locationOff && !locationDismissed) PairCard( + title = "Location is off", + body = "Map apps can't show where you are or give directions. Turn on location " + + "in your phone's quick settings when you're parked.", + secondary = "Don't show again" to LocationCheck::neverAgain, + primary = "OK" to LocationCheck::dismissOnCar, + ) + } + CarBluetooth.Prompt.ASK -> PairCard( + title = "Pair with this car?", + body = (if (bluetoothAudio) "Calls and music play" else "Calls play") + + " through the car's speakers once your phone and the car are paired. Your phone " + + "then asks you to confirm a code.", + secondary = "Not now" to CarBluetooth::notNow, + primary = "Pair" to CarBluetooth::pair, + ) + CarBluetooth.Prompt.PAIRING -> PairCard( + title = "Confirm on your phone", + body = "Check the code on your phone and tap Pair. The car may ask too.", + secondary = null, + primary = "Hide" to CarBluetooth::hide, + ) + } +} + +@Composable +private fun PairCard(title: String, body: String, secondary: Pair Unit>?, primary: Pair Unit>) { + Box( + Modifier.fillMaxSize().background(Color.Black.copy(alpha = 0.6f)).clickable( + indication = null, + interactionSource = remember { MutableInteractionSource() }, + onClick = {}, + ), + contentAlignment = Alignment.Center, + ) { + Surface( + shape = MaterialTheme.shapes.extraLarge, + color = MaterialTheme.colorScheme.surfaceContainerHigh, + modifier = Modifier.fillMaxWidth(0.85f).padding(16.dp), + ) { + Column( + Modifier.padding(32.dp), + verticalArrangement = Arrangement.spacedBy(20.dp), + horizontalAlignment = Alignment.CenterHorizontally, + ) { + Text(title, style = MaterialTheme.typography.headlineSmall, fontWeight = FontWeight.Bold) + Text( + body, + style = MaterialTheme.typography.titleMedium, + color = MaterialTheme.colorScheme.onSurfaceVariant, + textAlign = TextAlign.Center, + ) + Row(Modifier.fillMaxWidth(), horizontalArrangement = Arrangement.spacedBy(16.dp)) { + secondary?.let { (label, onClick) -> + GsButton(onClick = onClick, tone = GsTone.Outline, modifier = Modifier.weight(1f).height(56.dp)) { + Text(label, style = MaterialTheme.typography.titleMedium) + } + } + GsButton(onClick = primary.second, modifier = Modifier.weight(1f).height(56.dp)) { + Text(primary.first, style = MaterialTheme.typography.titleMedium) + } + } + } + } + } +} diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/car/CarApps.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/car/CarApps.kt index 51e41f8..bea237a 100644 --- a/gearslip/src/main/java/app/seb3thehacker/gearslip/car/CarApps.kt +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/car/CarApps.kt @@ -22,7 +22,7 @@ class CarApp( object CarApps { val all: List = listOf( - CarApp("web", "Web", Icons.Filled.Search) { + CarApp("web", "Browser", Icons.Filled.Search) { WebApp(content = ProjectedContent.resolve(LocalContext.current)) }, CarApp("phone", "Screen sharing", Icons.Filled.Share) { diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/car/CarAssistant.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/car/CarAssistant.kt index cf4b27a..96e564d 100644 --- a/gearslip/src/main/java/app/seb3thehacker/gearslip/car/CarAssistant.kt +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/car/CarAssistant.kt @@ -38,6 +38,8 @@ object CarAssistant { val heard: String = "", val reply: String = "", val missingPermission: Boolean = false, + /** False from the press until the microphone is actually open, so the overlay doesn't claim to listen early. */ + val ready: Boolean = false, ) private val _state = MutableStateFlow(State()) @@ -89,6 +91,7 @@ object CarAssistant { tts?.stop() _state.value = State(Phase.LISTENING) listenOnce( + onListening = { _state.value = _state.value.copy(ready = true) }, onPartial = { text -> _state.value = _state.value.copy(heard = text) }, onHeard = { _state.value = _state.value.copy(phase = Phase.THINKING) }, onResult = { text -> @@ -174,15 +177,16 @@ object CarAssistant { onSpeaking: () -> Unit = {}, ) { cancel() - listenOnce(onPartial, onHeard = {}, onResult, onError, cue, onSpeaking) + listenOnce(onPartial, onHeard = {}, onResult = onResult, onError = onError, cue = cue, onSpeaking = onSpeaking) } /** - * One phrase, through Gearslip's own [SpeechEngine]. The phone's recognizer is only a fallback - * for when the engine itself can't run, since plenty of phones (GrapheneOS among them) have none. + * One phrase, through Gearslip's own [SpeechEngine], or the phone's recognizer when the driver + * chose it in settings. Never the phone's on its own: on most phones that's Google's. */ private fun listenOnce( onPartial: (String) -> Unit, + onListening: () -> Unit = {}, onHeard: () -> Unit, onResult: (String) -> Unit, onError: (String) -> Unit, @@ -190,24 +194,24 @@ object CarAssistant { onSpeaking: () -> Unit = {}, ) { val context = appContext ?: return onError("Speech isn't ready yet.") + if (CarSettings.systemSpeech.value && SpeechRecognizer.isRecognitionAvailable(context)) { + return systemListen(context, onListening, onPartial, onHeard, onResult, onError) + } SpeechEngine.listen( context, cue = cue, onSpeaking = onSpeaking, + onPartial = onPartial, + onListening = onListening, onHeard = onHeard, onResult = onResult, - onError = { message -> - if (message == SpeechEngine.NOT_LOADED && SpeechRecognizer.isRecognitionAvailable(context)) { - systemListen(context, onPartial, onHeard, onResult, onError) - } else { - onError(message) - } - }, + onError = onError, ) } private fun systemListen( context: Context, + onListening: () -> Unit, onPartial: (String) -> Unit, onHeard: () -> Unit, onResult: (String) -> Unit, @@ -216,7 +220,7 @@ object CarAssistant { val r = SpeechRecognizer.createSpeechRecognizer(context) recognizer = r r.setRecognitionListener(object : RecognitionListener { - override fun onReadyForSpeech(params: Bundle?) {} + override fun onReadyForSpeech(params: Bundle?) = onListening() override fun onBeginningOfSpeech() {} override fun onRmsChanged(rmsdB: Float) {} override fun onBufferReceived(buffer: ByteArray?) {} diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/car/CarBluetooth.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/car/CarBluetooth.kt new file mode 100644 index 0000000..f211656 --- /dev/null +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/car/CarBluetooth.kt @@ -0,0 +1,255 @@ +package app.seb3thehacker.gearslip.car + +import android.Manifest +import android.annotation.SuppressLint +import android.bluetooth.BluetoothAdapter +import android.bluetooth.BluetoothDevice +import android.bluetooth.BluetoothManager +import android.content.BroadcastReceiver +import android.content.Context +import android.content.Intent +import android.content.IntentFilter +import android.content.pm.PackageManager +import android.os.Handler +import android.os.Looper +import androidx.car.app.CarToast +import app.seb3thehacker.gearslip.AppSettings +import app.seb3thehacker.gearslip.GearslipLog +import app.seb3thehacker.gearslip.Protobuf +import app.seb3thehacker.gearslip.ServiceDiscovery +import app.seb3thehacker.gearslip.Wire +import kotlinx.coroutines.flow.MutableStateFlow +import kotlinx.coroutines.flow.StateFlow +import java.security.MessageDigest + +/** + * Pairs the phone with the car over Bluetooth the first time they meet, so calls (and music, when + * the driver chose Bluetooth audio) reach the car's speakers. Android Auto does this on its own; + * Gearslip asks first, on the car screen. + * + * The car names its Bluetooth address in its service list. When the driver taps Pair, Gearslip + * opens the car's Bluetooth channel and sends what Android Auto sends: + * + * -> CHANNEL_OPEN_REQUEST <- CHANNEL_OPEN_RESPONSE + * -> PairingRequest { phone_address, method } + * <- PairingResponse { status, already_paired } + * + * then starts pairing from the phone. Two things Android Auto has that Gearslip can't: the + * phone's own Bluetooth address (apps get a placeholder), and confirming the pairing code without + * asking. So a car may turn the request down, and the phone always shows its pairing dialog. The + * channel stays closed unless the driver taps Pair: every new channel is a risk to cars that work. + */ +object CarBluetooth { + + enum class Prompt { NONE, ASK, PAIRING } + + private val _prompt = MutableStateFlow(Prompt.NONE) + val prompt: StateFlow = _prompt + + private val log = GearslipLog.tagged("bluetooth") + private val main = Handler(Looper.getMainLooper()) + private var appContext: Context? = null + + /** This session's car: its channel, and how to open and write to it. */ + private class Car( + val service: ServiceDiscovery.BluetoothService, + val open: () -> Unit, + val send: (messageId: Int, body: ByteArray) -> Unit, + ) + + @Volatile private var car: Car? = null + @Volatile private var bondStarted = false + /** From the Pair tap until it succeeds or fails, whether or not the card still shows. */ + @Volatile private var pairing = false + private var receiver: BroadcastReceiver? = null + + /** The car's Bluetooth address, when it named one in its service list. */ + val carAddress: String? get() = car?.service?.carAddress + + /** The car's Bluetooth channel once the driver has asked to pair, -1 otherwise. */ + val channelId: Int get() = car?.takeIf { pairing }?.service?.serviceId ?: -1 + + fun init(context: Context) { + appContext = context.applicationContext + } + + /** Called once the car lists its services. Asks on the car screen only if there's pairing to do. */ + fun onCarConnected( + service: ServiceDiscovery.BluetoothService, + open: () -> Unit, + send: (messageId: Int, body: ByteArray) -> Unit, + ) { + car = Car(service, open, send) + bondStarted = false + log.i("the car offers Bluetooth pairing, methods=${service.pairingMethods}") + val device = device() ?: return + when { + device.bondState == BluetoothDevice.BOND_BONDED -> log.i("already paired with this car") + declined(service.carAddress) -> log.i("not paired, and the driver said not now for this car") + else -> _prompt.value = Prompt.ASK + } + } + + fun onCarDisconnected() { + car = null + pairing = false + main.post { + stopWatching() + _prompt.value = Prompt.NONE + } + } + + /** The driver tapped Pair. */ + fun pair() { + val c = car ?: return + val device = device() ?: return + pairing = true + _prompt.value = Prompt.PAIRING + watchBonding(device) + c.open() + log.i("-> ChannelOpenRequest on channel ${c.service.serviceId}") + // If the car never answers on its channel, pair from the phone anyway. + main.postDelayed({ startBond("the car didn't answer the pairing request") }, CAR_REPLY_WAIT_MS) + main.postDelayed({ + if (pairing) finish("Pairing didn't finish. Pair from the car's Bluetooth menu instead.") + }, GIVE_UP_MS) + } + + /** The driver tapped Not now: this car isn't asked about again. */ + fun notNow() { + car?.service?.carAddress?.let { address -> + val context = appContext ?: return@let + val list = AppSettings.getString(context, KEY_DECLINED, "").split(",").filter { it.isNotEmpty() } + AppSettings.putString(context, KEY_DECLINED, (list + hash(address)).distinct().joinToString(",")) + } + _prompt.value = Prompt.NONE + } + + /** Hides the pairing card. The phone's own dialog carries on. */ + fun hide() { + _prompt.value = Prompt.NONE + } + + fun onMessage(messageId: Int, body: ByteArray) { + val c = car ?: return + when (messageId) { + MSG_CHANNEL_OPEN_RESPONSE -> { + val status = Protobuf.readInt32Field(body, 1) ?: 0 + log.i("<- ChannelOpenResponse: status=$status") + if (status != 0) { + main.post { startBond("the car wouldn't open its Bluetooth channel") } + return + } + val method = c.service.pairingMethods.firstOrNull { it == METHOD_NUMERIC_COMPARISON } + ?: c.service.pairingMethods.firstOrNull() ?: METHOD_NUMERIC_COMPARISON + c.send( + MSG_PAIRING_REQUEST, + Protobuf.stringField(1, phoneAddress()) + Protobuf.varintField(2, method.toLong()), + ) + log.i("-> PairingRequest(method=$method)") + } + MSG_PAIRING_RESPONSE -> { + val fields = Wire.fields(body) + log.i("<- PairingResponse status=${Wire.varint(fields, 1)} already_paired=${Wire.varint(fields, 2) == 1L}") + // Some cars start pairing themselves once they answer; give them a moment first. + main.postDelayed({ startBond("the car answered") }, CAR_BOND_WAIT_MS) + } + // Carries the pairing code. Android shows it to the driver, so it stays out of the log. + MSG_AUTHENTICATION_DATA -> log.i("<- AuthenticationData (${body.size} bytes)") + else -> log.i("<- unhandled message id=$messageId (${body.size} bytes)") + } + } + + @SuppressLint("MissingPermission") + private fun startBond(why: String) { + if (bondStarted || !pairing) return + val device = device() ?: return + if (device.bondState != BluetoothDevice.BOND_NONE) return + bondStarted = true + val started = runCatching { device.createBond() }.getOrDefault(false) + log.i("pairing from the phone ($why): ${if (started) "started" else "refused"}") + if (!started) finish("Pairing didn't start. Pair from the car's Bluetooth menu instead.") + } + + private fun watchBonding(device: BluetoothDevice) { + val context = appContext ?: return + stopWatching() + val r = object : BroadcastReceiver() { + @Suppress("DEPRECATION") + override fun onReceive(context: Context, intent: Intent) { + val changed = intent.getParcelableExtra(BluetoothDevice.EXTRA_DEVICE) ?: return + if (changed.address != device.address) return + val state = intent.getIntExtra(BluetoothDevice.EXTRA_BOND_STATE, BluetoothDevice.BOND_NONE) + val previous = intent.getIntExtra(BluetoothDevice.EXTRA_PREVIOUS_BOND_STATE, BluetoothDevice.BOND_NONE) + log.i("bond state $previous -> $state") + when { + state == BluetoothDevice.BOND_BONDING -> bondStarted = true + state == BluetoothDevice.BOND_BONDED -> finish("Paired. Calls now play through the car.") + previous == BluetoothDevice.BOND_BONDING -> finish("Pairing didn't finish. Pair from the car's Bluetooth menu instead.") + } + } + } + context.registerReceiver(r, IntentFilter(BluetoothDevice.ACTION_BOND_STATE_CHANGED), Context.RECEIVER_EXPORTED) + receiver = r + } + + private fun finish(message: String) { + pairing = false + stopWatching() + main.removeCallbacksAndMessages(null) + _prompt.value = Prompt.NONE + CarToasts.show(this, message, CarToast.LENGTH_LONG) + } + + private fun stopWatching() { + val r = receiver ?: return + receiver = null + runCatching { appContext?.unregisterReceiver(r) } + } + + /** The car as a Bluetooth device, or null (and logged) when Gearslip can't reach Bluetooth. */ + private fun device(): BluetoothDevice? { + val context = appContext ?: return null + val address = car?.service?.carAddress ?: return null + if (context.checkSelfPermission(Manifest.permission.BLUETOOTH_CONNECT) != PackageManager.PERMISSION_GRANTED) { + log.i("can't check pairing: the Bluetooth permission isn't granted") + return null + } + val adapter = context.getSystemService(BluetoothManager::class.java)?.adapter + if (adapter == null || !adapter.isEnabled) { + log.i("can't check pairing: Bluetooth is off") + return null + } + if (!BluetoothAdapter.checkBluetoothAddress(address)) { + log.w("the car sent an address that isn't one") + return null + } + return adapter.getRemoteDevice(address) + } + + /** Apps get 02:00:00:00:00:00 here, not the real address. Sent anyway, as the field is required. */ + @SuppressLint("MissingPermission", "HardwareIds") + private fun phoneAddress(): String = + runCatching { appContext?.getSystemService(BluetoothManager::class.java)?.adapter?.address }.getOrNull() + ?: "02:00:00:00:00:00" + + private fun declined(address: String): Boolean { + val context = appContext ?: return false + return hash(address) in AppSettings.getString(context, KEY_DECLINED, "").split(",") + } + + /** Settings keep a short hash, not the car's address. */ + private fun hash(address: String): String = + MessageDigest.getInstance("SHA-256").digest(address.uppercase().toByteArray()) + .take(8).joinToString("") { "%02x".format(it) } + + private const val KEY_DECLINED = "bluetooth_pair_declined" + private const val CAR_REPLY_WAIT_MS = 5_000L + private const val CAR_BOND_WAIT_MS = 3_000L + private const val GIVE_UP_MS = 90_000L + private const val METHOD_NUMERIC_COMPARISON = 2 + private const val MSG_CHANNEL_OPEN_RESPONSE = 8 + private const val MSG_PAIRING_REQUEST = 32769 + private const val MSG_PAIRING_RESPONSE = 32770 + private const val MSG_AUTHENTICATION_DATA = 32771 +} diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/car/CarKeyboard.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/car/CarKeyboard.kt index 0dfa2fc..6139770 100644 --- a/gearslip/src/main/java/app/seb3thehacker/gearslip/car/CarKeyboard.kt +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/car/CarKeyboard.kt @@ -138,18 +138,18 @@ fun CarKeyboard( } // Control row: no backspace here - it stays with the letters above, not next to the - // key most likely to be hit right after it by mistake. Dismiss (when offered) is the - // last key, bottom-right corner of the keyboard. + // key most likely to be hit right after it by mistake. Search/enter takes the + // bottom-right corner, where a thumb expects it; dismiss (when offered) sits left of it. Row( Modifier.fillMaxWidth(), horizontalArrangement = Arrangement.spacedBy(4.dp), ) { Key(if (symbols) "abc" else "?123", Modifier.weight(1.4f)) { symbols = !symbols } Key("space", Modifier.weight(if (onDismiss != null) 3.2f else 4f)) { onTextChange("$text ") } - SubmitKey(Modifier.weight(1.6f), submitIcon, submitImage, onSubmit) if (onDismiss != null) { DismissKey(onDismiss, Modifier.weight(1.4f)) } + SubmitKey(Modifier.weight(1.6f), submitIcon, submitImage, onSubmit) } } } @@ -222,9 +222,9 @@ private fun SubmitKey(modifier: Modifier, search: Boolean, image: ImageVector?, * Big enough to hit on a bumpy road, small enough that four rows plus a field plus a few results * still fit. A 480-tall screen sets the floor; a taller one (LIVI's 1280x720) gets bigger keys. */ -private fun keyHeight(screenHeightDp: Int): Dp = (screenHeightDp * 0.075f).dp.coerceIn(36.dp, 52.dp) +private fun keyHeight(screenHeightDp: Int): Dp = (screenHeightDp * 0.09f).dp.coerceIn(44.dp, 64.dp) -private val LocalKeyHeight = compositionLocalOf { 36.dp } +private val LocalKeyHeight = compositionLocalOf { 44.dp } private val LETTER_ROWS = listOf( "qwertyuiop".map { it.toString() }, diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/car/CarKeys.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/car/CarKeys.kt index 3eeae41..3eed2a1 100644 --- a/gearslip/src/main/java/app/seb3thehacker/gearslip/car/CarKeys.kt +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/car/CarKeys.kt @@ -27,6 +27,11 @@ object CarKeys { private val _screenKeys = MutableSharedFlow(extraBufferCapacity = 8) val screenKeys: SharedFlow = _screenKeys + private val _focusMoves = MutableSharedFlow(extraBufferCapacity = 1) + + /** A knob turn or D-pad press, so controls that hide when left alone can come back. */ + val focusMoves: SharedFlow = _focusMoves + /** Delivers an Android key event to the car display; set while the car UI is showing. */ @Volatile var focusSink: ((keyCode: Int, down: Boolean) -> Unit)? = null @@ -58,6 +63,7 @@ object CarKeys { */ fun onKey(keyCode: Int, down: Boolean, longPress: Boolean): Boolean { if (keyCode in FOCUS_KEYS) { + _focusMoves.tryEmit(Unit) focusSink?.invoke(keyCode, down) return true } @@ -86,6 +92,7 @@ object CarKeys { /** A turn of the rotary knob: each click moves focus one step, like Tab or Shift+Tab. */ fun onRotate(delta: Int) { val sink = focusSink ?: return + _focusMoves.tryEmit(Unit) val code = if (delta > 0) KeyEvent.KEYCODE_TAB else KEYCODE_SHIFT_TAB repeat(kotlin.math.abs(delta).coerceAtMost(MAX_ROTARY_STEPS)) { sink(code, true) @@ -95,15 +102,10 @@ object CarKeys { /** * The voice key: answers the message on screen if there is one, otherwise starts the - * assistant when it's turned on. + * assistant. */ private fun voice() = main.post { - if (VoiceReply.onVoiceKey()) return@post - if (CarSettings.experimentalFeaturesEnabled.value && CarSettings.voiceAssistantEnabled.value) { - CarAssistant.start() - } else { - GearslipLog.i("keys: voice pressed, but the assistant is off in settings") - } + if (!VoiceReply.onVoiceKey()) CarAssistant.start() } /** Answers a ringing call; with nothing ringing, opens the phone screen. */ diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/car/CarLauncher.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/car/CarLauncher.kt index 6669c07..a5fc8a1 100644 --- a/gearslip/src/main/java/app/seb3thehacker/gearslip/car/CarLauncher.kt +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/car/CarLauncher.kt @@ -70,12 +70,8 @@ private class Tile( val label: String, val icon: ImageBitmap? = null, val glyph: ImageVector? = null, - /** Tested from start to finish with Gearslip; drawn with a check mark. */ - val verified: Boolean = false, /** Known not to work; drawn with a red X instead of being left off the launcher. */ val broken: Boolean = false, - /** Starts and can be used, but rough enough not to call working; drawn with a yellow mark. */ - val partial: Boolean = false, val onLongClick: (() -> Unit)? = null, val onClick: () -> Unit, ) @@ -187,6 +183,8 @@ private fun groupOf(entry: Entry, mediaPackages: Set): AppGroup { return when (app.kind) { "calling" -> AppGroup.PHONE "navigation", "poi", "parking", "charging" -> AppGroup.MAPS + // Template-only media apps may have no legacy browser/player tile to identify them. + "media" -> AppGroup.MUSIC "messaging" -> AppGroup.MESSAGING "settings" -> AppGroup.SETTINGS else -> AppGroup.OTHER @@ -248,18 +246,15 @@ fun CarLauncher() { val all = rememberAllEntries() val running = rememberRunningApps() val showVehicleData = rememberVehicleDataShown() - var showBadgeKey by remember { mutableStateOf(false) } fun tileOf(entry: Entry): Tile { - // The badges rate an app's own car screens; a messenger shown on Gearslip's own Messages - // screen has none, so it gets no badge. + // Only apps that don't work get a mark. The X rates an app's own car screens; a messenger + // shown on Gearslip's own Messages screen has none, so it never gets one. val pkg = entry.packageName?.takeIf { entry.messaging == null } val id = entry.componentId return Tile( entry.label, entry.icon, entry.builtIn?.glyph, - verified = pkg != null && !KnownApps.isPartial(pkg) && (KnownApps.works(pkg) || (entry.media != null && KnownApps.playerWorks(pkg))), broken = pkg != null && (KnownApps.isBroken(pkg) || (entry.template != null && KnownApps.screenBroken(pkg))), - partial = pkg != null && KnownApps.isPartial(pkg), onLongClick = id?.let { { navigator.showAppMenu(it) } }, ) { openApp(entry, id?.let { running[it] }, navigator, frame) } } @@ -275,60 +270,6 @@ fun CarLauncher() { verticalArrangement = Arrangement.spacedBy(12.dp), ) { items(tiles) { AppTile(it) } - if (tiles.any { it.verified || it.broken || it.partial }) { - // Folded away behind one button: it's reference, read once, not something every - // visit to the launcher needs spelled out under the grid. - item(span = { GridItemSpan(maxLineSpan) }) { - Column(horizontalAlignment = Alignment.CenterHorizontally) { - GsButton(onClick = { showBadgeKey = !showBadgeKey }, tone = GsTone.Neutral) { - Icon(Icons.Filled.Info, contentDescription = null, modifier = Modifier.size(18.dp)) - Spacer(Modifier.width(8.dp)) - Text(if (showBadgeKey) "Hide badge key" else "What the badges mean") - } - if (showBadgeKey) { - BadgeLegend( - verified = tiles.any { it.verified }, - partial = tiles.any { it.partial }, - broken = tiles.any { it.broken }, - ) - } - } - } - } - } -} - -/** Says what each badge on a tile means, once, under the grid - only the ones actually in use. */ -@Composable -private fun BadgeLegend(verified: Boolean, partial: Boolean, broken: Boolean) { - Column( - Modifier.fillMaxWidth().padding(top = 4.dp), - horizontalAlignment = Alignment.CenterHorizontally, - verticalArrangement = Arrangement.spacedBy(4.dp), - ) { - if (verified) LegendRow(size = 16.dp, badge = { VerifiedBadge(it) }, text = "Tested and working with Gearslip") - if (partial) LegendRow(size = 16.dp, badge = { PartialBadge(it) }, text = "Runs, but not cleanly, with Gearslip") - if (broken) LegendRow(size = 16.dp, badge = { BrokenBadge(it) }, text = "Doesn't work with Gearslip yet") - } -} - -@Composable -private fun LegendRow(size: androidx.compose.ui.unit.Dp, badge: @Composable (androidx.compose.ui.unit.Dp) -> Unit, text: String) { - Row(horizontalArrangement = Arrangement.Center, verticalAlignment = Alignment.CenterVertically) { - badge(size) - Spacer(Modifier.width(8.dp)) - Text(text, style = MaterialTheme.typography.bodyMedium, color = MaterialTheme.colorScheme.onSurfaceVariant) - } -} - -/** A green disc with a white check: the same mark on every tile and in the legend. */ -@Composable -private fun VerifiedBadge(size: androidx.compose.ui.unit.Dp, modifier: Modifier = Modifier) { - Box( - modifier.size(size).background(Color(0xFF43A047), CircleShape), - contentAlignment = Alignment.Center, - ) { - Icon(Icons.Filled.Check, contentDescription = "Tested and working", tint = Color.White, modifier = Modifier.size(size * 0.66f)) } } @@ -343,39 +284,16 @@ private fun BrokenBadge(size: androidx.compose.ui.unit.Dp, modifier: Modifier = } } -/** A yellow disc with a warning mark: neither the check nor the X, since this app runs but isn't clean about it. */ -@Composable -private fun PartialBadge(size: androidx.compose.ui.unit.Dp, modifier: Modifier = Modifier) { - Box( - modifier.size(size).background(Color(0xFFF9A825), CircleShape), - contentAlignment = Alignment.Center, - ) { - Icon(Icons.Filled.Warning, contentDescription = "Works, but not cleanly, with Gearslip", tint = Color.White, modifier = Modifier.size(size * 0.6f)) - } -} - -/** Trial: where a tile says whether its app works. See [AppTileBody]. */ -private enum class BadgeStyle { ON_ICON, OUTLINE } -private val BADGE_STYLE = BadgeStyle.ON_ICON - @OptIn(ExperimentalFoundationApi::class) @Composable private fun AppTile(tile: Tile) { - val status = when { - tile.verified -> Color(0xFF43A047) - tile.partial -> Color(0xFFF9A825) - tile.broken -> Color(0xFFD32F2F) - else -> null - } val shape = MaterialTheme.shapes.extraLarge // A long press opens the app's menu (pin, close) - the same gesture everywhere apps are // shown, here and on the nav bar. GsIconBox marks it in the theme's style. GsIconBox( onClick = tile.onClick, onLongClick = tile.onLongClick, - modifier = Modifier.height(120.dp).then( - if (BADGE_STYLE == BadgeStyle.OUTLINE && status != null) Modifier.border(2.5.dp, status, shape) else Modifier, - ), + modifier = Modifier.height(120.dp), colors = GsColors(MaterialTheme.colorScheme.surfaceVariant, MaterialTheme.colorScheme.onSurfaceVariant), shape = shape, ) { @@ -387,16 +305,14 @@ private fun AppTile(tile: Tile) { ) { Box { TileIcon(tile) - if (BADGE_STYLE == BadgeStyle.ON_ICON) { + if (tile.broken) { // Pinned to the icon's lower corner, ringed in the tile's colour so it // reads as sitting on top of the icon. - val ring = Modifier.align(Alignment.BottomEnd).offset(x = 6.dp, y = 4.dp) - .border(2.dp, MaterialTheme.colorScheme.surfaceVariant, CircleShape) - when { - tile.verified -> VerifiedBadge(22.dp, ring) - tile.broken -> BrokenBadge(22.dp, ring) - tile.partial -> PartialBadge(22.dp, ring) - } + BrokenBadge( + 22.dp, + Modifier.align(Alignment.BottomEnd).offset(x = 6.dp, y = 4.dp) + .border(2.dp, MaterialTheme.colorScheme.surfaceVariant, CircleShape), + ) } } Spacer(Modifier.height(8.dp)) diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/car/CarServices.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/car/CarServices.kt index cb8e114..47f084d 100644 --- a/gearslip/src/main/java/app/seb3thehacker/gearslip/car/CarServices.kt +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/car/CarServices.kt @@ -33,6 +33,12 @@ object CarServices { val nav: CarAppConnection by lazy { CarAppConnection(requireNotNull(context)) } val media: CarMedia by lazy { CarMedia(requireNotNull(context)) } + /** [nav], or null before the car screen has set things up. */ + fun navOrNull(): CarAppConnection? = if (context != null) nav else null + + /** [media], or null before the car screen has set things up: for code outside the car UI. */ + fun mediaOrNull(): CarMedia? = if (context != null) media else null + /** * A second, independent connection for templated apps that aren't navigation apps - Spotify's * own Car App Library service is the first of these. Kept apart from [nav] so opening one @@ -59,6 +65,10 @@ object CarServices { app.seb3thehacker.gearslip.notify.CarMotion.start(appContext) app.seb3thehacker.gearslip.call.CarCalls.start(appContext) CarAssistant.init(appContext) + DirectionsVoice.init(appContext) + CarBluetooth.init(appContext) + BluetoothAudioCheck.init(appContext) + LocationCheck.init(appContext) CarKeys.init(appContext) VoiceReply.init(appContext) // Loaded now, while nothing's waiting on it, so the first reply isn't slower than the rest. @@ -79,15 +89,22 @@ object CarServices { if (destination != null && nav.connectedComponent == app.component) { nav.navigateTo(destination) } else { + // Another map app taking over ends the old one's route, or it keeps guiding unseen. + if (nav.connectedComponent != app.component) endRoute() nav.connect(app, frame.width, frame.height, frame.densityDpi, destination) } } fun stopNav() { navStopped = true + endRoute() nav.disconnect() } + private fun endRoute() { + if (nav.navigating) nav.stopNavigating() + } + /** Undoes [stopNav]'s hold so the last map app comes back, same as it would after a restart. */ suspend fun reconnectNav(frame: CarEnvironment.Frame) { navStopped = false @@ -121,6 +138,19 @@ object CarServices { if (CarSettings.pipeAudio.value) CarAudio.request() } + /** + * Reopens the media app used last, for the car's media button when nothing is open. False + * when there's none on record or it's been uninstalled since. + */ + suspend fun reopenLastMedia(): Boolean { + val ctx = context ?: return false + val last = CarSettings.lastMedia.value?.let(ComponentName::unflattenFromString) ?: return false + val app = withContext(Dispatchers.IO) { MediaCatalog.installed(ctx) } + .firstOrNull { it.component == last } ?: return false + openMedia(app) + return true + } + /** Closes the media app from the car: stops what it's playing and drops the connection. */ fun closeMedia() { if (media.now.value.playing) media.togglePlay() @@ -194,6 +224,32 @@ object CarServices { return playingPackages.firstNotNullOfOrNull { pkg -> catalog.firstOrNull { it.component.packageName == pkg } } } + /** + * The car went away mid-song: pause it, as Android Auto does, so it doesn't carry on out of + * the phone's speaker. Unplugging a USB accessory isn't "audio becoming noisy" the way + * headphones are, so the apps won't pause themselves. Every playing session is paused, not + * just the one Gearslip opened: anything the car was playing was going through the car. + */ + fun pausePhoneMedia(why: String = "the car left") { + val ctx = context ?: return + val manager = ctx.getSystemService(MediaSessionManager::class.java) + val listener = ComponentName(ctx, GearslipNotificationListener::class.java) + val playing = try { + manager.getActiveSessions(listener).filter { it.playbackState?.state == PlaybackState.STATE_PLAYING } + } catch (e: SecurityException) { + null + } + if (playing == null) { + // No notification access: the app Gearslip itself is driving is all it can reach. + mediaOrNull()?.let { if (it.now.value.playing) it.togglePlay() } + return + } + playing.forEach { runCatching { it.transportControls.pause() } } + if (playing.isNotEmpty()) { + app.seb3thehacker.gearslip.GearslipLog.i("media: paused ${playing.joinToString { it.packageName }}: $why") + } + } + /** The car went away: nothing should keep running for a screen that no longer exists. */ fun shutdown() { context?.let { app.seb3thehacker.gearslip.notify.CarMotion.stop(it) } @@ -202,6 +258,8 @@ object CarServices { runCatching { media.disconnect() } runCatching { browse.disconnect() } CarAssistant.cancel() + // About 900 MB that nothing needs once the car's gone. + app.seb3thehacker.gearslip.speech.SpeechEngine.unload() CarAudio.release() _mediaApp.value = null navStopped = false diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/car/CarSettings.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/car/CarSettings.kt index 99616be..1be58f2 100644 --- a/gearslip/src/main/java/app/seb3thehacker/gearslip/car/CarSettings.kt +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/car/CarSettings.kt @@ -2,6 +2,7 @@ package app.seb3thehacker.gearslip.car import android.content.Context import app.seb3thehacker.gearslip.AppSettings +import app.seb3thehacker.gearslip.audio.CarAudio import app.seb3thehacker.gearslip.car.theme.GsTheme import app.seb3thehacker.gearslip.car.theme.GsThemes import kotlinx.coroutines.flow.MutableStateFlow @@ -27,6 +28,10 @@ object CarSettings { private const val KEY_OPEN = "car_open_on_connect" private const val KEY_NIGHT = "car_night_mode" private const val KEY_PIPE_AUDIO = "car_pipe_audio" +private const val KEY_SPEAK_DIRECTIONS = "car_speak_directions" +private const val KEY_BLUETOOTH_WARNING = "car_bluetooth_warning" +private const val KEY_LOCATION_WARNING = "car_location_warning" + private const val KEY_AUDIO_CHOICE_ASKED = "car_audio_choice_asked" private const val KEY_LAST_NAV = "car_last_nav" private const val KEY_LAST_MEDIA = "car_last_media" private const val KEY_AUTOPLAY = "car_autoplay" @@ -36,9 +41,8 @@ object CarSettings { private const val KEY_PINNED = "car_pinned_apps" private const val KEY_APP_ORDER = "car_app_order" private const val KEY_EXPERIMENTAL_FEATURES = "car_experimental_features" - private const val KEY_VOICE_ASSISTANT = "car_voice_assistant" + private const val KEY_SYSTEM_SPEECH = "car_system_speech" private const val KEY_CAR_SENSORS = "car_sensors_enabled" - private const val KEY_MEDIA_SEARCH = "car_media_search_enabled" private const val KEY_SEEN_SAFETY_WARNING = "car_seen_safety_warning" private const val KEY_UNITS = "car_units" private const val KEY_WHATS_NEW = "car_whats_new_after_updates" @@ -57,6 +61,9 @@ object CarSettings { private val openFlow = MutableStateFlow(HOME) private val nightFlow = MutableStateFlow(NightMode.AUTO) private val pipeAudioFlow = MutableStateFlow(true) + private val speakDirectionsFlow = MutableStateFlow(true) + private val bluetoothWarningFlow = MutableStateFlow(true) + private val locationWarningFlow = MutableStateFlow(true) private val lastNavFlow = MutableStateFlow(null) private val lastMediaFlow = MutableStateFlow(null) private val autoplayFlow = MutableStateFlow(false) @@ -70,9 +77,8 @@ object CarSettings { private val pinnedFlow = MutableStateFlow>(emptySet()) private val appOrderFlow = MutableStateFlow>(emptyList()) private val experimentalFeaturesFlow = MutableStateFlow(false) - private val voiceAssistantFlow = MutableStateFlow(false) + private val systemSpeechFlow = MutableStateFlow(false) private val carSensorsFlow = MutableStateFlow(false) - private val mediaSearchFlow = MutableStateFlow(false) private val seenSafetyWarningFlow = MutableStateFlow(false) private val unitsFlow = MutableStateFlow(Units.AUTO) @@ -148,22 +154,41 @@ object CarSettings { * Whether a media app's sound is captured and sent over the car link. On by default, since * sound in the car is the point of the exercise; the platform allows the capture only behind a * consent dialog, which [app.seb3thehacker.gearslip.audio.CarAudio] raises once the car has - * somewhere to play it. Turned off, the app plays through the phone's normal output instead. + * somewhere to play it. Turned off ("Bluetooth" in the settings), the app plays through the + * phone's normal output instead, and Android never asks. */ val pipeAudio: StateFlow = pipeAudioFlow + /** Gearslip reads the map app's turns aloud on the car's directions channel; see DirectionsVoice. */ + val speakDirections: StateFlow = speakDirectionsFlow + + /** With Bluetooth audio, warn when the phone isn't connected to the car; see BluetoothAudioCheck. */ + val bluetoothWarning: StateFlow = bluetoothWarningFlow + + /** Warn on the car when the phone's location is off; see LocationCheck. */ + val locationWarning: StateFlow = locationWarningFlow + + /** + * Whether the driver has already been offered Bluetooth after turning down the capture + * dialog. Offered once only: someone who chose to keep the USB route shouldn't be nagged. + */ + /** False until the driver picks USB or Bluetooth, in setup or in settings. */ + val pipeAudioChosen: Boolean + get() = AppSettings.getString(app, KEY_PIPE_AUDIO, "") != "" + + val audioChoiceAsked: Boolean + get() = AppSettings.getString(app, KEY_AUDIO_CHOICE_ASKED, "false") == "true" + + fun markAudioChoiceAsked() = AppSettings.putString(app, KEY_AUDIO_CHOICE_ASKED, "true") + /** * Off by default: a driver has to opt into experimental features as a whole before any of * them - the voice assistant included - show up to turn on individually. */ val experimentalFeaturesEnabled: StateFlow = experimentalFeaturesFlow - /** - * The voice assistant is experimental and off by default; a driver has to turn it on - * knowingly, from the settings screen, after reading the warning shown there. This flag - * alone does not run the assistant - [experimentalFeaturesEnabled] must also be on. - */ - val voiceAssistantEnabled: StateFlow = voiceAssistantFlow + /** Listen with the phone's own speech service instead of Gearslip's on-device model. */ + val systemSpeech: StateFlow = systemSpeechFlow /** * Vehicle data is experimental and off by default: the launcher's tile for it, and the @@ -173,13 +198,6 @@ object CarSettings { */ val carSensorsEnabled: StateFlow = carSensorsFlow - /** - * Searching inside a media app is experimental and off by default: many apps claim to take - * searches and then ignore them or play something else. The media screen's search button - * shows only once this and [experimentalFeaturesEnabled] are both on. - */ - val mediaSearchEnabled: StateFlow = mediaSearchFlow - /** * Whether the car screen's own "drive safely" warning has already been shown and dismissed * once, from this same install. Separate from the phone side's own compat/safety dialogs - @@ -206,6 +224,11 @@ object CarSettings { autoplayFlow.value = AppSettings.getString(app, KEY_AUTOPLAY, "false") == "true" playerTitleFlow.value = AppSettings.getString(app, KEY_PLAYER_TITLE, "false") == "true" pipeAudioFlow.value = AppSettings.getString(app, KEY_PIPE_AUDIO, "true") == "true" + // Until the driver sets it, on for USB audio and off for Bluetooth, where the map app's + // own voice already reaches the car. + speakDirectionsFlow.value = AppSettings.getString(app, KEY_SPEAK_DIRECTIONS, pipeAudioFlow.value.toString()) == "true" + bluetoothWarningFlow.value = AppSettings.getString(app, KEY_BLUETOOTH_WARNING, "true") == "true" + locationWarningFlow.value = AppSettings.getString(app, KEY_LOCATION_WARNING, "true") == "true" nightFlow.value = runCatching { NightMode.valueOf(AppSettings.getString(app, KEY_NIGHT, NightMode.AUTO.name)) }.getOrDefault(NightMode.AUTO) @@ -214,9 +237,8 @@ object CarSettings { appOrderFlow.value = AppSettings.getString(app, KEY_APP_ORDER, "") .split(",").filter { it.isNotEmpty() } experimentalFeaturesFlow.value = AppSettings.getString(app, KEY_EXPERIMENTAL_FEATURES, "false") == "true" - voiceAssistantFlow.value = AppSettings.getString(app, KEY_VOICE_ASSISTANT, "false") == "true" + systemSpeechFlow.value = AppSettings.getString(app, KEY_SYSTEM_SPEECH, "false") == "true" carSensorsFlow.value = AppSettings.getString(app, KEY_CAR_SENSORS, "false") == "true" - mediaSearchFlow.value = AppSettings.getString(app, KEY_MEDIA_SEARCH, "false") == "true" seenSafetyWarningFlow.value = AppSettings.getString(app, KEY_SEEN_SAFETY_WARNING, "false") == "true" unitsFlow.value = runCatching { Units.valueOf(AppSettings.getString(app, KEY_UNITS, Units.AUTO.name)) @@ -265,9 +287,32 @@ object CarSettings { AppSettings.putString(app, KEY_AUTOPLAY, on.toString()) } + /** + * Takes effect at once: switching to Bluetooth mid-drive stops the capture, and switching + * back asks for it again (or waits for the car, if none is connected). + */ fun setPipeAudio(on: Boolean) { - pipeAudioFlow.value = on + // Saved even when unchanged, so picking the default still counts as a choice. AppSettings.putString(app, KEY_PIPE_AUDIO, on.toString()) + if (pipeAudioFlow.value == on) return + pipeAudioFlow.value = on + if (AppSettings.getString(app, KEY_SPEAK_DIRECTIONS, "") == "") speakDirectionsFlow.value = on + if (on) CarAudio.request() else CarAudio.release() + } + + fun setSpeakDirections(on: Boolean) { + speakDirectionsFlow.value = on + AppSettings.putString(app, KEY_SPEAK_DIRECTIONS, on.toString()) + } + + fun setBluetoothWarning(on: Boolean) { + bluetoothWarningFlow.value = on + AppSettings.putString(app, KEY_BLUETOOTH_WARNING, on.toString()) + } + + fun setLocationWarning(on: Boolean) { + locationWarningFlow.value = on + AppSettings.putString(app, KEY_LOCATION_WARNING, on.toString()) } fun setExperimentalFeaturesEnabled(on: Boolean) { @@ -275,9 +320,9 @@ object CarSettings { AppSettings.putString(app, KEY_EXPERIMENTAL_FEATURES, on.toString()) } - fun setVoiceAssistantEnabled(on: Boolean) { - voiceAssistantFlow.value = on - AppSettings.putString(app, KEY_VOICE_ASSISTANT, on.toString()) + fun setSystemSpeech(on: Boolean) { + systemSpeechFlow.value = on + AppSettings.putString(app, KEY_SYSTEM_SPEECH, on.toString()) } fun setCarSensorsEnabled(on: Boolean) { @@ -285,11 +330,6 @@ object CarSettings { AppSettings.putString(app, KEY_CAR_SENSORS, on.toString()) } - fun setMediaSearchEnabled(on: Boolean) { - mediaSearchFlow.value = on - AppSettings.putString(app, KEY_MEDIA_SEARCH, on.toString()) - } - fun setNightMode(mode: NightMode) { nightFlow.value = mode AppSettings.putString(app, KEY_NIGHT, mode.name) diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/car/CarSettingsScreen.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/car/CarSettingsScreen.kt index fe1a617..c6248d4 100644 --- a/gearslip/src/main/java/app/seb3thehacker/gearslip/car/CarSettingsScreen.kt +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/car/CarSettingsScreen.kt @@ -53,14 +53,16 @@ fun CarSettingsScreen() { val units by CarSettings.units.collectAsState() val whatsNew by CarSettings.whatsNewAfterUpdates.collectAsState() val pipeAudio by CarSettings.pipeAudio.collectAsState() + val speakDirections by CarSettings.speakDirections.collectAsState() + val bluetoothWarning by CarSettings.bluetoothWarning.collectAsState() + val locationWarning by CarSettings.locationWarning.collectAsState() val autoplay by CarSettings.autoplay.collectAsState() val playerTitle by CarSettings.playerTitle.collectAsState() val theme by CarSettings.appTheme.collectAsState() val uiTheme by CarSettings.uiTheme.collectAsState() val experimentalFeatures by CarSettings.experimentalFeaturesEnabled.collectAsState() - val voiceAssistant by CarSettings.voiceAssistantEnabled.collectAsState() val carSensors by CarSettings.carSensorsEnabled.collectAsState() - val mediaSearch by CarSettings.mediaSearchEnabled.collectAsState() + val systemSpeech by CarSettings.systemSpeech.collectAsState() val display by CarEnvironment.display.collectAsState() val vehicle by CarEnvironment.vehicle.collectAsState() val context = LocalContext.current @@ -126,17 +128,49 @@ fun CarSettingsScreen() { ) { CarSettings.setAutoplay(it) } ChoiceRow( - "Media audio", - listOf(false to "Phone audio", true to "Car audio"), + "Car audio", + listOf(false to "Bluetooth", true to "USB"), pipeAudio, ) { CarSettings.setPipeAudio(it) } Text( - "Car audio sends the phone's sound to the car over USB. Android asks for permission " + - "when the car connects, and a few apps block it. Phone audio plays as the phone " + - "normally would, for example over Bluetooth to the car.", + "USB sends the phone's sound through Gearslip. Android asks to share your screen each " + + "drive, and a few apps block it. Bluetooth plays as the phone normally would, and " + + "Android never asks.", style = MaterialTheme.typography.bodySmall, color = MaterialTheme.colorScheme.onSurfaceVariant, ) + if (!pipeAudio) { + ChoiceRow( + "Warn when Bluetooth isn't connected", + listOf(false to "Off", true to "On"), + bluetoothWarning, + ) { CarSettings.setBluetoothWarning(it) } + } + ChoiceRow( + "Warn when location is off", + listOf(false to "Off", true to "On"), + locationWarning, + ) { CarSettings.setLocationWarning(it) } + + ChoiceRow( + "Spoken directions", + listOf(false to "Off", true to "On"), + speakDirections, + ) { CarSettings.setSpeakDirections(it) } + Text( + "Gearslip reads each turn aloud through the car's speakers, and the music dips while it " + + "speaks. Turn it off if your map app's own voice already plays in the car.", + style = MaterialTheme.typography.bodySmall, + color = MaterialTheme.colorScheme.onSurfaceVariant, + ) + + ExperimentalFeatureRow( + title = "Use the phone's speech recognition", + description = "Listens through the phone's own speech service instead of Gearslip's. On " + + "most phones that's Google's, which may send your voice to Google. GrapheneOS has " + + "none, so Gearslip keeps using its own.", + checked = systemSpeech, + ) { CarSettings.setSystemSpeech(it) } WebPageSetting() @@ -148,7 +182,7 @@ fun CarSettingsScreen() { experimentalFeatures, ) { CarSettings.setExperimentalFeaturesEnabled(it) } if (experimentalFeatures) { - ExperimentalFeaturesList(voiceAssistant, carSensors, mediaSearch) + ExperimentalFeaturesList(carSensors) } Column(verticalArrangement = Arrangement.spacedBy(8.dp)) { @@ -205,21 +239,11 @@ private fun ChoiceRow(title: String, options: List>, selecte * so the list reads fine whether there's one entry or several. */ @Composable -private fun ExperimentalFeaturesList(voiceAssistant: Boolean, carSensors: Boolean, mediaSearch: Boolean) { +private fun ExperimentalFeaturesList(carSensors: Boolean) { Column( Modifier.fillMaxWidth(), verticalArrangement = Arrangement.spacedBy(10.dp), ) { - ExperimentalFeatureRow( - title = "Voice assistant", - description = "Push-to-talk and the car's voice button start listening: navigate, " + - "play media, call, or open an app by saying so. Can mishear you or act on the " + - "wrong command - keep your eyes on the road, and confirm anything it does " + - "before you rely on it.", - warning = true, - checked = voiceAssistant, - ) { CarSettings.setVoiceAssistantEnabled(it) } - ExperimentalFeatureRow( title = "Vehicle data", description = "Adds a Vehicle data tile: a live, raw dump of the head unit's sensor " + @@ -228,13 +252,6 @@ private fun ExperimentalFeaturesList(voiceAssistant: Boolean, carSensors: Boolea "guess, not a fact, until you've watched one change for yourself.", checked = carSensors, ) { CarSettings.setCarSensorsEnabled(it) } - - ExperimentalFeatureRow( - title = "Media search", - description = "Adds a search button to the media screen. Some apps ignore the search " + - "or play something else.", - checked = mediaSearch, - ) { CarSettings.setMediaSearchEnabled(it) } } } @@ -272,7 +289,7 @@ private fun ExperimentalFeatureRow( } } -/** The URL (or raw HTML) the Web app shows. Edited with [CarKeyboard] - no phone in hand needed. */ +/** The URL (or raw HTML) the Browser app shows. Edited with [CarKeyboard] - no phone in hand needed. */ @Composable private fun WebPageSetting() { val context = LocalContext.current @@ -297,7 +314,7 @@ private fun WebPageSetting() { ) } Text( - "Shown by the Web app on the launcher. Applies on the next connection.", + "Shown by the Browser app on the launcher. Applies on the next connection.", style = MaterialTheme.typography.bodySmall, color = MaterialTheme.colorScheme.onSurfaceVariant, ) diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/car/CarToastOverlay.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/car/CarToastOverlay.kt new file mode 100644 index 0000000..fb877e7 --- /dev/null +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/car/CarToastOverlay.kt @@ -0,0 +1,62 @@ +package app.seb3thehacker.gearslip.car + +import androidx.compose.foundation.background +import androidx.compose.foundation.layout.Box +import androidx.compose.foundation.layout.padding +import androidx.compose.foundation.layout.widthIn +import androidx.compose.foundation.shape.RoundedCornerShape +import androidx.compose.material3.MaterialTheme +import androidx.compose.material3.Text +import androidx.compose.runtime.Composable +import androidx.compose.runtime.LaunchedEffect +import androidx.compose.runtime.collectAsState +import androidx.compose.runtime.getValue +import androidx.compose.ui.Modifier +import androidx.compose.ui.draw.clip +import androidx.compose.ui.platform.LocalAccessibilityManager +import androidx.compose.ui.semantics.LiveRegionMode +import androidx.compose.ui.semantics.liveRegion +import androidx.compose.ui.semantics.semantics +import androidx.compose.ui.text.style.TextOverflow +import androidx.compose.ui.unit.dp +import kotlinx.coroutines.delay +import kotlinx.coroutines.flow.first +import kotlinx.coroutines.withTimeoutOrNull + +/** Draw inside the encoded car display; a normal Android Toast would appear on the phone. */ +@Composable +internal fun CarToastOverlay(modifier: Modifier = Modifier) { + val current by CarToasts.message.collectAsState() + val message = current ?: return + val accessibility = LocalAccessibilityManager.current + LaunchedEffect(message, accessibility) { + val timeout = accessibility?.calculateRecommendedTimeoutMillis( + message.durationMs, containsIcons = false, containsText = true, containsControls = false, + ) ?: message.durationMs + // The full time, unless another toast is waiting: then it gives way after the minimum. + delay(minOf(timeout, CarToasts.MIN_SHOWN_MS)) + withTimeoutOrNull((timeout - CarToasts.MIN_SHOWN_MS).coerceAtLeast(0)) { + CarToasts.waiting.first { it > 0 } + } + CarToasts.dismiss(message) + } + + // A non-interactive overlay leaves the map's touch handling and focus intact. + Box( + modifier + .padding(horizontal = 24.dp, vertical = 16.dp) + .widthIn(max = 560.dp) + .clip(RoundedCornerShape(20.dp)) + .background(MaterialTheme.colorScheme.surfaceContainerHigh) + .semantics { liveRegion = LiveRegionMode.Polite }, + ) { + Text( + message.text, + modifier = Modifier.padding(horizontal = 20.dp, vertical = 12.dp), + color = MaterialTheme.colorScheme.onSurface, + style = MaterialTheme.typography.titleMedium, + maxLines = 3, + overflow = TextOverflow.Ellipsis, + ) + } +} diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/car/CarToasts.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/car/CarToasts.kt new file mode 100644 index 0000000..6d3e300 --- /dev/null +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/car/CarToasts.kt @@ -0,0 +1,65 @@ +package app.seb3thehacker.gearslip.car + +import androidx.annotation.MainThread +import androidx.car.app.CarToast +import kotlinx.coroutines.flow.MutableStateFlow +import kotlinx.coroutines.flow.StateFlow +import kotlinx.coroutines.flow.asStateFlow + +/** + * One toast at a time across navigation and browse apps. A newer one waits its turn instead of + * replacing the one on screen, which gets at least [MIN_SHOWN_MS] before it gives way. + */ +@MainThread +internal object CarToasts { + // Identity matters: repeating identical text must restart its display timer. + class Message(val owner: Any, val text: String, val durationMs: Long) + + /** Long enough to read a short toast at a glance; a waiting toast cuts the rest short. */ + const val MIN_SHOWN_MS = 2_000L + + /** A burst beyond this many is stale by the time it would show; the oldest waiting go first. */ + private const val MAX_WAITING = 3 + + private val current = MutableStateFlow(null) + val message: StateFlow = current.asStateFlow() + + private val queue = ArrayDeque() + private val _waiting = MutableStateFlow(0) + val waiting: StateFlow = _waiting.asStateFlow() + + fun show(owner: Any, text: String, duration: Int) { + if (text.isBlank()) return + // Longer than a phone toast: the driver only glances at the screen now and then. + val durationMs = if (duration == CarToast.LENGTH_LONG) 7_000L else 4_000L + val next = Message(owner, text, durationMs) + val shown = current.value + when { + shown == null -> current.value = next + // The same words again restart the one on screen rather than queueing a copy. + shown.text == text -> current.value = next + queue.lastOrNull()?.text == text -> Unit + else -> { + queue.addLast(next) + while (queue.size > MAX_WAITING) queue.removeFirst() + } + } + _waiting.value = queue.size + } + + fun dismiss(message: Message) { + // An old timer must not dismiss a newer toast, even if its text is identical. + if (current.value === message) advance() + } + + fun clear(owner: Any) { + // Disconnecting the map must not clear a toast belonging to the browse app. + queue.removeAll { it.owner === owner } + if (current.value?.owner === owner) advance() else _waiting.value = queue.size + } + + private fun advance() { + current.value = queue.removeFirstOrNull() + _waiting.value = queue.size + } +} diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/car/CarUi.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/car/CarUi.kt index 03baae2..8ca4826 100644 --- a/gearslip/src/main/java/app/seb3thehacker/gearslip/car/CarUi.kt +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/car/CarUi.kt @@ -2,6 +2,7 @@ package app.seb3thehacker.gearslip.car import androidx.compose.foundation.Image import androidx.compose.foundation.clickable +import androidx.compose.foundation.LocalIndication import androidx.compose.foundation.layout.Arrangement import androidx.compose.foundation.layout.Box import androidx.compose.foundation.layout.BoxWithConstraints @@ -43,6 +44,10 @@ import androidx.compose.material3.dynamicDarkColorScheme import androidx.compose.material3.dynamicLightColorScheme import androidx.compose.runtime.Composable import androidx.compose.runtime.CompositionLocalProvider +import androidx.compose.ui.hapticfeedback.HapticFeedback +import androidx.compose.ui.hapticfeedback.HapticFeedbackType +import androidx.compose.ui.platform.LocalHapticFeedback +import androidx.compose.ui.platform.LocalView import androidx.compose.runtime.State import androidx.compose.runtime.collectAsState import androidx.compose.runtime.getValue @@ -120,12 +125,16 @@ fun CarUi() { controlsWake.tick() } } + // The knob counts as a touch: the map's buttons can't be found while they're faded out. + LaunchedEffect(controlsWake) { CarKeys.focusMoves.collect { controlsWake.touch() } } remember(appContext) { Prefetch.warm(appContext, force = true); CarServices.init(appContext) } DisposableEffect(Unit) { onDispose { CarServices.shutdown() } } // Ask for the phone's sound as soon as the car connects, not only when a media app opens: // a video in the web browser or a map's voice needs it too. Asks once, after the car offers // somewhere to play it. LaunchedEffect(Unit) { if (CarSettings.pipeAudio.value) app.seb3thehacker.gearslip.audio.CarAudio.request() } + // A new version gets one card in the top right, a few seconds in and only while parked. + LaunchedEffect(Unit) { app.seb3thehacker.gearslip.update.UpdateChecker.onCarConnected(appContext) } // The map follows the light outside, whatever the app's own theme is set to. LaunchedEffect(darkOutside) { CarServices.nav.pushConfiguration() } @@ -150,14 +159,27 @@ fun CarUi() { VoiceReply.state.value.phase != VoiceReply.Phase.IDLE -> VoiceReply.cancel() else -> navigator.back() } - CarKeys.ScreenKey.HOME, CarKeys.ScreenKey.MAP -> navigator.home() - CarKeys.ScreenKey.MEDIA -> if (CarServices.mediaApp.value != null) navigator.media() else navigator.apps() + // Home is the dashboard, over the map, so Back from it lands on the map. + CarKeys.ScreenKey.HOME -> { navigator.home(); navigator.dashboard() } + CarKeys.ScreenKey.MAP -> navigator.home() + CarKeys.ScreenKey.MEDIA -> + if (CarServices.mediaApp.value != null || CarServices.reopenLastMedia()) navigator.media() + else navigator.apps() CarKeys.ScreenKey.PHONE -> BuiltInApps.Phone.open(navigator) } } } + // The car screen is drawn on the phone, so every long press or text-handle drag would buzz + // the phone in its mount. Nobody feels that while driving; it's only noise. + val view = LocalView.current + DisposableEffect(view) { + view.isHapticFeedbackEnabled = false + onDispose { } + } + CompositionLocalProvider( + LocalHapticFeedback provides NoHaptics, LocalDensity provides Density(base.density * scale, base.fontScale), LocalCarNavigator provides navigator, LocalDarkOutside provides darkOutside, @@ -165,6 +187,7 @@ fun CarUi() { LocalGsTheme provides GsThemes.byId(uiThemeId), ) { MaterialTheme(colorScheme = colors) { + CompositionLocalProvider(LocalIndication provides FocusRingIndication(MaterialTheme.colorScheme.primary)) { Surface( Modifier .fillMaxSize() @@ -224,6 +247,9 @@ fun CarUi() { } } } + CarToastOverlay(Modifier.align(Alignment.BottomCenter)) + CallPromptOverlay() + BluetoothPairOverlay() PopupTimer() // The dashboard is already listing them, so it needs no popup. NotificationPopup( @@ -231,14 +257,15 @@ fun CarUi() { modifier = Modifier.align(Alignment.TopEnd), ) // Above the popup: a call is more urgent than any notification. + VoiceHelpOverlay(Modifier.align(Alignment.Center)) CallOverlay(Modifier.align(Alignment.TopCenter)) - AssistantOverlay(Modifier.align(Alignment.TopCenter)) + AssistantOverlay(Modifier.align(Alignment.BottomCenter)) if (app.seb3thehacker.gearslip.BuildConfig.DEBUG) { // Bottom corner, not top: a screen's own header almost always puts // something in the top-right (settings, search, a Liked Songs // shortcut) and this badge used to sit right on top of it, hiding // whatever the screen drew there. - DevBuildBadge(Modifier.align(Alignment.BottomEnd)) + DebugBuildBadge(Modifier.align(Alignment.BottomEnd)) } } // The keyboard takes the nav bar's row instead of squeezing in above it - @@ -252,6 +279,7 @@ fun CarUi() { if (safetyWarningSeen) CarWhatsNew() } } + } } } } @@ -354,7 +382,7 @@ private fun AppWontStart(app: String, onBack: () -> Unit) { * that matters while driving, since a debug build is slower and less tested than what ships. */ @Composable -private fun DevBuildBadge(modifier: Modifier = Modifier) { +private fun DebugBuildBadge(modifier: Modifier = Modifier) { Surface( modifier = modifier.padding(6.dp), color = MaterialTheme.colorScheme.error, @@ -362,7 +390,7 @@ private fun DevBuildBadge(modifier: Modifier = Modifier) { shape = MaterialTheme.shapes.small, ) { Text( - "DEV BUILD", + "DEBUG BUILD", style = MaterialTheme.typography.labelSmall, fontWeight = FontWeight.Bold, modifier = Modifier.padding(horizontal = 8.dp, vertical = 3.dp), @@ -443,12 +471,8 @@ private fun CarNavBar(navigator: CarNavigator) { NavHomeButton(navigator, entries, running) Spacer(Modifier.width(6.dp)) NavItem(MediaIcons.Apps, navigator.current == CarScreen.Apps) { navigator.apps() } - val experimentalFeatures by CarSettings.experimentalFeaturesEnabled.collectAsState() - val voiceEnabled by CarSettings.voiceAssistantEnabled.collectAsState() - if (experimentalFeatures && voiceEnabled) { - Spacer(Modifier.width(6.dp)) - NavAssistantIcon() - } + Spacer(Modifier.width(6.dp)) + NavAssistantIcon() // Pinned shortcuts, then whatever else is open right now. A pinned app that's also // open stays in its pinned spot, marked with a dot, instead of showing up twice. @@ -879,3 +903,8 @@ private fun ChargingGlyph(modifier: Modifier = Modifier, tint: Color = MaterialT drawPath(bolt, tint) } } + +/** Stands in for the phone's vibration motor on the car screen: does nothing. */ +private object NoHaptics : HapticFeedback { + override fun performHapticFeedback(hapticFeedbackType: HapticFeedbackType) {} +} diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/car/ClusterNav.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/car/ClusterNav.kt new file mode 100644 index 0000000..8a4f8f6 --- /dev/null +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/car/ClusterNav.kt @@ -0,0 +1,199 @@ +package app.seb3thehacker.gearslip.car + +import androidx.car.app.model.CarText +import androidx.car.app.model.Distance +import androidx.car.app.navigation.model.Maneuver +import androidx.car.app.navigation.model.NavigationTemplate +import androidx.car.app.navigation.model.RoutingInfo +import androidx.car.app.navigation.model.Step +import app.seb3thehacker.gearslip.GearslipLog +import app.seb3thehacker.gearslip.Protobuf + +/** + * The next turn on the car's instrument cluster, over its NavigationStatusService (Service + * field 8), the way Android Auto feeds it: + * + * -> NavigationStatus { status = 1 (1 active, 2 inactive, 3 rerouting) } + * -> NavigationNextTurnEvent { road = 1; turn_side = 2; event = 3; turn_number = 5; turn_angle = 6 } + * -> NavigationNextTurnDistanceEvent { distance_meters = 1; time_to_turn_seconds = 2; + * display_distance_e3 = 3; display_distance_unit = 4 } + * + * Newer cars also take a NavigationState message; the deprecated pair above is what a 2018 + * Uconnect understands. The turn comes from the map app's trip (what the Car App Library hands + * to clusters), or from its routing card when it sends no trip. + */ +class ClusterNav( + val channelId: Int, + private val sendOnChannel: (messageId: Int, body: ByteArray) -> Unit, +) { + @Volatile private var running = false + private var sentStatus = -1 + private var sentTurn: String? = null + private var sentDistance: String? = null + + fun onMessage(messageId: Int, body: ByteArray) { + when (messageId) { + MSG_CHANNEL_OPEN_RESPONSE -> { + val status = Protobuf.readInt32Field(body, 1) ?: 0 + GearslipLog.i("cluster: <- ChannelOpenResponse: status=$status") + if (status == 0) start() + } + // The car saying when it wants turns. Sending all along is harmless, so these are only logged. + MSG_START -> GearslipLog.i("cluster: <- start") + MSG_STOP -> GearslipLog.i("cluster: <- stop") + else -> GearslipLog.i("cluster: <- unhandled message id=$messageId (${body.size} bytes)") + } + } + + private fun start() { + if (running) return + running = true + Thread({ + while (running) { + runCatching { tick() }.onFailure { GearslipLog.w("cluster: ${it.message}") } + Thread.sleep(TICK_MS) + } + }, "gearslip-cluster").start() + } + + fun stop() { + running = false + } + + private fun tick() { + val turn = NavTurn.current() + val status = when { + turn == null -> STATUS_INACTIVE + turn.rerouting -> STATUS_REROUTING + else -> STATUS_ACTIVE + } + if (status != sentStatus) { + sendOnChannel(MSG_STATUS, Protobuf.varintField(1, status.toLong())) + GearslipLog.i("cluster: -> ${STATUS_NAMES[status]}") + sentStatus = status + if (status == STATUS_INACTIVE) { sentTurn = null; sentDistance = null } + } + if (turn == null || status == STATUS_REROUTING) return + sendTurn(turn.step) + turn.distance?.let { sendDistance(it, turn.seconds) } + } + + private fun sendTurn(step: Step) { + val maneuver = step.maneuver + val (event, side) = maneuver?.let { eventFor(it.type) } ?: (EVENT_UNKNOWN to SIDE_UNSPECIFIED) + val road = step.road.text().ifBlank { step.cue.text() } + val key = "$road|$event|$side|${maneuver?.roundaboutExitNumber}" + if (key == sentTurn) return + sentTurn = key + var body = Protobuf.stringField(1, road) + + Protobuf.varintField(2, side.toLong()) + + Protobuf.varintField(3, event.toLong()) + maneuver?.roundaboutExitNumber?.takeIf { it > 0 }?.let { body += Protobuf.varintField(5, it.toLong()) } + maneuver?.roundaboutExitAngle?.takeIf { it > 0 }?.let { body += Protobuf.varintField(6, it.toLong()) } + sendOnChannel(MSG_TURN, body) + GearslipLog.i("cluster: -> turn event=$event side=$side road=\"$road\"") + } + + /** + * The Car App Library's distance units use the same numbers as the protocol's (1 metres ... + * 7 yards), so the app's own display choice passes straight through. + */ + private fun sendDistance(distance: Distance, seconds: Long?) { + val meters = NavTurn.toMeters(distance) + val key = "${distance.displayDistance}|${distance.displayUnit}|${seconds ?: -1}" + if (key == sentDistance) return + sentDistance = key + sendOnChannel( + MSG_DISTANCE, + Protobuf.varintField(1, meters.toLong()) + + Protobuf.varintField(2, (seconds ?: 0L).coerceAtLeast(0)) + + Protobuf.varintField(3, (distance.displayDistance * 1000).toLong()) + + Protobuf.varintField(4, distance.displayUnit.toLong()), + ) + } + + private fun CarText?.text() = this?.toString().orEmpty() + + companion object { + private const val TICK_MS = 500L + private const val MSG_CHANNEL_OPEN_RESPONSE = 8 + private const val MSG_START = 32769 + private const val MSG_STOP = 32770 + private const val MSG_STATUS = 32771 + private const val MSG_TURN = 32772 + private const val MSG_DISTANCE = 32773 + + private const val STATUS_ACTIVE = 1 + private const val STATUS_INACTIVE = 2 + private const val STATUS_REROUTING = 3 + private val STATUS_NAMES = mapOf(1 to "navigating", 2 to "not navigating", 3 to "rerouting") + + private const val SIDE_LEFT = 1 + private const val SIDE_RIGHT = 2 + private const val SIDE_UNSPECIFIED = 3 + + private const val EVENT_UNKNOWN = 0 + + /** The car's turn icon and side for a Car App Library manoeuvre. */ + fun eventFor(type: Int): Pair = when (type) { + Maneuver.TYPE_DEPART -> 1 to SIDE_UNSPECIFIED + Maneuver.TYPE_NAME_CHANGE -> 2 to SIDE_UNSPECIFIED + Maneuver.TYPE_KEEP_LEFT, Maneuver.TYPE_TURN_SLIGHT_LEFT -> 3 to SIDE_LEFT + Maneuver.TYPE_KEEP_RIGHT, Maneuver.TYPE_TURN_SLIGHT_RIGHT -> 3 to SIDE_RIGHT + Maneuver.TYPE_TURN_NORMAL_LEFT -> 4 to SIDE_LEFT + Maneuver.TYPE_TURN_NORMAL_RIGHT -> 4 to SIDE_RIGHT + Maneuver.TYPE_TURN_SHARP_LEFT -> 5 to SIDE_LEFT + Maneuver.TYPE_TURN_SHARP_RIGHT -> 5 to SIDE_RIGHT + Maneuver.TYPE_U_TURN_LEFT -> 6 to SIDE_LEFT + Maneuver.TYPE_U_TURN_RIGHT -> 6 to SIDE_RIGHT + Maneuver.TYPE_ON_RAMP_SLIGHT_LEFT, Maneuver.TYPE_ON_RAMP_NORMAL_LEFT, Maneuver.TYPE_ON_RAMP_SHARP_LEFT, + Maneuver.TYPE_ON_RAMP_U_TURN_LEFT -> 7 to SIDE_LEFT + Maneuver.TYPE_ON_RAMP_SLIGHT_RIGHT, Maneuver.TYPE_ON_RAMP_NORMAL_RIGHT, Maneuver.TYPE_ON_RAMP_SHARP_RIGHT, + Maneuver.TYPE_ON_RAMP_U_TURN_RIGHT -> 7 to SIDE_RIGHT + Maneuver.TYPE_OFF_RAMP_SLIGHT_LEFT, Maneuver.TYPE_OFF_RAMP_NORMAL_LEFT -> 8 to SIDE_LEFT + Maneuver.TYPE_OFF_RAMP_SLIGHT_RIGHT, Maneuver.TYPE_OFF_RAMP_NORMAL_RIGHT -> 8 to SIDE_RIGHT + Maneuver.TYPE_FORK_LEFT -> 9 to SIDE_LEFT + Maneuver.TYPE_FORK_RIGHT -> 9 to SIDE_RIGHT + Maneuver.TYPE_MERGE_LEFT -> 10 to SIDE_LEFT + Maneuver.TYPE_MERGE_RIGHT -> 10 to SIDE_RIGHT + Maneuver.TYPE_MERGE_SIDE_UNSPECIFIED -> 10 to SIDE_UNSPECIFIED + Maneuver.TYPE_ROUNDABOUT_ENTER_CW, Maneuver.TYPE_ROUNDABOUT_ENTER_CCW -> 11 to SIDE_UNSPECIFIED + Maneuver.TYPE_ROUNDABOUT_EXIT_CW, Maneuver.TYPE_ROUNDABOUT_EXIT_CCW -> 12 to SIDE_UNSPECIFIED + Maneuver.TYPE_ROUNDABOUT_ENTER_AND_EXIT_CW, Maneuver.TYPE_ROUNDABOUT_ENTER_AND_EXIT_CW_WITH_ANGLE, + Maneuver.TYPE_ROUNDABOUT_ENTER_AND_EXIT_CCW, Maneuver.TYPE_ROUNDABOUT_ENTER_AND_EXIT_CCW_WITH_ANGLE -> 13 to SIDE_UNSPECIFIED + Maneuver.TYPE_STRAIGHT -> 14 to SIDE_UNSPECIFIED + Maneuver.TYPE_FERRY_BOAT, Maneuver.TYPE_FERRY_BOAT_LEFT, Maneuver.TYPE_FERRY_BOAT_RIGHT -> 16 to SIDE_UNSPECIFIED + Maneuver.TYPE_FERRY_TRAIN, Maneuver.TYPE_FERRY_TRAIN_LEFT, Maneuver.TYPE_FERRY_TRAIN_RIGHT -> 17 to SIDE_UNSPECIFIED + Maneuver.TYPE_DESTINATION, Maneuver.TYPE_DESTINATION_STRAIGHT -> 19 to SIDE_UNSPECIFIED + Maneuver.TYPE_DESTINATION_LEFT -> 19 to SIDE_LEFT + Maneuver.TYPE_DESTINATION_RIGHT -> 19 to SIDE_RIGHT + else -> EVENT_UNKNOWN to SIDE_UNSPECIFIED + } + } +} + +/** One turn as the car wants it, from whichever source the map app fills. */ +internal class NavTurn(val step: Step, val distance: Distance?, val seconds: Long?, val rerouting: Boolean) { + companion object { + /** The map app's next turn: from its trip (what clusters get), or its routing card. */ + fun current(): NavTurn? { + val nav = CarServices.navOrNull() ?: return null + nav.trip.value?.let { trip -> + val step = trip.steps.firstOrNull() ?: return@let + val estimate = trip.stepTravelEstimates.firstOrNull() + return NavTurn(step, estimate?.remainingDistance, estimate?.remainingTimeSeconds?.takeIf { it >= 0 }, trip.isLoading) + } + val info = (nav.template.value as? NavigationTemplate)?.navigationInfo as? RoutingInfo ?: return null + val step = info.currentStep ?: return null + return NavTurn(step, info.currentDistance, null, info.isLoading) + } + + fun toMeters(d: Distance): Int = (d.displayDistance * when (d.displayUnit) { + Distance.UNIT_KILOMETERS, Distance.UNIT_KILOMETERS_P1 -> 1000.0 + Distance.UNIT_MILES, Distance.UNIT_MILES_P1 -> 1609.344 + Distance.UNIT_FEET -> 0.3048 + Distance.UNIT_YARDS -> 0.9144 + else -> 1.0 + }).toInt() + } +} diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/car/DirectionsVoice.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/car/DirectionsVoice.kt new file mode 100644 index 0000000..6b9abf3 --- /dev/null +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/car/DirectionsVoice.kt @@ -0,0 +1,326 @@ +package app.seb3thehacker.gearslip.car + +import android.content.Context +import android.os.Bundle +import android.speech.tts.TextToSpeech +import android.speech.tts.UtteranceProgressListener +import androidx.car.app.model.CarText +import androidx.car.app.model.Distance +import androidx.car.app.navigation.model.Maneuver +import androidx.car.app.navigation.model.Step +import app.seb3thehacker.gearslip.GearslipLog +import app.seb3thehacker.gearslip.audio.AudioLink +import java.io.File +import java.nio.ByteBuffer +import java.nio.ByteOrder +import java.util.concurrent.ConcurrentHashMap +import java.util.concurrent.CountDownLatch +import java.util.concurrent.Executors +import java.util.concurrent.TimeUnit +import kotlin.math.roundToInt + +/** + * Spoken turn-by-turn on the car's directions channel (the GUIDANCE audio sink). + * + * A map app on the phone speaks its own prompts as navigation audio, which Android can't hand to + * Gearslip without root, and which the phone mutes while USB audio runs. So Gearslip reads the + * same turn the instrument cluster gets ([NavTurn]) and says it itself: once when the turn comes + * up, again a kilometre or so out on a long stretch, and once more just before it. The words are + * rendered with the phone's text-to-speech to a file, never played on the phone, and streamed to + * the car, which ducks the music under them the way it does for real Android Auto. + * + * Nothing spoken is logged: road names are where the driver is going. + */ +object DirectionsVoice { + + private enum class Stage { FAR, PREPARED, DONE } + + private var appContext: Context? = null + private var tts: TextToSpeech? = null + @Volatile private var ttsReady = false + private val pending = ConcurrentHashMap() + + @Volatile private var link: AudioLink? = null + @Volatile private var running = false + private val speaker = Executors.newSingleThreadExecutor { Thread(it, "gearslip-directions").apply { isDaemon = true } } + @Volatile private var latest = 0 + + private var turnKey: String? = null + private var stage = Stage.DONE + + fun init(context: Context) { + if (tts != null) return + appContext = context.applicationContext + tts = TextToSpeech(appContext) { status -> ttsReady = status == TextToSpeech.SUCCESS }.apply { + setOnUtteranceProgressListener(object : UtteranceProgressListener() { + override fun onStart(utteranceId: String?) {} + override fun onDone(utteranceId: String?) { pending.remove(utteranceId)?.countDown() } + @Deprecated("Deprecated in Java, no replacement on this API level") + override fun onError(utteranceId: String?) { pending.remove(utteranceId)?.countDown() } + }) + } + } + + /** The car opened its directions sink: start following the map app's turns. */ + fun attach(link: AudioLink) { + this.link = link + GearslipLog.i("guidance: the car offered a directions sink at ${link.sampleRate}Hz x${link.channels}") + if (running) return + running = true + turnKey = null + Thread({ + while (running) { + runCatching { tick() }.onFailure { GearslipLog.w("guidance: ${it.message}") } + Thread.sleep(TICK_MS) + } + }, "gearslip-directions-watch").apply { isDaemon = true }.start() + } + + fun detach(link: AudioLink) { + if (this.link !== link) return + this.link = null + running = false + } + + private fun tick() { + if (!CarSettings.speakDirections.value) return + val turn = NavTurn.current() + if (turn == null) { + turnKey = null + return + } + if (turn.rerouting) return + val step = turn.step + val instruction = instruction(step) ?: return + val meters = turn.distance?.let(NavTurn::toMeters) + val close = (turn.seconds != null && turn.seconds <= NEAR_SECONDS) || (meters != null && meters <= NEAR_METERS) + + val key = "${step.road.text()}|${step.maneuver?.type}|${step.maneuver?.roundaboutExitNumber}|${step.cue.text()}" + if (key != turnKey) { + turnKey = key + if (step.maneuver?.type == Maneuver.TYPE_DEPART) { + stage = Stage.DONE + return + } + if (close || turn.distance == null) { + say(instruction) + stage = Stage.DONE + } else { + say("In ${spoken(turn.distance)}, $instruction") + stage = if (meters != null && meters > PREPARE_FROM_METERS) Stage.FAR else Stage.PREPARED + } + return + } + when { + stage != Stage.DONE && close -> { + say(instruction) + stage = Stage.DONE + } + stage == Stage.FAR && meters != null && meters <= PREPARE_AT_METERS -> { + say("In ${spoken(turn.distance)}, $instruction") + stage = Stage.PREPARED + } + } + } + + /** What to do, in words: built from the manoeuvre when the app names one, else the app's own text. */ + private fun instruction(step: Step): String? { + val road = step.road.text() + val onto = if (road.isBlank()) "" else " onto $road" + val toward = if (road.isBlank()) "" else " toward $road" + val maneuver = step.maneuver + val built = when (maneuver?.type) { + Maneuver.TYPE_TURN_SLIGHT_LEFT -> "Turn slightly left$onto" + Maneuver.TYPE_TURN_SLIGHT_RIGHT -> "Turn slightly right$onto" + Maneuver.TYPE_TURN_NORMAL_LEFT -> "Turn left$onto" + Maneuver.TYPE_TURN_NORMAL_RIGHT -> "Turn right$onto" + Maneuver.TYPE_TURN_SHARP_LEFT -> "Turn sharply left$onto" + Maneuver.TYPE_TURN_SHARP_RIGHT -> "Turn sharply right$onto" + Maneuver.TYPE_U_TURN_LEFT, Maneuver.TYPE_U_TURN_RIGHT -> "Make a U-turn" + Maneuver.TYPE_KEEP_LEFT -> "Keep left$toward" + Maneuver.TYPE_KEEP_RIGHT -> "Keep right$toward" + Maneuver.TYPE_FORK_LEFT -> "Keep left at the fork$toward" + Maneuver.TYPE_FORK_RIGHT -> "Keep right at the fork$toward" + Maneuver.TYPE_ON_RAMP_SLIGHT_LEFT, Maneuver.TYPE_ON_RAMP_NORMAL_LEFT, Maneuver.TYPE_ON_RAMP_SHARP_LEFT, + Maneuver.TYPE_ON_RAMP_U_TURN_LEFT -> "Take the ramp on the left$onto" + Maneuver.TYPE_ON_RAMP_SLIGHT_RIGHT, Maneuver.TYPE_ON_RAMP_NORMAL_RIGHT, Maneuver.TYPE_ON_RAMP_SHARP_RIGHT, + Maneuver.TYPE_ON_RAMP_U_TURN_RIGHT -> "Take the ramp on the right$onto" + Maneuver.TYPE_OFF_RAMP_SLIGHT_LEFT, Maneuver.TYPE_OFF_RAMP_NORMAL_LEFT -> "Take the exit on the left$toward" + Maneuver.TYPE_OFF_RAMP_SLIGHT_RIGHT, Maneuver.TYPE_OFF_RAMP_NORMAL_RIGHT -> "Take the exit on the right$toward" + Maneuver.TYPE_MERGE_LEFT -> "Merge left$onto" + Maneuver.TYPE_MERGE_RIGHT -> "Merge right$onto" + Maneuver.TYPE_MERGE_SIDE_UNSPECIFIED -> "Merge$onto" + Maneuver.TYPE_ROUNDABOUT_ENTER_AND_EXIT_CW, Maneuver.TYPE_ROUNDABOUT_ENTER_AND_EXIT_CW_WITH_ANGLE, + Maneuver.TYPE_ROUNDABOUT_ENTER_AND_EXIT_CCW, Maneuver.TYPE_ROUNDABOUT_ENTER_AND_EXIT_CCW_WITH_ANGLE -> + maneuver.roundaboutExitNumber.takeIf { it in 1..ORDINALS.size } + ?.let { "At the roundabout, take the ${ORDINALS[it - 1]} exit$onto" } + ?: "Go through the roundabout$onto" + Maneuver.TYPE_ROUNDABOUT_ENTER_CW, Maneuver.TYPE_ROUNDABOUT_ENTER_CCW -> "Enter the roundabout" + Maneuver.TYPE_ROUNDABOUT_EXIT_CW, Maneuver.TYPE_ROUNDABOUT_EXIT_CCW -> "Exit the roundabout$onto" + Maneuver.TYPE_STRAIGHT -> "Continue straight$onto" + Maneuver.TYPE_NAME_CHANGE -> "Continue$onto" + Maneuver.TYPE_FERRY_BOAT, Maneuver.TYPE_FERRY_BOAT_LEFT, Maneuver.TYPE_FERRY_BOAT_RIGHT -> "Take the ferry" + Maneuver.TYPE_FERRY_TRAIN, Maneuver.TYPE_FERRY_TRAIN_LEFT, Maneuver.TYPE_FERRY_TRAIN_RIGHT -> "Take the train" + Maneuver.TYPE_DESTINATION, Maneuver.TYPE_DESTINATION_STRAIGHT -> "Your destination is ahead" + Maneuver.TYPE_DESTINATION_LEFT -> "Your destination is on the left" + Maneuver.TYPE_DESTINATION_RIGHT -> "Your destination is on the right" + else -> null + } + return built ?: step.cue.text().ifBlank { null } + } + + /** The app's own rounded distance, read the way a person would say it. */ + private fun spoken(d: Distance): String { + val n = d.displayDistance + val number = if (n % 1.0 == 0.0) n.toInt().toString() else "%.1f".format(n) + val one = n == 1.0 + return when (d.displayUnit) { + Distance.UNIT_KILOMETERS, Distance.UNIT_KILOMETERS_P1 -> if (one) "1 kilometer" else "$number kilometers" + Distance.UNIT_MILES, Distance.UNIT_MILES_P1 -> when (n) { + 0.25 -> "a quarter mile" + 0.5 -> "half a mile" + 0.75 -> "three quarters of a mile" + 1.0 -> "1 mile" + else -> "$number miles" + } + Distance.UNIT_FEET -> "$number feet" + Distance.UNIT_YARDS -> "$number yards" + else -> "${n.roundToInt()} meters" + } + } + + /** Newest wins: a prompt still waiting when the next one arrives is already out of date. */ + private fun say(text: String) { + val id = ++latest + speaker.execute { + if (id != latest) return@execute + runCatching { play(text, id) }.onFailure { GearslipLog.w("guidance: could not speak a turn: ${it.message}") } + } + } + + private fun play(text: String, id: Int) { + val engine = tts ?: return + val context = appContext ?: return + val sink = link ?: return + if (!ttsReady) { + GearslipLog.w("guidance: text-to-speech isn't ready, a turn went unspoken") + return + } + val file = File(context.cacheDir, "directions.wav") + val utterance = "gearslip-directions-$id" + val done = CountDownLatch(1) + pending[utterance] = done + if (engine.synthesizeToFile(text, Bundle(), file, utterance) != TextToSpeech.SUCCESS) { + pending.remove(utterance) + return + } + if (!done.await(SYNTH_TIMEOUT_S, TimeUnit.SECONDS)) { + pending.remove(utterance) + GearslipLog.w("guidance: text-to-speech took too long, a turn went unspoken") + return + } + val pcm = Wav.read(file)?.let { Wav.convert(it, sink.sampleRate, sink.channels) } ?: run { + GearslipLog.w("guidance: text-to-speech gave audio Gearslip can't read") + return + } + stream(sink, pcm) + GearslipLog.i("guidance: spoke a turn (${pcm.size / (2 * sink.channels) * 1000 / sink.sampleRate} ms)") + } + + /** Start, the words in 20 ms chunks at about real time, a little silence, Stop. */ + private fun stream(sink: AudioLink, pcm: ByteArray) { + val frame = 2 * sink.channels + val chunk = sink.sampleRate / 50 * frame + val chunkMs = 20L + val padded = pcm + ByteArray(sink.sampleRate / 5 * frame) // 200 ms tail so the last word isn't clipped + sink.start() + try { + val began = System.currentTimeMillis() + var offset = 0 + var sent = 0 + while (offset < padded.size && link === sink) { + val size = minOf(chunk, padded.size - offset) + sink.write(padded.copyOfRange(offset, offset + size), size) + offset += size + sent++ + val ahead = sent * chunkMs - (System.currentTimeMillis() - began) + if (ahead > LEAD_MS) Thread.sleep(ahead - LEAD_MS) + } + // Let the car play what it has before closing the stream on it. + Thread.sleep(LEAD_MS) + } finally { + sink.stop() + } + } + + private fun CarText?.text() = this?.toString().orEmpty() + + /** 16-bit PCM from the WAV text-to-speech writes, reshaped for the car's sink. */ + private object Wav { + class Pcm(val samples: ShortArray, val rate: Int, val channels: Int) + + fun read(file: File): Pcm? { + val bytes = file.readBytes() + if (bytes.size < 12 || String(bytes, 0, 4) != "RIFF" || String(bytes, 8, 4) != "WAVE") return null + val buf = ByteBuffer.wrap(bytes).order(ByteOrder.LITTLE_ENDIAN) + var pos = 12 + var rate = 0 + var channels = 0 + var bits = 0 + while (pos + 8 <= bytes.size) { + val name = String(bytes, pos, 4) + val size = buf.getInt(pos + 4) + val body = pos + 8 + // Some engines write a streaming WAV with a zero or oversized data length. + val end = if (size <= 0 || body + size > bytes.size) bytes.size else body + size + when (name) { + "fmt " -> { + channels = buf.getShort(body + 2).toInt() + rate = buf.getInt(body + 4) + bits = buf.getShort(body + 14).toInt() + } + "data" -> { + if (bits != 16 || rate <= 0 || channels <= 0) return null + val samples = ShortArray((end - body) / 2) + buf.position(body) + buf.asShortBuffer().get(samples, 0, samples.size) + return Pcm(samples, rate, channels) + } + } + pos = end + (end and 1) + } + return null + } + + /** Mixes to mono, resamples linearly, then copies to each of the car's channels. */ + fun convert(pcm: Pcm, rate: Int, channels: Int): ByteArray { + val frames = pcm.samples.size / pcm.channels + val mono = FloatArray(frames) { f -> + var sum = 0f + for (c in 0 until pcm.channels) sum += pcm.samples[f * pcm.channels + c] + sum / pcm.channels + } + val outFrames = (frames.toLong() * rate / pcm.rate).toInt() + val out = ByteBuffer.allocate(outFrames * channels * 2).order(ByteOrder.LITTLE_ENDIAN) + val step = pcm.rate.toDouble() / rate + for (i in 0 until outFrames) { + val at = i * step + val j = at.toInt().coerceAtMost(frames - 1) + val next = (j + 1).coerceAtMost(frames - 1) + val v = mono[j] + (mono[next] - mono[j]) * (at - j).toFloat() + val s = v.roundToInt().coerceIn(Short.MIN_VALUE.toInt(), Short.MAX_VALUE.toInt()).toShort() + repeat(channels) { out.putShort(s) } + } + return out.array() + } + } + + private const val TICK_MS = 500L + private const val NEAR_SECONDS = 12L + private const val NEAR_METERS = 120 + private const val PREPARE_FROM_METERS = 2_500 + private const val PREPARE_AT_METERS = 1_000 + private const val LEAD_MS = 100L + private const val SYNTH_TIMEOUT_S = 10L + private val ORDINALS = listOf("first", "second", "third", "fourth", "fifth", "sixth", "seventh", "eighth") +} diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/car/FocusRing.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/car/FocusRing.kt new file mode 100644 index 0000000..bc063c5 --- /dev/null +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/car/FocusRing.kt @@ -0,0 +1,84 @@ +package app.seb3thehacker.gearslip.car + +import androidx.compose.foundation.IndicationNodeFactory +import androidx.compose.foundation.interaction.FocusInteraction +import androidx.compose.foundation.interaction.InteractionSource +import androidx.compose.material3.ripple +import androidx.compose.ui.geometry.CornerRadius +import androidx.compose.ui.geometry.Offset +import androidx.compose.ui.geometry.Size +import androidx.compose.ui.graphics.Color +import androidx.compose.ui.graphics.drawscope.ContentDrawScope +import androidx.compose.ui.graphics.drawscope.Stroke +import androidx.compose.ui.node.DelegatableNode +import androidx.compose.ui.node.DelegatingNode +import androidx.compose.ui.node.DrawModifierNode +import androidx.compose.ui.node.invalidateDraw +import androidx.compose.ui.unit.dp +import kotlinx.coroutines.launch + +/** + * What every car button shows: the usual ripple when pressed, and a ring in the accent color + * while the knob or D-pad is on it. Cars without a knob never see the ring, since nothing takes + * focus while the screen is in touch mode. + * + * It replaces the theme's ripple (LocalIndication) for the whole car UI, so any `clickable` that + * doesn't pass its own indication gets the ring for free. + */ +internal class FocusRingIndication(private val color: Color) : IndicationNodeFactory { + private val ripple = ripple() + + override fun create(interactionSource: InteractionSource): DelegatableNode = + FocusRingNode(ripple.create(interactionSource), interactionSource, color) + + override fun equals(other: Any?) = other is FocusRingIndication && other.color == color + override fun hashCode() = color.hashCode() +} + +private class FocusRingNode( + rippleNode: DelegatableNode, + private val source: InteractionSource, + private val color: Color, +) : DelegatingNode(), DrawModifierNode { + private var focused = false + + init { delegate(rippleNode) } + + override fun onAttach() { + coroutineScope.launch { + var holding = 0 + source.interactions.collect { + when (it) { + is FocusInteraction.Focus -> holding++ + is FocusInteraction.Unfocus -> holding-- + } + val now = holding > 0 + if (now != focused) { + focused = now + invalidateDraw() + } + } + } + } + + override fun ContentDrawScope.draw() { + drawContent() + if (!focused) return + // Inside the button's bounds, since most buttons clip to their shape: drawn outside, a + // rounded tile would cut its own ring off. + val width = RING_WIDTH.toPx() + val radius = minOf(CORNER.toPx(), size.minDimension / 2) + drawRoundRect( + color, + topLeft = Offset(width / 2, width / 2), + size = Size(size.width - width, size.height - width), + cornerRadius = CornerRadius(radius), + style = Stroke(width), + ) + } + + private companion object { + val RING_WIDTH = 3.dp + val CORNER = 16.dp + } +} diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/car/LocationCheck.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/car/LocationCheck.kt new file mode 100644 index 0000000..74bb711 --- /dev/null +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/car/LocationCheck.kt @@ -0,0 +1,93 @@ +package app.seb3thehacker.gearslip.car + +import android.content.BroadcastReceiver +import android.content.Context +import android.content.Intent +import android.content.IntentFilter +import android.location.LocationManager +import android.os.Handler +import android.os.Looper +import app.seb3thehacker.gearslip.GearslipLog +import kotlinx.coroutines.flow.MutableStateFlow +import kotlinx.coroutines.flow.StateFlow + +/** + * Map apps can't find the car with the phone's location turned off. Ten seconds after the car + * shows Gearslip, this checks, and the car says so until location comes on or the drive ends. + */ +object LocationCheck { + + private val _off = MutableStateFlow(false) + /** True while the phone's location is turned off during a drive. */ + val off: StateFlow = _off + + private val _carDismissed = MutableStateFlow(false) + /** The driver tapped OK on the car's card; it stays away for the rest of this drive. */ + val carDismissed: StateFlow = _carDismissed + + private val log = GearslipLog.tagged("location") + private val main = Handler(Looper.getMainLooper()) + private var appContext: Context? = null + private var armed = false + private var receiver: BroadcastReceiver? = null + + fun init(context: Context) { + appContext = context.applicationContext + } + + /** The car shows Gearslip. Called again whenever the video restarts, so it arms once per drive. */ + fun onProjecting() = main.post { + if (armed) return@post + armed = true + main.postDelayed(::start, WAIT_MS) + } + + fun onCarDisconnected() = main.post { + armed = false + main.removeCallbacksAndMessages(null) + stopWatching() + _off.value = false + _carDismissed.value = false + } + + fun dismissOnCar() { + _carDismissed.value = true + } + + /** "Don't show again" on the car: turns the warning off in Settings, for every car. */ + fun neverAgain() { + CarSettings.setLocationWarning(false) + log.i("location warning turned off") + _carDismissed.value = true + _off.value = false + } + + private fun start() { + if (!armed) return + check() + // Turning location on clears the warning, and turning it off again brings it back. + val context = appContext ?: return + val r = object : BroadcastReceiver() { + override fun onReceive(context: Context, intent: Intent) = check() + } + context.registerReceiver(r, IntentFilter(LocationManager.MODE_CHANGED_ACTION), Context.RECEIVER_EXPORTED) + receiver = r + } + + private fun check() { + if (!armed) return + val manager = appContext?.getSystemService(LocationManager::class.java) ?: return + val off = CarSettings.locationWarning.value && !manager.isLocationEnabled + if (_off.value == off) return + _off.value = off + log.i(if (off) "the phone's location is off" else "the phone's location is on") + } + + private fun stopWatching() { + val r = receiver ?: return + receiver = null + runCatching { appContext?.unregisterReceiver(r) } + } + + private const val WAIT_MS = 10_000L +} diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/car/MediaScreen.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/car/MediaScreen.kt index 40fc17e..13cf66e 100644 --- a/gearslip/src/main/java/app/seb3thehacker/gearslip/car/MediaScreen.kt +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/car/MediaScreen.kt @@ -61,7 +61,6 @@ import androidx.compose.ui.text.font.FontWeight import androidx.compose.ui.text.style.TextOverflow import androidx.compose.ui.unit.dp import androidx.lifecycle.compose.collectAsStateWithLifecycle -import app.seb3thehacker.gearslip.audio.CarAudio import app.seb3thehacker.gearslip.media.CarMedia import app.seb3thehacker.gearslip.media.MediaApp import app.seb3thehacker.gearslip.media.MediaArt @@ -88,11 +87,9 @@ fun MediaScreen(app: MediaApp, onExit: () -> Unit) { val queue by media.queue.collectAsStateWithLifecycle() val browse by media.browse.collectAsStateWithLifecycle() val rejection by media.rejection.collectAsStateWithLifecycle() - val capture by CarAudio.capture.collectAsStateWithLifecycle() val search by media.search.collectAsStateWithLifecycle() var tab by remember { mutableStateOf(Tab.NOW_PLAYING) } var searching by remember { mutableStateOf(false) } - val searchAllowed = CarSettings.experimentalFeaturesEnabled.collectAsState().value && CarSettings.mediaSearchEnabled.collectAsState().value // The connection outlives this screen: the home screen's player is the same one. LaunchedEffect(app) { CarServices.openMedia(app) } @@ -174,7 +171,6 @@ fun MediaScreen(app: MediaApp, onExit: () -> Unit) { } else { CompactPlayer(now, media, Modifier.weight(1f).fillMaxWidth()) } - AudioBadge(capture) } Column(Modifier.weight(1.2f).fillMaxHeight()) { Row(Modifier.fillMaxWidth(), horizontalArrangement = Arrangement.spacedBy(8.dp)) { @@ -182,7 +178,7 @@ fun MediaScreen(app: MediaApp, onExit: () -> Unit) { MediaTab("Browse", tab == Tab.BROWSE, Modifier.weight(1f)) { tab = Tab.BROWSE } MediaTab("Up next", tab == Tab.QUEUE, Modifier.weight(1f)) { tab = Tab.QUEUE } MediaTab("Lyrics", tab == Tab.LYRICS, Modifier.weight(1f)) { tab = Tab.LYRICS } - if (searchAllowed && (now.canSearch || media.canSearchLibrary)) { + if (now.canSearch || media.canSearchLibrary) { GsIconButton( Icons.Filled.Search, "Search ${app.label}", @@ -447,17 +443,6 @@ private sealed interface LyricsState { class Found(val result: LyricsResult) : LyricsState } -@Composable -private fun AudioBadge(capture: CarAudio.Capture) { - val text = when (capture) { - CarAudio.Capture.ON -> "Audio to car" - CarAudio.Capture.ASKING -> "Allow audio capture on the phone" - CarAudio.Capture.DENIED -> "Audio stays on the phone" - CarAudio.Capture.OFF -> return - } - Text(text, style = MaterialTheme.typography.labelLarge, color = MaterialTheme.colorScheme.onSurfaceVariant) -} - @Composable private fun Notice(title: String, detail: String?, action: Pair Unit>? = null) { Column( diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/car/NavApps.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/car/NavApps.kt index ce706fe..0a50ca5 100644 --- a/gearslip/src/main/java/app/seb3thehacker/gearslip/car/NavApps.kt +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/car/NavApps.kt @@ -72,7 +72,7 @@ internal class BuiltInApp( ) internal object BuiltInApps { - val Web = BuiltInApp("web", "Web", Icons.Filled.Search, CarScreen.App("web")) { it.open("web") } + val Web = BuiltInApp("web", "Browser", Icons.Filled.Search, CarScreen.App("web")) { it.open("web") } val ScreenSharing = BuiltInApp("phone", "Screen sharing", Icons.Filled.Share, CarScreen.App("phone")) { it.open("phone") } val Phone = BuiltInApp("dialer", "Phone", Icons.Filled.Call, CarScreen.App("dialer")) { it.open("dialer") } val VehicleData = BuiltInApp("vehicle", "Vehicle data", Icons.Filled.Info, CarScreen.VehicleData) { it.vehicleData() } diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/car/PhoneDialerScreen.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/car/PhoneDialerScreen.kt index cb21b59..a7ba655 100644 --- a/gearslip/src/main/java/app/seb3thehacker/gearslip/car/PhoneDialerScreen.kt +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/car/PhoneDialerScreen.kt @@ -32,6 +32,7 @@ import androidx.compose.material3.MaterialTheme import androidx.compose.material3.Surface import androidx.compose.material3.Text import androidx.compose.runtime.Composable +import androidx.compose.runtime.collectAsState import androidx.compose.runtime.getValue import androidx.compose.runtime.mutableStateOf import androidx.compose.runtime.produceState @@ -89,6 +90,31 @@ internal fun placeCall(context: Context, number: String) { } } +/** A call another car screen wants to place - a number on a place page - waiting on the driver. */ +internal object CallPrompt { + val pending = kotlinx.coroutines.flow.MutableStateFlow(null) + + fun ask(name: String, number: String) { + pending.value = Contact(name, number) + } +} + +/** The dialer's own confirmation, drawn over the whole car screen for [CallPrompt]. */ +@Composable +internal fun CallPromptOverlay() { + val context = LocalContext.current + val contact by CallPrompt.pending.collectAsState() + val c = contact ?: return + ConfirmCallOverlay( + c, + onConfirm = { + CallPrompt.pending.value = null + placeCall(context, c.number) + }, + onDismiss = { CallPrompt.pending.value = null }, + ) +} + private const val TAB_CONTACTS = 0 private const val TAB_KEYPAD = 1 diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/car/PlayerParts.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/car/PlayerParts.kt index 987b89c..36915b6 100644 --- a/gearslip/src/main/java/app/seb3thehacker/gearslip/car/PlayerParts.kt +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/car/PlayerParts.kt @@ -138,29 +138,47 @@ internal fun PlayModeRow( val repeatOn = repeatKnown && modes.repeat != PlaybackStateCompat.REPEAT_MODE_NONE val icon = size * 0.52f Row(modifier, horizontalArrangement = Arrangement.SpaceEvenly, verticalAlignment = Alignment.CenterVertically) { - GsIconButton( - if (liked == true) Icons.Filled.Favorite else Icons.Filled.FavoriteBorder, - if (liked == true) "Unlike" else "Like", - media::toggleLike, - enabled = liked != null, - tone = if (liked == true) GsTone.Primary else GsTone.Neutral, - size = size, iconSize = icon, - ) - GsIconButton( - MediaIcons.Shuffle, if (shuffleOn) "Shuffle off" else "Shuffle on", - media::toggleShuffle, - enabled = shuffleKnown || now.shuffleAction != null, - tone = if (shuffleOn) GsTone.Primary else GsTone.Neutral, - size = size, iconSize = icon, - ) - GsIconButton( - if (modes.repeat == PlaybackStateCompat.REPEAT_MODE_ONE) MediaIcons.RepeatOne else MediaIcons.Repeat, - "Repeat", - media::cycleRepeat, - enabled = repeatKnown || now.repeatAction != null, - tone = if (repeatOn) GsTone.Primary else GsTone.Neutral, - size = size, iconSize = icon, - ) + // An app's own like, shuffle or repeat button shows whether it's on only through its icon, + // so that icon is drawn, as Android Auto does, and lit when its name says it's on. + // Gearslip's own stand in for apps that report a heart rating or shuffle and repeat modes. + val likeAction = now.likeAction + if (likeAction != null) { + CustomActionButton(likeAction, media, size, toggle = true) + } else { + GsIconButton( + if (liked == true) Icons.Filled.Favorite else Icons.Filled.FavoriteBorder, + if (liked == true) "Unlike" else "Like", + media::toggleLike, + enabled = liked != null, + tone = if (liked == true) GsTone.Primary else GsTone.Neutral, + size = size, iconSize = icon, + ) + } + val shuffleAction = now.shuffleAction + if (shuffleAction != null) { + CustomActionButton(shuffleAction, media, size, toggle = true) + } else { + GsIconButton( + MediaIcons.Shuffle, if (shuffleOn) "Shuffle off" else "Shuffle on", + media::toggleShuffle, + enabled = shuffleKnown, + tone = if (shuffleOn) GsTone.Primary else GsTone.Neutral, + size = size, iconSize = icon, + ) + } + val repeatAction = now.repeatAction + if (repeatAction != null) { + CustomActionButton(repeatAction, media, size, toggle = true) + } else { + GsIconButton( + if (modes.repeat == PlaybackStateCompat.REPEAT_MODE_ONE) MediaIcons.RepeatOne else MediaIcons.Repeat, + "Repeat", + media::cycleRepeat, + enabled = repeatKnown, + tone = if (repeatOn) GsTone.Primary else GsTone.Neutral, + size = size, iconSize = icon, + ) + } now.otherActions.take(maxOther).forEach { CustomActionButton(it, media, size) } trailing() } @@ -171,12 +189,12 @@ internal fun PlayModeRow( * not Gearslip's, so it's loaded from there. Its name stands in if the icon won't load. */ @Composable -private fun CustomActionButton(action: CustomAction, media: CarMedia, size: Dp) { +private fun CustomActionButton(action: CustomAction, media: CarMedia, size: Dp, toggle: Boolean = false) { val icon = rememberActionIcon(action) GsIconBox( onClick = { media.custom(action) }, modifier = Modifier.size(size), - colors = gsColors(GsTone.Neutral), + colors = gsColors(if (toggle && action.on) GsTone.Primary else GsTone.Neutral), ) { if (icon != null) { Icon(icon, action.name, Modifier.size(size * 0.52f)) diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/car/TemplateContent.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/car/TemplateContent.kt index cd3c04e..4b29ac9 100644 --- a/gearslip/src/main/java/app/seb3thehacker/gearslip/car/TemplateContent.kt +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/car/TemplateContent.kt @@ -45,6 +45,7 @@ import androidx.car.app.model.MessageTemplate import androidx.car.app.model.PaneTemplate import androidx.car.app.model.Row as CarRow import androidx.car.app.model.SearchTemplate +import androidx.compose.foundation.clickable import androidx.car.app.model.Section import androidx.car.app.model.SectionedItemTemplate import androidx.car.app.model.TabTemplate @@ -238,6 +239,9 @@ private fun SearchContent(template: SearchTemplate, modifier: Modifier) { // initialSearchText would fight whatever is being typed. var query by remember { mutableStateOf(template.initialSearchText.orEmpty()) } val delegate = template.searchCallbackDelegate + // An app that opens on results ("navigate to Starbucks") asks for the keyboard to stay down, + // so the results aren't hidden under it. Tapping the field brings it up. + var typing by remember { mutableStateOf(template.isShowKeyboardByDefault) } ContentSurface(modifier) { // Back button and field share a row: on a 480px screen the keyboard and the results @@ -251,7 +255,7 @@ private fun SearchContent(template: SearchTemplate, modifier: Modifier) { SearchField( query, template.searchHint.orEmpty(), - Modifier.weight(1f), + Modifier.weight(1f).clickable { typing = true }, onClear = { query = "" delegate.textChanged("") @@ -265,14 +269,17 @@ private fun SearchContent(template: SearchTemplate, modifier: Modifier) { else ItemListColumn(template.itemList, modifier = Modifier.fillMaxSize()) } - CarKeyboard( - text = query, - onTextChange = { - query = it - delegate.textChanged(it) - }, - onSubmit = { delegate.submitted(query) }, - ) + if (typing) { + CarKeyboard( + text = query, + onTextChange = { + query = it + delegate.textChanged(it) + }, + onSubmit = { delegate.submitted(query) }, + onDismiss = { typing = false }, + ) + } } } diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/car/TemplateParts.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/car/TemplateParts.kt index 0d07349..1111404 100644 --- a/gearslip/src/main/java/app/seb3thehacker/gearslip/car/TemplateParts.kt +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/car/TemplateParts.kt @@ -3,8 +3,10 @@ package app.seb3thehacker.gearslip.car import app.seb3thehacker.gearslip.car.theme.* import androidx.compose.runtime.produceState import androidx.compose.runtime.remember +import androidx.compose.ui.focus.onFocusChanged import androidx.compose.material3.LocalContentColor import androidx.compose.ui.graphics.ColorFilter +import androidx.compose.ui.graphics.luminance import androidx.compose.ui.graphics.ImageBitmap import androidx.compose.foundation.Image import kotlinx.coroutines.Dispatchers @@ -29,6 +31,7 @@ import androidx.compose.foundation.lazy.grid.items import androidx.compose.foundation.lazy.items import androidx.compose.foundation.shape.RoundedCornerShape import androidx.compose.material.icons.Icons +import androidx.compose.material.icons.filled.Call import androidx.compose.material.icons.filled.ArrowBack import androidx.compose.material.icons.filled.Menu import androidx.compose.material.icons.filled.Warning @@ -231,7 +234,9 @@ internal fun RightEdgeControls( // the first touch on one wakes them rather than pressing it. val awake = LocalMapControlsWake.current.awake val pan by CarServices.nav.panMode.collectAsState() - val visible = awake || pan + // Kept up while the knob is on one of them, or the ring would fade out along with the button. + var holdsFocus by remember { mutableStateOf(false) } + val visible = awake || pan || holdsFocus val alpha by androidx.compose.animation.core.animateFloatAsState( targetValue = if (visible) 1f else 0f, animationSpec = androidx.compose.animation.core.tween(if (visible) 150 else 500), @@ -240,6 +245,7 @@ internal fun RightEdgeControls( Column( modifier .alpha(alpha) + .onFocusChanged { holdsFocus = it.hasFocus } .let { base -> if (visible) base else base.pointerInput(Unit) { awaitPointerEventScope { @@ -468,7 +474,10 @@ internal fun CarGlyph(icon: CarIcon?, modifier: Modifier = Modifier, standard: I val white = remember(bitmap) { bitmap.isPlainWhite() } // Untinted white artwork is a template-app habit from dark surfaces; it is coloured like // text so it stays readable on a light one. - val tint = icon?.tint?.color(dark, content) ?: content.takeIf { white } + // An app's own tint is kept only if it can be seen: MAPS.ME asks for dark grey on dark + // buttons. Android Auto drops colours that fail contrast the same way. + val tint = icon?.tint?.color(dark, content)?.takeIf { it.readableOn(content) } + ?: content.takeIf { white || icon?.tint != null } Image( bitmap, contentDescription = null, modifier = modifier, colorFilter = tint?.let { ColorFilter.tint(it) }, @@ -587,16 +596,141 @@ internal fun PaneRows(pane: Pane?, modifier: Modifier = Modifier, autoStart: Boo // Some apps pad a pane with a row of blank strings; drawn, it is just a gap. val rows = pane?.rows.orEmpty().filterNot { it.isBlank() } val actions = pane?.actions.orEmpty().filter { it.isDrawable() } + val card = PlaceCard.of(rows) Column(modifier) { - rows.forEach { RowItem(it, large = true) } + card.heading?.let { PlaceHeading(it) } + if (card.buttons.isNotEmpty()) RowButtons(card.buttons, Modifier.padding(horizontal = 14.dp, vertical = 6.dp)) if (autoStart && actions.size == 1) { AutoStartButton(actions.single(), Modifier.padding(horizontal = 14.dp, vertical = 10.dp)) } else { ActionRow(actions, Modifier.padding(horizontal = 14.dp, vertical = 10.dp), large = true) } + if (card.heading == null && card.buttons.isEmpty()) { + card.details.forEach { RowItem(it, large = true) } + } else { + card.details.forEach { DetailRow(it, card.heading?.title?.text().orEmpty()) } + } } } +/** + * A pane split the way a place page reads: a name on top, the things to do with it as buttons, + * then the facts. Apps send all three as plain rows, so the split is worked out from the rows' + * shape alone. A pane that doesn't fit the shape - all of it tappable, like a menu - gets no + * heading and no buttons, and draws exactly as a plain list of rows. + */ +private class PlaceCard(val heading: CarRow?, val buttons: List, val details: List) { + companion object { + fun of(rows: List): PlaceCard { + // A short tappable row with an icon and nothing under it is a button in all but name: + // MAPS.ME's "Let's go" and "Save". + val buttons = rows.filter { + it.onClickDelegate != null && it.toggle == null && it.image != null && + it.texts.orEmpty().isEmpty() && it.title.text().length <= 24 + }.takeIf { it.size in 1..4 && it.size < rows.size }.orEmpty() + // A first row that can't be tapped and has no icon is the place's name and kind. + val heading = rows.firstOrNull()?.takeIf { + buttons.isNotEmpty() && it.onClickDelegate == null && it.image == null && it.toggle == null + } + return PlaceCard(heading, buttons, rows - buttons.toSet() - listOfNotNull(heading).toSet()) + } + } +} + +@Composable +private fun PlaceHeading(row: CarRow) { + Column(Modifier.fillMaxWidth().padding(horizontal = 14.dp, vertical = 6.dp)) { + Text(row.title.text(), style = ChromeType.title, maxLines = 2, overflow = TextOverflow.Ellipsis) + row.texts.orEmpty().take(2).forEach { + Text(it.text(), style = ChromeType.body, color = MaterialTheme.colorScheme.onSurfaceVariant, maxLines = 1, overflow = TextOverflow.Ellipsis) + } + } +} + +/** Button-shaped rows side by side, the first one filled as the thing most drivers came to do. */ +@Composable +private fun RowButtons(rows: List, modifier: Modifier) { + val scheme = MaterialTheme.colorScheme + Row(modifier.fillMaxWidth(), horizontalArrangement = Arrangement.spacedBy(8.dp)) { + rows.forEachIndexed { i, row -> + GsIconBox( + onClick = { row.onClickDelegate.click("row") }, + modifier = Modifier.weight(1f), + colors = if (i == 0) GsColors(scheme.primary, scheme.onPrimary) else GsColors(scheme.surfaceContainerHighest, scheme.onSurface), + enabled = row.isEnabled, + shape = RoundedCornerShape(24.dp), + ) { + Row( + Modifier.fillMaxWidth().padding(horizontal = 16.dp, vertical = 14.dp), + horizontalArrangement = Arrangement.Center, + verticalAlignment = Alignment.CenterVertically, + ) { + CarGlyph(row.image, Modifier.size(24.dp)) + Spacer(Modifier.width(8.dp)) + Text(row.title.text(), style = ChromeType.body, maxLines = 1, overflow = TextOverflow.Ellipsis) + } + } + } + } +} + +/** + * One fact about a place, tighter than a list row: a small label over its value, or just the + * value when the row has no label. A phone number gets a Call button that asks first. + */ +@Composable +private fun DetailRow(row: CarRow, placeName: String) { + val context = LocalContext.current + val texts = row.texts.orEmpty().map { it.text() }.filter { it.isNotBlank() } + val label = row.title.text().takeIf { texts.isNotEmpty() } + val values = texts.ifEmpty { listOf(row.title.text()) } + val number = values.firstOrNull { it.isPhoneNumber() } + val canCall = number != null && + context.checkSelfPermission(android.Manifest.permission.CALL_PHONE) == android.content.pm.PackageManager.PERMISSION_GRANTED + Row( + Modifier + .fillMaxWidth() + .let { if (row.onClickDelegate != null && row.isEnabled) it.clickable { row.onClickDelegate.click("row") } else it } + .padding(horizontal = 14.dp, vertical = 6.dp), + verticalAlignment = Alignment.CenterVertically, + ) { + row.image?.let { + CarGlyph(it, Modifier.size(24.dp)) + Spacer(Modifier.width(12.dp)) + } + Column(Modifier.weight(1f)) { + label?.let { + Text(it, style = MaterialTheme.typography.labelLarge, color = MaterialTheme.colorScheme.onSurfaceVariant, maxLines = 1) + } + values.take(3).forEach { + Text(it, style = ChromeType.body, maxLines = 2, overflow = TextOverflow.Ellipsis) + } + } + if (canCall) { + Spacer(Modifier.width(8.dp)) + GsIconBox( + onClick = { CallPrompt.ask(placeName.ifBlank { number!! }, number!!) }, + colors = GsColors(MaterialTheme.colorScheme.primary, MaterialTheme.colorScheme.onPrimary), + shape = RoundedCornerShape(24.dp), + ) { + Row(Modifier.padding(horizontal = 16.dp, vertical = 10.dp), verticalAlignment = Alignment.CenterVertically) { + androidx.compose.material3.Icon(Icons.Filled.Call, null, Modifier.size(20.dp)) + Spacer(Modifier.width(6.dp)) + Text("Call", style = ChromeType.body) + } + } + } + ActionRow(row.actions.orEmpty()) + } +} + +/** A line that is only a phone number: digits with the usual separators, 7 to 15 digits long. */ +private fun String.isPhoneNumber(): Boolean { + val t = trim() + if (!Regex("""^\+?[0-9 ().\-]+$""").matches(t)) return false + return t.count { it.isDigit() } in 7..15 +} + @Composable internal fun RowItem(row: CarRow, selection: RowSelection? = null, large: Boolean = false) { val toggle = row.toggle @@ -661,6 +795,17 @@ internal fun RowItem(row: CarRow, selection: RowSelection? = null, large: Boolea /** A row's place in a single-choice list: whether it is the chosen one, and what picking it does. */ internal class RowSelection(val selected: Boolean, val onPick: () -> Unit) +/** + * Whether this colour stands out against the background that [content] was chosen for: at least + * 3:1, the WCAG minimum for icons. The background is taken as the opposite of the content colour. + */ +private fun Color.readableOn(content: Color): Boolean { + val background = if (content.luminanceIsDark()) Color.White else Color(0xFF121212) + val a = luminance() + 0.05f + val b = background.luminance() + 0.05f + return maxOf(a, b) / minOf(a, b) >= 3f +} + internal fun Color.luminanceIsDark(): Boolean = (red * 0.299f + green * 0.587f + blue * 0.114f) < 0.5f diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/car/VoiceHelpOverlay.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/car/VoiceHelpOverlay.kt new file mode 100644 index 0000000..5257a31 --- /dev/null +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/car/VoiceHelpOverlay.kt @@ -0,0 +1,94 @@ +package app.seb3thehacker.gearslip.car + +import androidx.compose.foundation.layout.Arrangement +import androidx.compose.foundation.layout.Column +import androidx.compose.foundation.layout.Row +import androidx.compose.foundation.layout.fillMaxWidth +import androidx.compose.foundation.layout.padding +import androidx.compose.foundation.layout.widthIn +import androidx.compose.foundation.rememberScrollState +import androidx.compose.foundation.verticalScroll +import androidx.compose.material.icons.Icons +import androidx.compose.material.icons.filled.Close +import androidx.compose.material3.MaterialTheme +import androidx.compose.material3.Surface +import androidx.compose.material3.Text +import androidx.compose.runtime.Composable +import androidx.compose.runtime.LaunchedEffect +import androidx.compose.runtime.collectAsState +import androidx.compose.runtime.getValue +import androidx.compose.ui.Modifier +import androidx.compose.ui.text.font.FontWeight +import androidx.compose.ui.unit.dp +import app.seb3thehacker.gearslip.car.theme.GsIconButton +import kotlinx.coroutines.delay +import kotlinx.coroutines.flow.MutableStateFlow +import kotlinx.coroutines.flow.StateFlow +import kotlinx.coroutines.flow.asStateFlow + +/** What the assistant understands, shown on the car screen when the driver says "help". */ +object VoiceHelp { + private val _shown = MutableStateFlow(false) + val shown: StateFlow = _shown.asStateFlow() + + fun show() { _shown.value = true } + fun hide() { _shown.value = false } + + /** Say this, and this happens. Kept in step with [AssistantCommands]. */ + val COMMANDS = listOf( + "Navigate to…" to "Directions there. Add \"with Organic Maps\" to pick the app", + "Play…" to "A song, artist, or album in the open music app", + "Pause · Resume · Next · Previous" to "Control the music", + "What's playing" to "The song and artist", + "Call…" to "A contact by name", + "Open…" to "Any app, like Browser, Phone, or Screen sharing", + "Exit" to "Back to the car's own screen", + "Go home · Open apps · Settings" to "Move around Gearslip", + "What time is it · Weather" to "The time, or the forecast", + "Help" to "This list", + ) +} + +/** The command list over the car screen: read it at a glance, tap the cross or wait to dismiss. */ +@Composable +fun VoiceHelpOverlay(modifier: Modifier = Modifier) { + val shown by VoiceHelp.shown.collectAsState() + if (!shown) return + // A list left up would cover the map for the rest of the drive. + LaunchedEffect(Unit) { + delay(30_000) + VoiceHelp.hide() + } + Surface( + color = MaterialTheme.colorScheme.surfaceContainerHigh, + contentColor = MaterialTheme.colorScheme.onSurface, + shape = MaterialTheme.shapes.extraLarge, + tonalElevation = 6.dp, + shadowElevation = 8.dp, + modifier = modifier.padding(16.dp).widthIn(max = 640.dp), + ) { + Column(Modifier.padding(20.dp)) { + Row(Modifier.fillMaxWidth()) { + Text( + "You can say", + style = MaterialTheme.typography.titleLarge, + fontWeight = FontWeight.Bold, + modifier = Modifier.weight(1f).padding(top = 6.dp), + ) + GsIconButton(Icons.Filled.Close, "Close", VoiceHelp::hide) + } + // Full width, so a drag anywhere on the card scrolls, not just one on the words. + Column( + Modifier.fillMaxWidth().verticalScroll(rememberScrollState()).padding(top = 8.dp), + verticalArrangement = Arrangement.spacedBy(10.dp), + ) { + VoiceHelp.COMMANDS.forEach { (say, does) -> + Column(Modifier.fillMaxWidth()) { + Text(say, style = MaterialTheme.typography.titleMedium, fontWeight = FontWeight.SemiBold) + Text(does, style = MaterialTheme.typography.bodyMedium, color = MaterialTheme.colorScheme.onSurfaceVariant) + } + } + } + } + } +} diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/car/WebApp.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/car/WebApp.kt index 868fbb7..54cbd47 100644 --- a/gearslip/src/main/java/app/seb3thehacker/gearslip/car/WebApp.kt +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/car/WebApp.kt @@ -12,6 +12,7 @@ import androidx.compose.foundation.layout.Column import androidx.compose.foundation.layout.fillMaxSize import androidx.compose.ui.unit.dp import androidx.compose.foundation.layout.padding +import androidx.compose.material3.MaterialTheme import androidx.compose.runtime.Composable import androidx.compose.runtime.DisposableEffect import androidx.compose.runtime.getValue @@ -19,6 +20,7 @@ import androidx.compose.runtime.mutableStateOf import androidx.compose.runtime.remember import androidx.compose.runtime.setValue import androidx.compose.ui.Modifier +import androidx.compose.ui.graphics.toArgb import androidx.compose.ui.viewinterop.AndroidView import app.seb3thehacker.gearslip.GearslipLog import org.json.JSONObject @@ -38,6 +40,7 @@ import org.json.JSONObject fun WebApp(content: String) { val navigator = LocalCarNavigator.current val darkOutside = LocalDarkOutside.current + val ring = String.format("#%06X", MaterialTheme.colorScheme.primary.toArgb() and 0xFFFFFF) var web by remember { mutableStateOf(null) } var fieldText by remember { mutableStateOf(null) } @@ -56,7 +59,7 @@ fun WebApp(content: String) { Column(Modifier.fillMaxSize()) { // Back steps through the page's history first (the interceptor above), then leaves. - ScreenHeader("Web", Modifier.padding(horizontal = 16.dp, vertical = 8.dp)) + ScreenHeader("Browser", Modifier.padding(horizontal = 16.dp, vertical = 8.dp)) AndroidView( modifier = Modifier.weight(1f).fillMaxSize(), factory = { context -> @@ -65,7 +68,7 @@ fun WebApp(content: String) { settings.domStorageEnabled = true settings.loadWithOverviewMode = true settings.useWideViewPort = true - webViewClient = LoggingClient() + webViewClient = LoggingClient(ring) addJavascriptInterface( FieldFocusBridge(onFocus = { fieldText = it }, onBlur = { fieldText = null }), "GearslipField", @@ -99,10 +102,11 @@ fun WebApp(content: String) { } } -private class LoggingClient : WebViewClient() { +private class LoggingClient(private val ring: String) : WebViewClient() { override fun onPageFinished(view: WebView?, url: String?) { GearslipLog.i("car web page loaded: $url") view?.evaluateJavascript(FOCUS_TRACKING_JS, null) + view?.evaluateJavascript(focusRingJs(ring), null) } // onPageFinished fires for error pages too, so without this a blocked load (missing @@ -170,6 +174,20 @@ private const val SUBMIT_FIELD_JS = """ })(); """ +/** + * The same ring the rest of the car UI shows, on whatever link or field the knob is on. Many + * sites hide the browser's own focus outline, which leaves a knob driver with no idea where they are. + */ +private fun focusRingJs(color: String) = """ + (function() { + if (document.getElementById('gearslip-focus-ring')) return; + var style = document.createElement('style'); + style.id = 'gearslip-focus-ring'; + style.textContent = ':focus-visible { outline: 3px solid $color !important; outline-offset: 2px !important; }'; + (document.head || document.documentElement).appendChild(style); + })(); +""" + /** Reports focus/blur on text inputs, textareas and contenteditable elements to [FieldFocusBridge]. */ private const val FOCUS_TRACKING_JS = """ (function() { diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/car/theme/GsHoldMarker.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/car/theme/GsHoldMarker.kt index 44cafd3..7488389 100644 --- a/gearslip/src/main/java/app/seb3thehacker/gearslip/car/theme/GsHoldMarker.kt +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/car/theme/GsHoldMarker.kt @@ -50,6 +50,8 @@ enum class HoldMarkerStyle { * colour, so the peel's flap matches whatever it sits on. Round and pill-shaped controls get the * dot whatever the theme says: a circle has no corner to peel. */ +private const val SHOW_HOLD_DOT = false + @Composable internal fun BoxScope.GsHoldMarker(style: HoldMarkerStyle, container: Color, shape: Shape, press: InteractionSource) { if (style == HoldMarkerStyle.NONE) return @@ -68,8 +70,9 @@ internal fun BoxScope.GsHoldMarker(style: HoldMarkerStyle, container: Color, sha val side = min(size.width, size.height) val r = (shape as? CornerBasedShape)?.topEnd?.toPx(size, this) ?: 0f val round = r >= side * 0.4f + // The dot is off until it's redesigned: flat themes and round controls show nothing. if (style == HoldMarkerStyle.DOT || round) { - drawHoldDot(r, side, ink) + if (SHOW_HOLD_DOT) drawHoldDot(r, side, ink) } else { // Drawn for the bottom-right corner, then flipped up to the top-right. scale(1f, -1f, pivot = center) { diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/car/theme/GsTheme.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/car/theme/GsTheme.kt index e16aeab..2f284fe 100644 --- a/gearslip/src/main/java/app/seb3thehacker/gearslip/car/theme/GsTheme.kt +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/car/theme/GsTheme.kt @@ -112,7 +112,7 @@ object GsThemes { control = ControlLook(holdMarker = HoldMarkerStyle.DOT), ) - val default = Tactile + val default = Flat val builtIn = listOf(Tactile, Keycap, Flat) fun byId(id: String?): GsTheme = builtIn.firstOrNull { it.id == id } ?: default diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/host/AutoRoute.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/host/AutoRoute.kt new file mode 100644 index 0000000..78920c7 --- /dev/null +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/host/AutoRoute.kt @@ -0,0 +1,98 @@ +package app.seb3thehacker.gearslip.host + +import android.os.SystemClock +import android.text.Spanned +import androidx.car.app.model.CarText +import androidx.car.app.model.DistanceSpan +import androidx.car.app.model.ItemList +import androidx.car.app.model.ListTemplate +import androidx.car.app.model.PaneTemplate +import androidx.car.app.model.PlaceListMapTemplate +import androidx.car.app.model.Row +import androidx.car.app.model.SearchTemplate +import androidx.car.app.model.Template +import androidx.car.app.navigation.model.MapTemplate +import androidx.car.app.navigation.model.MapWithContentTemplate +import androidx.car.app.navigation.model.NavigationTemplate +import androidx.car.app.navigation.model.PlaceListNavigationTemplate +import androidx.car.app.navigation.model.RoutePreviewNavigationTemplate +import app.seb3thehacker.gearslip.GearslipLog + +/** + * "Navigate to X" the way Android Auto does it: straight onto a route, no list to pick from. + * + * Android Auto looks the place up on Google's servers first. Gearslip stays on the phone, so it + * asks the map app to search, then taps the nearest result in the app's list for the driver. + * It stops at the app's start screen, so the driver sees where it's going and taps Start. It + * gives up after [WINDOW_MS], or as soon as the driver taps anything themselves. + */ +internal class AutoRoute { + private val log = GearslipLog.tagged("route") + private var until = 0L + private var pickedRow = false + + /** A destination was just sent to the map app. */ + fun arm() { + armedAt = SystemClock.elapsedRealtime() + until = armedAt + WINDOW_MS + pickedRow = false + } + + fun disarm() { + until = 0L + } + + private var armedAt = 0L + private val armed get() = SystemClock.elapsedRealtime() < until && driverTappedAt < armedAt + + /** Each screen the map app sends while armed: tap the next step if it's there. */ + @Suppress("DEPRECATION") // Apps still send the older navigation templates. + fun onTemplate(template: Template) { + if (!armed) return + when (template) { + is NavigationTemplate -> disarm() + // The start screen: the driver takes it from here. + is RoutePreviewNavigationTemplate -> arrived() + is MapWithContentTemplate -> onTemplate(template.contentTemplate) + is PaneTemplate -> arrived() + is MapTemplate -> if (template.pane != null) arrived() else template.itemList?.let(::onList) + is SearchTemplate -> if (!template.isLoading) template.itemList?.let(::onList) + is ListTemplate -> if (!template.isLoading) template.singleList?.let(::onList) + is PlaceListMapTemplate -> if (!template.isLoading) template.itemList?.let(::onList) + is PlaceListNavigationTemplate -> if (!template.isLoading) template.itemList?.let(::onList) + } + } + + /** Search results: the nearest place, or the first when the app gives no distances. */ + private fun onList(list: ItemList) { + if (pickedRow) return + val places = list.items.filterIsInstance() + .filter { it.isEnabled && it.onClickDelegate != null && !it.isBrowsable } + if (places.isEmpty()) return + val nearest = places.minByOrNull { it.meters() ?: Double.MAX_VALUE } ?: return + pickedRow = true + log.i("picking the nearest of ${places.size} results") + nearest.onClickDelegate.click(AUTO_LABEL) + } + + private fun arrived() { + if (pickedRow) log.i("at the start screen") + disarm() + } + + /** The first distance the app attached to this row's title or text lines. */ + private fun Row.meters(): Double? = (listOfNotNull(title) + texts).firstNotNullOfOrNull { it.distanceMeters() } + + private fun CarText.distanceMeters(): Double? { + val chars = toCharSequence() as? Spanned ?: return null + return chars.getSpans(0, chars.length, DistanceSpan::class.java).firstOrNull()?.distance?.meters() + } + + companion object { + /** Taps sent by this class carry this label, so they don't count as the driver's. */ + const val AUTO_LABEL = "auto-route" + /** When the driver last tapped one of the app's buttons; that hands control back. */ + @Volatile var driverTappedAt = 0L + private const val WINDOW_MS = 30_000L + } +} diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/host/CarAppCatalog.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/host/CarAppCatalog.kt index a9b2ca6..31ad0db 100644 --- a/gearslip/src/main/java/app/seb3thehacker/gearslip/host/CarAppCatalog.kt +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/host/CarAppCatalog.kt @@ -78,5 +78,7 @@ object CarAppCatalog { CarAppService.CATEGORY_SETTINGS_APP, CarAppService.CATEGORY_IOT_APP, CarAppService.CATEGORY_WEATHER_APP, + // Newer apps declare MEDIA, but our AndroidX 1.7 dependency has no constant for it. + "androidx.car.app.category.MEDIA", ) } diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/host/CarAppConnection.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/host/CarAppConnection.kt index a332887..0be82f8 100644 --- a/gearslip/src/main/java/app/seb3thehacker/gearslip/host/CarAppConnection.kt +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/host/CarAppConnection.kt @@ -1,6 +1,7 @@ package app.seb3thehacker.gearslip.host import app.seb3thehacker.gearslip.car.CarEnvironment +import app.seb3thehacker.gearslip.car.CarToasts import android.content.ComponentName import android.content.Context import android.content.Intent @@ -9,6 +10,7 @@ import android.content.res.Configuration import android.graphics.Rect import android.location.Location import android.net.Uri +import android.os.Build import android.os.Handler import android.os.IBinder import android.os.Looper @@ -80,6 +82,8 @@ class CarAppConnection(private val context: Context) { val template: StateFlow = _template.asStateFlow() private val main = Handler(Looper.getMainLooper()) + /** Captured by each host Binder so callbacks queued by a replaced app cannot show a toast. */ + private var toastOwner = Any() private var carApp: ICarApp? = null private var appManager: IAppManager? = null @@ -139,7 +143,11 @@ class CarAppConnection(private val context: Context) { // Replaced by another connect (or a disconnect) while waiting on the location service. if (binding !== connection) return@bind var bindError: Throwable? = null - val flags = Context.BIND_AUTO_CREATE or LocationKeepAlive.BIND_INCLUDE_CAPABILITIES + // BIND_ALLOW_ACTIVITY_STARTS lets the app start its own location service while + // Gearslip's car window is showing. Since Android 14 a bound app only inherits that + // from a visible host that opts in, and MapQuest crashes as navigation starts without it. + val flags = Context.BIND_AUTO_CREATE or LocationKeepAlive.BIND_INCLUDE_CAPABILITIES or + (if (Build.VERSION.SDK_INT >= 34) Context.BIND_ALLOW_ACTIVITY_STARTS else 0) val bound = runCatching { // Without BIND_FOREGROUND_SERVICE, an app bound by a foreground service (which is // all Gearslip is once the phone screen leaves it) only reaches "important @@ -215,6 +223,7 @@ class CarAppConnection(private val context: Context) { fun disconnect() = disconnect(keepLocation = false) private fun disconnect(keepLocation: Boolean) { + clearToasts() lent?.let { sendSurfaceDestroyed(it) } lent = null val carApp = this.carApp @@ -231,6 +240,10 @@ class CarAppConnection(private val context: Context) { this.carApp = null appManager = null navigationManager = null + navigating = false + main.removeCallbacks(sendReroute) + reroute = null + autoRoute.disarm() _panMode.value = false surfaceCallback = null component = null @@ -281,34 +294,66 @@ class CarAppConnection(private val context: Context) { private fun onHandshakeAccepted() { val app = carApp ?: return GearslipLog.i("host: handshake accepted; creating the app") - val intent = pendingDestination?.let { navigateIntent(it) } + val intent = pendingDestination?.let { autoRoute.arm(); navigateIntent(it) } ?: Intent(Intent.ACTION_MAIN).setComponent(component) pendingDestination = null call("onAppCreate", onValue = { onCreated() }) { - app.onAppCreate(carHost, intent, carConfiguration(), it) + app.onAppCreate(createCarHost(toastOwner), intent, carConfiguration(), it) } } /** * "navigate to X" deep link, the same one Android Auto's own Assistant integration sends - - * [ACTION_NAVIGATE] with the destination as a `geo:` search query, rather than anything - * app-specific. + * [ACTION_NAVIGATE] with the destination as a `geo:0,0` search query, rather than anything + * app-specific. The app searches near its own location; [AutoRoute] then picks the nearest + * result. Coordinates in the link would mean "go exactly here": Organic Maps routed to the + * phone's own position when it was given them. */ - private fun navigateIntent(destination: String) = + private fun navigateIntent(destination: String): Intent = Intent(CarContext.ACTION_NAVIGATE, Uri.parse("geo:0,0?q=" + Uri.encode(destination))) .setComponent(component) /** * Sends a fresh destination to an app that's already running, instead of tearing it down * and rebuilding the whole session - see [app.seb3thehacker.gearslip.car.CarServices.connectNav]. + * + * An app already on a route ends it first: Organic Maps ignored a new destination mid-route + * and kept the old directions. The new one goes once the app says the route ended, or after + * [REROUTE_WAIT_MS] if it never does. */ fun navigateTo(destination: String) { + if (carApp == null) return + main.removeCallbacks(sendReroute) + if (!navigating) { + reroute = null + sendDestination(destination) + return + } + reroute = destination + stopNavigating() + main.postDelayed(sendReroute, REROUTE_WAIT_MS) + } + + private fun sendDestination(destination: String) { val app = carApp ?: return + autoRoute.arm() call("onNewIntent", onValue = {}) { app.onNewIntent(navigateIntent(destination), it) } } + private val autoRoute = AutoRoute() + + /** A destination waiting for the current route to end; see [navigateTo]. */ + private var reroute: String? = null + private val sendReroute = Runnable { + reroute?.let { reroute = null; sendDestination(it) } + } + + /** Whether the app has said it's on a route, between navigationStarted and navigationEnded. */ + @Volatile var navigating = false + private set + private fun onCreated() { val app = carApp ?: return GearslipLog.i("host: created; starting") @@ -380,6 +425,7 @@ class CarAppConnection(private val context: Context) { GearslipLog.i("host: template = ${template.javaClass.simpleName}") _template.value = template update { it.copy(template = template.javaClass.simpleName) } + autoRoute.onTemplate(template) }) { manager.getTemplate(it) } } @@ -558,14 +604,19 @@ class CarAppConnection(private val context: Context) { // --- host interfaces the app calls back into ----------------------------------------- - private val appHost = object : IAppHost.Stub() { + private fun createAppHost(owner: Any) = object : IAppHost.Stub() { override fun invalidate() { GearslipLog.i("host: app invalidated its template") main.post { requestTemplate() } } override fun showToast(text: CharSequence?, duration: Int) { - GearslipLog.i("host: toast \"$text\"") + val message = text?.toString() ?: return + main.post { + if (toastOwner !== owner) return@post + GearslipLog.i("host: toast \"$message\"") + CarToasts.show(owner, message, duration) + } } /** @@ -587,10 +638,24 @@ class CarAppConnection(private val context: Context) { override fun openMicrophone(request: Bundleable?): Bundleable? = null } + private val _trip = MutableStateFlow(null) + /** The route as the app tells clusters about it: steps and how far to each. Null when not navigating. */ + val trip: StateFlow = _trip.asStateFlow() + private val navigationHost = object : INavigationHost.Stub() { - override fun navigationStarted() = GearslipLog.i("host: navigation started") - override fun navigationEnded() = GearslipLog.i("host: navigation ended") - override fun updateTrip(trip: Bundleable?) = Unit + override fun navigationStarted() { + GearslipLog.i("host: navigation started") + navigating = true + } + override fun navigationEnded() { + GearslipLog.i("host: navigation ended") + navigating = false + _trip.value = null + main.post { if (reroute != null) { main.removeCallbacks(sendReroute); sendReroute.run() } } + } + override fun updateTrip(trip: Bundleable?) { + _trip.value = runCatching { trip?.get() as? androidx.car.app.navigation.model.Trip }.getOrNull() + } } /** @@ -656,7 +721,9 @@ class CarAppConnection(private val context: Context) { override fun updateSuggestions(suggestions: Bundleable?) = Unit } - private val carHost = object : ICarHost.Stub() { + private fun createCarHost(owner: Any) = object : ICarHost.Stub() { + private val appHost = createAppHost(owner) + override fun getHost(type: String?): IBinder? = when (type) { CarContext.APP_SERVICE -> appHost.asBinder() CarContext.NAVIGATION_SERVICE -> navigationHost.asBinder() @@ -745,10 +812,16 @@ class CarAppConnection(private val context: Context) { } private fun fail(phase: Phase, detail: String) { + clearToasts() GearslipLog.e("host: $detail") update { it.copy(phase = phase, detail = detail) } } + private fun clearToasts() { + CarToasts.clear(toastOwner) + toastOwner = Any() + } + private fun update(block: (Status) -> Status) { _status.value = block(_status.value) } @@ -757,6 +830,9 @@ class CarAppConnection(private val context: Context) { /** The newest template API this host claims to understand. */ const val HOST_API_LEVEL = CarAppApiLevels.LEVEL_8 + /** How long a new destination waits for the old route to end; see [navigateTo]. */ + const val REROUTE_WAIT_MS = 1_500L + /** Context.BIND_FOREGROUND_SERVICE; the constant itself is hidden from the SDK. */ const val BIND_FOREGROUND_SERVICE = 0x04000000 } diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/host/CarModels.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/host/CarModels.kt index 5db0845..f4fc8ed 100644 --- a/gearslip/src/main/java/app/seb3thehacker/gearslip/host/CarModels.kt +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/host/CarModels.kt @@ -141,7 +141,7 @@ fun Distance?.display(): String { return "%.${decimals}f $unit".format(distance.displayDistance) } -private fun Distance.meters(): Double = displayDistance * when (displayUnit) { +internal fun Distance.meters(): Double = displayDistance * when (displayUnit) { Distance.UNIT_KILOMETERS, Distance.UNIT_KILOMETERS_P1 -> 1000.0 Distance.UNIT_MILES, Distance.UNIT_MILES_P1 -> 1609.344 Distance.UNIT_FEET -> 0.3048 @@ -195,6 +195,7 @@ fun androidx.car.app.model.DateTimeWithZone?.clockTime(): String { */ fun OnClickDelegate?.click(label: String = "action") { val delegate = this ?: return + if (label != AutoRoute.AUTO_LABEL) AutoRoute.driverTappedAt = android.os.SystemClock.elapsedRealtime() runCatching { delegate.sendClick(doneCallback("a $label tap")) } .onFailure { GearslipLog.w("host: could not send a $label tap: ${it.message}") } } diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/host/KnownApps.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/host/KnownApps.kt index 79caa60..74289ee 100644 --- a/gearslip/src/main/java/app/seb3thehacker/gearslip/host/KnownApps.kt +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/host/KnownApps.kt @@ -18,6 +18,8 @@ object KnownApps { "com.vivi.vivimusic", // Vivi Music: browse, play, seek, works well "com.metrolist.music", // Metrolist: browse, play, works "nl.flitsmeister", // Flitsmeister: map, routing, alerts - works well + "com.mapquest.android.ace", // MapQuest: map, routing, navigation + "com.mapswithme.maps.pro", // MAPS.ME: map, search, place pages, navigation ) fun works(packageName: String): Boolean = packageName in working @@ -71,7 +73,6 @@ object KnownApps { */ private val partial = setOf( "app.vela", // Vela Maps: connects and draws, but rough enough not to call it working yet - "com.mapquest.android.ace", // MapQuest: map, search and routing run, but not cleanly "com.spotify.music", // Spotify: plays through Gearslip's player, but the app has to be opened on the phone first "deezer.android.app", // Deezer: the same - it has to be opened on the phone first ) diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/media/CarMedia.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/media/CarMedia.kt index 056621d..dd0d9a2 100644 --- a/gearslip/src/main/java/app/seb3thehacker/gearslip/media/CarMedia.kt +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/media/CarMedia.kt @@ -40,7 +40,19 @@ class MediaEntry( ) /** [icon] is a drawable id inside the media app's own package, not Gearslip's; 0 for none. */ -class CustomAction(val id: String, val name: String, val extras: Bundle?, val icon: Int = 0) +class CustomAction(val id: String, val name: String, val extras: Bundle?, val icon: Int = 0, val iconName: String = "") { + /** + * Whether a toggle like heart or shuffle is on, read from the icon's name - Tidal swaps + * ic_favorite for ic_favorite_filled and ic_shuffle for ic_shuffle_active, and keeps the + * button's name the same. Whole words only, so "inactive" doesn't count as "active". + */ + val on: Boolean + get() = iconName.lowercase().split('_', '.', '-').any { it in ON_WORDS } + + private companion object { + val ON_WORDS = setOf("filled", "fill", "active", "on", "selected", "checked", "enabled", "solid", "one", "all") + } +} /** One entry in the app's own queue - not its library, whatever it has lined up to play next. */ class QueueTrack( @@ -156,6 +168,9 @@ class CarMedia(private val context: Context) { private val main = Handler(Looper.getMainLooper()) private var connecting: MediaApp? = null + /** The music app Gearslip is attached to, for the assistant to name; null when there's none. */ + val appLabel: String? get() = connecting?.label?.takeIf { controller != null } + private val _phase = MutableStateFlow(Phase.IDLE) val phase: StateFlow = _phase.asStateFlow() @@ -225,7 +240,11 @@ class CarMedia(private val context: Context) { private fun refreshModes() { val c = compat - _modes.value = if (c == null) Modes() else Modes(shuffle = c.shuffleMode, repeat = c.repeatMode) + val modes = if (c == null) Modes() else Modes(shuffle = c.shuffleMode, repeat = c.repeatMode) + if (modes.shuffle != _modes.value.shuffle || modes.repeat != _modes.value.repeat) { + GearslipLog.i("media: shuffle mode ${modes.shuffle}, repeat mode ${modes.repeat}, rating type ${controller?.ratingType}") + } + _modes.value = modes } private var subscribed: String? = null private var rootId: String? = null @@ -236,6 +255,12 @@ class CarMedia(private val context: Context) { connecting = app _phase.value = Phase.CONNECTING _browse.value = BrowseState() + // Known to refuse its library (Spotify): asking only costs the seconds it takes to say no, + // so go straight to its playback session. + if (KnownApps.playerWorks(app.component.packageName)) { + attachToSession(app) + return + } val mb = MediaBrowser(context, app.component, object : MediaBrowser.ConnectionCallback() { override fun onConnected() = this@CarMedia.onConnected(app) override fun onConnectionFailed() { @@ -248,10 +273,20 @@ class CarMedia(private val context: Context) { } }, null) browser = mb + GearslipLog.i("media: connecting to ${app.label}'s library") runCatching { mb.connect() }.onFailure { GearslipLog.e("media: connect threw", it) _phase.value = Phase.REJECTED } + // An app that never answers would leave the player spinning, and every later tap on it + // would only reopen that spinner. Give up on the library and drive its playback instead. + main.postDelayed({ + if (browser !== mb || _phase.value != Phase.CONNECTING) return@postDelayed + GearslipLog.w("media: ${app.label} didn't answer in ${CONNECT_WAIT_MS / 1000} s, using its playback session") + runCatching { mb.disconnect() } + browser = null + attachToSession(app) + }, CONNECT_WAIT_MS) } /** @@ -273,7 +308,12 @@ class CarMedia(private val context: Context) { return } if (session == null) { - val limit = if (launched) SESSION_RETRIES_LAUNCHED else SESSION_RETRIES + // A known session-only app that isn't running won't start by waiting: say so at once. + val limit = when { + launched -> SESSION_RETRIES_LAUNCHED + KnownApps.playerWorks(app.component.packageName) -> 0 + else -> SESSION_RETRIES + } // The app only publishes its session once its service has started; give it a moment. if (attempt < limit) { main.postDelayed({ attachToSession(app, attempt + 1, launched) }, SESSION_RETRY_MS) @@ -342,16 +382,39 @@ class CarMedia(private val context: Context) { speed = state?.playbackSpeed ?: 1f, updatedAt = state?.lastPositionUpdateTime ?: 0, actions = state?.actions ?: 0, - custom = state?.customActions.orEmpty().map { CustomAction(it.action, it.name.toString(), it.extras, it.icon) }, + custom = state?.customActions.orEmpty().map { CustomAction(it.action, it.name.toString(), it.extras, it.icon, iconName(it.icon)) }, error = state?.errorMessage?.toString(), activeQueueItemId = state?.activeQueueItemId ?: MediaSession.QueueItem.UNKNOWN_ID.toLong(), heart = if (controller?.ratingType == Rating.RATING_HEART) { metadata?.getRating(MediaMetadata.METADATA_KEY_USER_RATING)?.takeIf { it.isRated }?.hasHeart() ?: false } else null, ) + logButtons() checkSearch() } + private var appResources: Pair? = null + + /** The icon's resource name in the app's package, or "" when it can't be read. */ + private fun iconName(icon: Int): String { + val pkg = controller?.packageName ?: return "" + if (appResources?.first != pkg) { + appResources = pkg to runCatching { context.packageManager.getResourcesForApplication(pkg) }.getOrNull() + } + val res = appResources?.second ?: return "" + return runCatching { res.getResourceEntryName(icon) }.getOrDefault("") + } + + private var loggedButtons: String? = null + + /** The app's own buttons, by id, name and icon, each time they change. */ + private fun logButtons() { + val text = _now.value.custom.joinToString { a -> "${a.id} \"${a.name}\" icon=${a.iconName.ifEmpty { a.icon.toString() }}" } + if (text == loggedButtons) return + loggedButtons = text + GearslipLog.i("media: app buttons: ${text.ifEmpty { "none" }}") + } + // --- browsing ------------------------------------------------------------------------ private fun open(parentId: String) { @@ -419,6 +482,12 @@ class CarMedia(private val context: Context) { val controls = controller?.transportControls ?: return if (_now.value.playing) controls.pause() else controls.play() } + /** + * Pauses whatever the app thinks it's doing. Voice pauses the music to listen, so by the time + * "pause" is heard the app already looks stopped; a toggle did nothing and the song came back + * as soon as the microphone let go. A real pause tells the app to stay stopped. + */ + fun pause() = controller?.transportControls?.pause() fun next() = controller?.transportControls?.skipToNext() fun previous() = controller?.transportControls?.skipToPrevious() fun seek(ms: Long) = controller?.transportControls?.seekTo(ms) @@ -454,7 +523,7 @@ class CarMedia(private val context: Context) { } val component = searchComponent if (component == null) { - _results.value = Results(q, emptyList(), supported = false) + noLibrarySearch(q) return } resultsQuery = q @@ -476,13 +545,13 @@ class CarMedia(private val context: Context) { override fun onConnectionFailed() { GearslipLog.w("media: no library connection for search") searchBrowser = null - resultsQuery?.let { _results.value = Results(it, emptyList(), supported = false) } + resultsQuery?.let { noLibrarySearch(it) } } }, null) searchBrowser = created runCatching { created.connect() }.onFailure { searchBrowser = null - _results.value = Results(q, emptyList(), supported = false) + noLibrarySearch(q) } } @@ -492,18 +561,42 @@ class CarMedia(private val context: Context) { override fun onSearchResult(query: String, extras: Bundle?, items: List) { if (query != resultsQuery) return GearslipLog.i("media: library search found ${items.size}") - _results.value = Results(query, items.map(::entryOfCompat)) + val entries = items.map(::entryOfCompat) + if (query == playFirstFor) { + playFirstFor = null + val first = entries.firstOrNull { it.playable } ?: entries.firstOrNull() + if (first == null) { + clearResults() + search(query) + } else { + GearslipLog.i("media: playing the library's best match") + clearResults() + searchBefore = _now.value + _search.value = Search(query) + main.removeCallbacks(searchTimeout) + controller?.transportControls?.playFromMediaId(first.id, null) + main.postDelayed(searchTimeout, SEARCH_WAIT_MS) + } + return + } + _results.value = Results(query, entries) } override fun onError(query: String, extras: Bundle?) { if (query != resultsQuery) return + if (query == playFirstFor) { + playFirstFor = null + clearResults() + search(query) + return + } GearslipLog.i("media: the app has no library search") _results.value = Results(query, emptyList(), supported = false) } }) }.onFailure { GearslipLog.w("media: library search failed: ${it.message}") - _results.value = Results(q, emptyList(), supported = false) + noLibrarySearch(q) } } @@ -520,6 +613,36 @@ class CarMedia(private val context: Context) { ) } + /** The library can't search: a voice "play X" falls back to the app's own search; a typed one says so. */ + private fun noLibrarySearch(q: String) { + if (q == playFirstFor) { + playFirstFor = null + clearResults() + search(q) + } else { + _results.value = Results(q, emptyList(), supported = false) + } + } + + /** A voice "play X" waiting on the library search, to play its first match instead of listing them. */ + private var playFirstFor: String? = null + + /** + * Plays the best match for [query], for voice. Asks the app's library first and plays its top + * result, which apps like ViVi honour where they ignore [search]; without a library search, + * falls back to [search]. + */ + fun playBestMatch(query: String) { + val q = query.trim() + if (q.isEmpty()) return + if (!canSearchLibrary) { + search(q) + return + } + playFirstFor = q + searchLibrary(q) + } + /** The connected app's service, for the search connection; null when nothing is connected. */ private var searchComponent: android.content.ComponentName? = null private var searchBrowser: MediaBrowserCompat? = null @@ -637,6 +760,7 @@ class CarMedia(private val context: Context) { } private companion object { + const val CONNECT_WAIT_MS = 8_000L const val SESSION_RETRIES = 6 const val SESSION_RETRY_MS = 500L /** After opening a cold app, how long to wait before the first look for its session. */ diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/media/CarMediaStatus.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/media/CarMediaStatus.kt new file mode 100644 index 0000000..0c1088a --- /dev/null +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/media/CarMediaStatus.kt @@ -0,0 +1,119 @@ +package app.seb3thehacker.gearslip.media + +import android.graphics.Bitmap +import android.media.session.PlaybackState +import app.seb3thehacker.gearslip.GearslipLog +import app.seb3thehacker.gearslip.Protobuf +import app.seb3thehacker.gearslip.car.CarServices +import java.io.ByteArrayOutputStream + +/** + * Tells the car what's playing, so its own screens show it: the instrument cluster, the media + * source screen, the steering wheel display. Android Auto does the same over the car's + * MediaPlaybackStatusService (Service field 9): + * + * -> MediaPlaybackMetadata { song = 1; artist = 2; album = 3; album_art = 4; duration_seconds = 6 } + * -> MediaPlaybackStatus { state = 1 (1 stopped, 2 playing, 3 paused); media_source = 2; + * playback_seconds = 3; shuffle = 4; repeat = 5; repeat_one = 6 } + * + * Read once a second from the media app Gearslip is driving. The track goes only when it changes; + * the status goes every second while playing, so the car's position counts along. + */ +class CarMediaStatus( + val channelId: Int, + private val sendOnChannel: (messageId: Int, body: ByteArray) -> Unit, +) { + @Volatile private var running = false + private var sentTrack: String? = null + private var sentState = -1 + private var inputsLogged = 0 + + fun onMessage(messageId: Int, body: ByteArray) { + when (messageId) { + MSG_CHANNEL_OPEN_RESPONSE -> { + val status = Protobuf.readInt32Field(body, 1) ?: 0 + GearslipLog.i("media status: <- ChannelOpenResponse: status=$status") + if (status == 0) start() + } + // The car's own buttons for the media source (play, next...). Logged to learn what + // this car sends; the steering wheel keys already arrive on the input channel. + MSG_INPUT -> if (inputsLogged++ < 10) GearslipLog.i("media status: <- input from the car:\n" + Protobuf.describe(body)) + else -> GearslipLog.i("media status: <- unhandled message id=$messageId (${body.size} bytes)") + } + } + + private fun start() { + if (running) return + running = true + Thread({ + while (running) { + runCatching { tick() }.onFailure { GearslipLog.w("media status: ${it.message}") } + Thread.sleep(TICK_MS) + } + }, "gearslip-media-status").start() + } + + fun stop() { + running = false + } + + private fun tick() { + val now = CarServices.mediaOrNull()?.now?.value + val active = now != null && now.isActive + val state = when { + !active -> STATE_STOPPED + now!!.state == PlaybackState.STATE_PAUSED -> STATE_PAUSED + else -> STATE_PLAYING + } + if (active) sendTrack(now!!) + // Nothing to say while stopped, beyond saying so once. + if (state == STATE_STOPPED && sentState == STATE_STOPPED) return + if (state == STATE_PAUSED && sentState == STATE_PAUSED) return + val source = CarServices.mediaApp.value?.label ?: "Gearslip" + val seconds = if (active) (now!!.currentPosition() / 1000).coerceAtLeast(0) else 0 + sendOnChannel( + MSG_STATUS, + Protobuf.varintField(1, state.toLong()) + + Protobuf.stringField(2, source) + + Protobuf.varintField(3, seconds) + + Protobuf.varintField(4, 0L) + + Protobuf.varintField(5, 0L) + + Protobuf.varintField(6, 0L), + ) + if (state != sentState) GearslipLog.i("media status: -> ${STATE_NAMES[state]} ($source)") + sentState = state + } + + private fun sendTrack(now: NowPlaying) { + val key = "${now.title}\u0000${now.artist}\u0000${now.album}\u0000${now.durationMs}\u0000${now.art?.generationId}" + if (key == sentTrack) return + sentTrack = key + var body = Protobuf.stringField(1, now.title) + + Protobuf.stringField(2, now.artist) + + Protobuf.stringField(3, now.album) + now.art?.let { art -> artBytes(art)?.let { body += Protobuf.bytesField(4, it) } } + if (now.durationMs > 0) body += Protobuf.varintField(6, now.durationMs / 1000) + sendOnChannel(MSG_METADATA, body) + GearslipLog.i("media status: -> track \"${now.title}\" by ${now.artist.ifBlank { "?" }}") + } + + /** Album art as a small PNG: car screens show it at thumbnail size, and the link is shared with video. */ + private fun artBytes(art: Bitmap): ByteArray? = runCatching { + val scale = minOf(1f, ART_PX.toFloat() / maxOf(art.width, art.height)) + val small = if (scale < 1f) Bitmap.createScaledBitmap(art, (art.width * scale).toInt(), (art.height * scale).toInt(), true) else art + ByteArrayOutputStream().also { small.compress(Bitmap.CompressFormat.PNG, 100, it) }.toByteArray() + }.getOrNull() + + companion object { + private const val TICK_MS = 1_000L + private const val ART_PX = 256 + private const val MSG_CHANNEL_OPEN_RESPONSE = 8 + private const val MSG_STATUS = 32769 + private const val MSG_INPUT = 32770 + private const val MSG_METADATA = 32771 + private const val STATE_STOPPED = 1 + private const val STATE_PLAYING = 2 + private const val STATE_PAUSED = 3 + private val STATE_NAMES = mapOf(1 to "stopped", 2 to "playing", 3 to "paused") + } +} diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/notify/GearslipNotificationListener.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/notify/GearslipNotificationListener.kt index 3906878..b6d588a 100644 --- a/gearslip/src/main/java/app/seb3thehacker/gearslip/notify/GearslipNotificationListener.kt +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/notify/GearslipNotificationListener.kt @@ -30,7 +30,7 @@ class GearslipNotificationListener : NotificationListenerService() { override fun onNotificationRemoved(sbn: StatusBarNotification) = CarNotifications.gone(sbn.key) private fun ingest(sbn: StatusBarNotification, quiet: Boolean) { - // Gearslip's own notifications are skipped, except a dev build's test message. + // Gearslip's own notifications are skipped, except a debug build's test message. if (sbn.packageName == packageName && !(app.seb3thehacker.gearslip.BuildConfig.DEBUG && sbn.notification.channelId == TestMessage.CHANNEL) ) return diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/notify/TestMessage.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/notify/TestMessage.kt index 1376e11..802b264 100644 --- a/gearslip/src/main/java/app/seb3thehacker/gearslip/notify/TestMessage.kt +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/notify/TestMessage.kt @@ -13,7 +13,7 @@ import androidx.core.app.RemoteInput import app.seb3thehacker.gearslip.GearslipLog /** - * Dev builds only: posts a chat message from Gearslip to itself, shaped like Signal's or + * Debug builds only: posts a chat message from Gearslip to itself, shaped like Signal's or * WhatsApp's (MessagingStyle, a Reply action, a Mark as read action), so popups and voice reply * can be tried without asking someone to text you. The listener lets these through only in a * debug build; everything else Gearslip posts is still ignored. diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/speech/SpeechEngine.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/speech/SpeechEngine.kt index 2fc90fa..8684259 100644 --- a/gearslip/src/main/java/app/seb3thehacker/gearslip/speech/SpeechEngine.kt +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/speech/SpeechEngine.kt @@ -4,7 +4,6 @@ import android.Manifest import android.annotation.SuppressLint import android.content.Context import android.content.pm.PackageManager -import android.content.res.AssetManager import android.media.AudioFormat import android.media.AudioManager import android.media.AudioRecord @@ -15,38 +14,50 @@ import android.os.Looper import android.os.SystemClock import app.seb3thehacker.gearslip.GearslipLog import app.seb3thehacker.gearslip.audio.CarMic -import java.util.Locale +import com.k2fsa.sherpa.onnx.EndpointConfig +import com.k2fsa.sherpa.onnx.EndpointRule +import com.k2fsa.sherpa.onnx.FeatureConfig +import com.k2fsa.sherpa.onnx.OfflineCanaryModelConfig +import com.k2fsa.sherpa.onnx.OfflineModelConfig +import com.k2fsa.sherpa.onnx.OfflineRecognizer +import com.k2fsa.sherpa.onnx.OfflineRecognizerConfig +import com.k2fsa.sherpa.onnx.OfflineTransducerModelConfig +import com.k2fsa.sherpa.onnx.OnlineModelConfig +import com.k2fsa.sherpa.onnx.OnlineRecognizer +import com.k2fsa.sherpa.onnx.OnlineRecognizerConfig +import com.k2fsa.sherpa.onnx.OnlineTransducerModelConfig +import java.io.File import java.util.concurrent.Executors -import kotlin.math.log10 import kotlin.math.max -import kotlin.math.sqrt /** * Gearslip's own speech-to-text, so voice works on phones with no speech service (GrapheneOS ships - * none). whisper.cpp runs FUTO's ACFT Whisper model, which are tuned for short phrases: a reply - * of a few seconds comes back in well under a second on a recent phone. + * none). sherpa-onnx runs a streaming model ([SpeechModels]) on the phone: words come out while + * the driver is still talking, and the model itself decides when they've finished. A larger model + * then reads the whole phrase once more for the final words, since a model that waits for the end + * hears names and places far better than one that has to keep up. The phrase is held in memory + * for that and nowhere else, and dropped as soon as it's read. * - * Whisper needs the whole phrase before it starts, so this records from the car's microphone (the - * phone's when the car has none or sends nothing), - * decides when the driver has stopped talking (a stretch of quiet after speech), then transcribes - * once. Everything runs on one worker thread; callbacks arrive on the main thread. + * Audio comes from the car's microphone, or the phone's when the car has none or sends nothing. + * Everything runs on one worker thread; callbacks arrive on the main thread. */ object SpeechEngine { - /** FUTO's ACFT Whisper tiny (Apache-2.0), re-quantized to 5 bits, inside the APK. */ - private const val MODEL = "speech/tiny_acft_q5_1.bin" - /** The error for an engine that can't run at all, which callers may answer with a fallback. */ const val NOT_LOADED = "The speech model didn't load." + /** Said when the chosen model hasn't been downloaded yet. */ + const val NOT_DOWNLOADED = "Download a voice model in Gearslip's settings on the phone first." + private const val RATE = 16_000 - private const val FRAME = 480 // 30 ms - private const val MAX_MS = 15_000L - private const val NO_SPEECH_MS = 6_000L - private const val END_SILENCE_MS = 1_100L - private const val CALIBRATE_MS = 300L - private const val PRE_ROLL = RATE * 3 / 10 // keep 300 ms before the first word + private const val CHUNK = 1_600 // 100 ms + private const val MAX_S = 15f + /** Quiet before the first word that counts as "nobody spoke". */ + private const val NO_SPEECH_S = 6f + /** Quiet after the last word that ends the phrase. */ + private const val END_SILENCE_S = 1.0f private const val CUE_VOLUME = 80 + private const val IDLE_UNLOAD_MS = 5 * 60_000L /** How long the car gets to start sending once asked, before the phone's microphone takes over. */ private const val CAR_FIRST_AUDIO_MS = 1_500L private const val CAR_GAP_MS = 1_000L @@ -55,27 +66,79 @@ object SpeechEngine { private const val CUE_TAIL_MS = 250L private val worker = Executors.newSingleThreadExecutor { Thread(it, "gearslip-speech") } + /** + * The final model takes seconds to load. On [worker], the first press after connecting + * waited behind it before listening at all; here it loads while the driver talks. + */ + private val finalWorker = Executors.newSingleThreadExecutor { Thread(it, "gearslip-speech-final") } + private val finalLock = Object() private val main = Handler(Looper.getMainLooper()) - private var libraryLoaded = false - private var handle = 0L + private var recognizer: OnlineRecognizer? = null + private var loadedModel: SpeechModels.Model? = null + @Volatile private var finalRecognizer: OfflineRecognizer? = null + @Volatile private var loadedFinal: SpeechModels.Model? = null + + /** + * The final model holds most of the memory the two models take, so it goes after a few + * minutes without a command. Kroko stays: it's small, and keeps listening instant. + */ + private val idleUnload = Runnable { unloadFinal("after ${IDLE_UNLOAD_MS / 60_000} minutes without use") } @Volatile private var run = 0 - /** Loads the model ahead of time, so the first reply doesn't wait for it. */ + /** Loads the model ahead of time, so the first command doesn't wait for it. */ fun warm(context: Context) { val app = context.applicationContext - worker.execute { ensureLoaded(app) } + SpeechModels.init(app) + // Kroko first, alone: loading both at once slowed it from a fraction of a second to six. + worker.execute { + ensureLoaded(app) + finalWorker.execute { ensureFinalLoaded(app) } + } + } + + private fun unloadFinal(why: String) { + finalWorker.execute { + synchronized(finalLock) { + val model = loadedFinal ?: return@execute + finalRecognizer?.release() + finalRecognizer = null + loadedFinal = null + GearslipLog.i("speech: unloaded ${model.label} $why") + } + } + } + + /** Drops the loaded models: the driver picked another or deleted one, or the car went away. */ + fun unload() { + main.removeCallbacks(idleUnload) + run++ + worker.execute { + recognizer?.release() + recognizer = null + loadedModel = null + } + finalWorker.execute { + synchronized(finalLock) { + finalRecognizer?.release() + finalRecognizer = null + loadedFinal = null + } + } } /** * Listens for one phrase. [cue] plays a short tone first, so the driver knows when to talk. - * [onSpeaking] fires at the first word, [onHeard] when the driver stops and the words are being - * worked out; [onResult] gets the text, [onError] a sentence fit to show on the car screen. + * [onPartial] gets the words so far as they're recognized, [onSpeaking] fires at the first + * word, [onHeard] when the driver stops; [onResult] gets the text, [onError] a sentence fit to + * show on the car screen. */ fun listen( context: Context, cue: Boolean = true, onSpeaking: () -> Unit = {}, + onPartial: (String) -> Unit = {}, + onListening: () -> Unit = {}, onHeard: () -> Unit, onResult: (String) -> Unit, onError: (String) -> Unit, @@ -87,28 +150,29 @@ object SpeechEngine { } val id = ++run fun post(block: () -> Unit) { main.post { if (id == run) block() } } + // Brings Parakeet back if it was unloaded, while the driver talks. + main.removeCallbacks(idleUnload) + finalWorker.execute { ensureFinalLoaded(app) } worker.execute { if (id != run) return@execute - if (!ensureLoaded(app)) return@execute post { onError(NOT_LOADED) } + if (!SpeechModels.isReady(app, SpeechModels.selected.value)) return@execute post { onError(NOT_DOWNLOADED) } + val r = ensureLoaded(app) ?: return@execute post { onError(NOT_LOADED) } VoiceFocus.acquire(app) - val audio = try { + val text = try { if (cue) cue() if (id != run) return@execute - record(id) { post(onSpeaking) } + post(onListening) + stream(app, r, id, onSpeaking = { post(onSpeaking) }, onPartial = { post { onPartial(it) } }, onStopped = { post(onHeard) }) } catch (t: Throwable) { - GearslipLog.e("speech: recording failed", t) + GearslipLog.e("speech: listening failed", t) return@execute post { onError("The microphone isn't available right now.") } } finally { VoiceFocus.release() } + main.removeCallbacks(idleUnload) + main.postDelayed(idleUnload, IDLE_UNLOAD_MS) if (id != run) return@execute - if (audio == null) return@execute post { onError("Didn't catch that.") } post(onHeard) - val started = SystemClock.uptimeMillis() - val text = clean(transcribe(audio)) - GearslipLog.i( - "speech: ${audio.size * 1000L / RATE} ms of audio took ${SystemClock.uptimeMillis() - started} ms", - ) post { if (text.isEmpty()) onError("Didn't catch that.") else onResult(text) } } } @@ -116,6 +180,185 @@ object SpeechEngine { /** Stops listening; a phrase already being worked out is dropped. */ fun cancel() { run++ } + /** + * Feeds the microphone to the recognizer in 100 ms pieces, reporting the words as they firm + * up, until the model hears the end of the phrase (or nobody speaks, or it runs too long). + */ + private fun stream( + context: Context, + r: OnlineRecognizer, + id: Int, + onSpeaking: () -> Unit, + onPartial: (String) -> Unit, + onStopped: () -> Unit, + ): String { + val mic = carMic() ?: phoneMic() + val s = r.createStream() + val frame = ShortArray(CHUNK) + val samples = FloatArray(CHUNK) + // The whole phrase, for the final read; [MAX_S] and a little over is the most it can hold. + val phrase = FloatArray((RATE * (MAX_S + 1)).toInt()) + var held = 0 + var last = "" + var spoke = false + var audioMs = 0L + var decodeMs = 0L + try { + while (id == run) { + val n = mic.read(frame) + if (n < 0) break + if (n == 0) continue + for (i in 0 until n) samples[i] = frame[i] / 32768f + val keep = minOf(n, phrase.size - held) + System.arraycopy(samples, 0, phrase, held, keep) + held += keep + s.acceptWaveform(if (n == CHUNK) samples else samples.copyOf(n), RATE) + audioMs += n * 1000L / RATE + val started = SystemClock.uptimeMillis() + while (r.isReady(s)) r.decode(s) + decodeMs += SystemClock.uptimeMillis() - started + val text = clean(r.getResult(s).text) + if (text != last) { + last = text + if (text.isNotEmpty() && !spoke) { + spoke = true + onSpeaking() + } + onPartial(text) + } + if (r.isEndpoint(s)) break + } + s.inputFinished() + while (r.isReady(s)) r.decode(s) + last = clean(r.getResult(s).text) + } finally { + mic.close() + s.release() + } + // Under 1.0 keeps up with the driver; over it, the words fall behind the voice. + GearslipLog.i( + "speech: $audioMs ms of audio, decoding took $decodeMs ms " + + "(${"%.2f".format(decodeMs.toFloat() / max(1L, audioMs))} of real time)", + ) + if (last.isEmpty() || id != run) { phrase.fill(0f); return last } + // Still loading (just connected, or back after going idle): the live words now beat + // Parakeet's in several seconds. + val wanted = SpeechModels.final.value + if (wanted == null || finalRecognizer == null || loadedFinal != wanted) { + if (wanted != null) GearslipLog.i("speech: ${wanted.label} isn't loaded yet, keeping the live words") + phrase.fill(0f) + return last + } + onStopped() + val started = SystemClock.uptimeMillis() + // Held for the whole read, so a model swap or the idle unload can't free it mid-decode. + val reread = synchronized(finalLock) { + val final = finalRecognizer?.takeIf { loadedFinal == wanted } ?: run { phrase.fill(0f); return last } + final.createStream().run { + try { + acceptWaveform(phrase.copyOf(held), RATE) + final.decode(this) + clean(final.getResult(this).text) + } finally { + release() + } + } + } + phrase.fill(0f) + GearslipLog.i("speech: the final read took ${SystemClock.uptimeMillis() - started} ms") + if (reread.isEmpty()) return last + if (reread != last) onPartial(reread) + return reread + } + + /** The model that rereads the phrase, when one is chosen and downloaded; null otherwise. */ + private fun ensureFinalLoaded(context: Context): OfflineRecognizer? = synchronized(finalLock) { + val model = SpeechModels.final.value ?: return null + if (finalRecognizer != null && loadedFinal == model) return finalRecognizer + finalRecognizer?.release() + finalRecognizer = null + if (!SpeechModels.isReady(context, model)) return null + val dir = SpeechModels.dir(context, model) + fun path(name: String) = File(dir, name).absolutePath + val started = SystemClock.uptimeMillis() + return runCatching { + OfflineRecognizer( + config = OfflineRecognizerConfig( + featConfig = FeatureConfig(sampleRate = RATE, featureDim = 80), + modelConfig = if (model.modelType == "canary") { + // English in, English out, and no punctuation: clean() strips it anyway. + OfflineModelConfig( + canary = OfflineCanaryModelConfig(path(model.encoder), path(model.decoder), srcLang = "en", tgtLang = "en", usePnc = false), + tokens = path("tokens.txt"), + numThreads = threads(), + ) + } else { + OfflineModelConfig( + transducer = OfflineTransducerModelConfig(path(model.encoder), path(model.decoder), path(model.joiner)), + tokens = path("tokens.txt"), + numThreads = threads(), + modelType = model.modelType, + ) + }, + decodingMethod = "greedy_search", + ), + ) + }.onSuccess { + finalRecognizer = it + loadedFinal = model + GearslipLog.i("speech: loaded ${model.label} in ${SystemClock.uptimeMillis() - started} ms") + }.onFailure { + GearslipLog.e("speech: couldn't load ${model.label}", it) + }.getOrNull() + } + + private fun ensureLoaded(context: Context): OnlineRecognizer? { + val model = SpeechModels.selected.value + if (recognizer != null && loadedModel == model) return recognizer + recognizer?.release() + recognizer = null + if (!SpeechModels.isReady(context, model)) return null + val dir = SpeechModels.dir(context, model) + fun path(name: String) = File(dir, name).absolutePath + val started = SystemClock.uptimeMillis() + return runCatching { + OnlineRecognizer( + config = OnlineRecognizerConfig( + featConfig = FeatureConfig(sampleRate = RATE, featureDim = 80), + modelConfig = OnlineModelConfig( + transducer = OnlineTransducerModelConfig(path(model.encoder), path(model.decoder), path(model.joiner)), + tokens = path("tokens.txt"), + numThreads = threads(), + modelType = model.modelType, + ), + endpointConfig = EndpointConfig( + rule1 = EndpointRule(false, NO_SPEECH_S, 0f), + rule2 = EndpointRule(true, END_SILENCE_S, 0f), + rule3 = EndpointRule(false, 0f, MAX_S), + ), + enableEndpoint = true, + decodingMethod = "greedy_search", + ), + ) + }.onSuccess { + recognizer = it + loadedModel = model + GearslipLog.i("speech: loaded ${model.label} in ${SystemClock.uptimeMillis() - started} ms") + }.onFailure { + GearslipLog.e("speech: couldn't load ${model.label}", it) + }.getOrNull() + } + + /** The big cores only: more threads than that slow the model down on a phone. */ + private fun threads() = Runtime.getRuntime().availableProcessors().let { if (it >= 8) 4 else max(1, it / 2) } + + /** + * Models write in upper case, or with punctuation and capitals of their own; commands are + * matched in plain lower case, so only words and apostrophes are kept. + */ + private fun clean(text: String): String = + text.lowercase().replace(Regex("[^\\p{L}\\p{N}' ]"), " ").replace(Regex("\\s+"), " ").trim() + /** * A short beep on the media stream, so it plays through the car like everything else, then a * pause long enough for it to finish there before the microphone opens and hears it. @@ -130,14 +373,9 @@ object SpeechEngine { } } - // --- recording -------------------------------------------------------------------------- + // --- microphones ----------------------------------------------------------------------- - /** - * Records until the driver stops talking. Returns the phrase from just before the first word, - * or null when nobody spoke. The speech threshold follows the noise in the first 300 ms, so a - * loud cabin needs a louder voice than a quiet one to count as talking. - */ - /** Where a phrase is recorded from: [read] fills [frame] and returns how many samples it got. */ + /** Where a phrase is heard from: [read] fills [frame] and returns how many samples it got, -1 at the end. */ private interface Mic { fun read(frame: ShortArray): Int fun close() @@ -148,10 +386,11 @@ object SpeechEngine { val min = AudioRecord.getMinBufferSize(RATE, AudioFormat.CHANNEL_IN_MONO, AudioFormat.ENCODING_PCM_16BIT) val recorder = AudioRecord( MediaRecorder.AudioSource.VOICE_RECOGNITION, RATE, - AudioFormat.CHANNEL_IN_MONO, AudioFormat.ENCODING_PCM_16BIT, max(min, FRAME * 8), + AudioFormat.CHANNEL_IN_MONO, AudioFormat.ENCODING_PCM_16BIT, max(min, CHUNK * 4), ) check(recorder.state == AudioRecord.STATE_INITIALIZED) { "AudioRecord didn't initialize" } recorder.startRecording() + GearslipLog.i("speech: listening through the phone's microphone") return object : Mic { override fun read(frame: ShortArray) = recorder.read(frame, 0, frame.size) override fun close() { @@ -163,7 +402,7 @@ object SpeechEngine { /** * The car's microphone, which hears the driver better than a phone in a cup holder. The car - * sends audio in its own sized pieces; they're cut to [FRAME]s here. Null when there's no car + * sends audio in its own sized pieces; they're cut to [CHUNK]s here. Null when there's no car * microphone, or it sends nothing at first, so the phone's is used instead. */ private fun carMic(): Mic? { @@ -190,105 +429,10 @@ object SpeechEngine { at += n if (at == chunk.size) { pending = null; at = 0 } } - // The car stopped sending mid-phrase: end the recording rather than wait forever. + // The car stopped sending mid-phrase: end the phrase rather than wait forever. return if (filled == 0) -1 else filled } override fun close() = car.stop() } } - - /** - * Records until the driver stops talking. Returns the phrase from just before the first word, - * or null when nobody spoke. The speech threshold follows the noise in the first 300 ms, so a - * loud cabin needs a louder voice than a quiet one to count as talking. - */ - private fun record(id: Int, onSpeaking: () -> Unit): FloatArray? { - val mic = carMic() ?: phoneMic() - val samples = FloatArray((RATE * MAX_MS / 1000).toInt()) - var count = 0 - val frame = ShortArray(FRAME) - var noise = 0.0 - var noiseFrames = 0 - var speechStart = -1 - var lastSpeech = 0 - try { - while (id == run && count + FRAME <= samples.size) { - val n = mic.read(frame) - if (n < 0) break - if (n == 0) continue - var sum = 0.0 - for (i in 0 until n) { - val v = frame[i] / 32768f - samples[count + i] = v - sum += v * v - } - count += n - val db = 20 * log10(sqrt(sum / n) + 1e-9) - val ms = count * 1000L / RATE - if (ms <= CALIBRATE_MS) { - noise += db; noiseFrames++ - continue - } - val floor = if (noiseFrames > 0) noise / noiseFrames else -60.0 - val talking = db > max(floor + 12, -45.0) - if (talking) { - if (speechStart < 0) { - speechStart = max(0, count - n - PRE_ROLL) - onSpeaking() - } - lastSpeech = count - } - if (speechStart < 0 && ms > NO_SPEECH_MS) return null - if (speechStart >= 0 && (count - lastSpeech) * 1000L / RATE > END_SILENCE_MS) break - } - } finally { - mic.close() - } - if (speechStart < 0) return null - return samples.copyOfRange(speechStart, count) - } - - // --- the model -------------------------------------------------------------------------- - - private fun ensureLoaded(context: Context): Boolean { - if (handle != 0L) return true - if (!libraryLoaded) { - runCatching { System.loadLibrary("gearslip_speech") } - .onFailure { GearslipLog.e("speech: native library missing", it); return false } - libraryLoaded = true - } - val started = SystemClock.uptimeMillis() - handle = nativeInitAsset(context.assets, MODEL) - if (handle == 0L) { - GearslipLog.w("speech: couldn't load the model") - return false - } - GearslipLog.i("speech: loaded the model in ${SystemClock.uptimeMillis() - started} ms") - return true - } - - private fun transcribe(audio: FloatArray): String { - val bytes = nativeTranscribe(handle, audio, language(), threads()) - return String(bytes, Charsets.UTF_8) - } - - /** The phone's language, as Whisper names it. Older Java codes differ for a few. */ - private fun language(): String = when (val lang = Locale.getDefault().language) { - "iw" -> "he" - "in" -> "id" - "ji" -> "yi" - "" -> "en" - else -> lang - } - - /** The big cores only: more threads than that slow Whisper down on a phone. */ - private fun threads() = Runtime.getRuntime().availableProcessors().coerceIn(1, 8).let { if (it >= 8) 4 else max(1, it / 2) } - - /** Whisper marks silence and noise as "[BLANK_AUDIO]" or "(wind blowing)"; none of it is words. */ - private fun clean(text: String): String = - text.replace(Regex("""\[[^\]]*\]|\([^)]*\)|\*[^*]*\*"""), " ").replace(Regex("\\s+"), " ").trim() - - private external fun nativeInitAsset(assets: AssetManager, path: String): Long - private external fun nativeTranscribe(handle: Long, samples: FloatArray, language: String, threads: Int): ByteArray - private external fun nativeFree(handle: Long) } diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/speech/SpeechModels.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/speech/SpeechModels.kt new file mode 100644 index 0000000..87a0693 --- /dev/null +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/speech/SpeechModels.kt @@ -0,0 +1,303 @@ +package app.seb3thehacker.gearslip.speech + +import android.Manifest +import android.app.NotificationChannel +import android.app.NotificationManager +import android.app.PendingIntent +import android.content.Context +import android.content.Intent +import android.content.pm.PackageManager +import androidx.core.app.NotificationCompat +import androidx.core.app.NotificationManagerCompat +import app.seb3thehacker.gearslip.GearslipActivity +import app.seb3thehacker.gearslip.GearslipLog +import java.io.File +import java.net.HttpURLConnection +import java.net.URL +import java.util.concurrent.Executors +import kotlinx.coroutines.flow.MutableStateFlow +import kotlinx.coroutines.flow.StateFlow +import kotlinx.coroutines.flow.asStateFlow +import org.apache.commons.compress.archivers.tar.TarArchiveInputStream +import org.apache.commons.compress.compressors.bzip2.BZip2CompressorInputStream + +/** + * The speech models Gearslip listens with, in two roles: a streaming one that writes the words + * while the driver talks, and an optional final one that reads the whole phrase again once they + * stop, for a more accurate result. None ships in the APK: each is fetched once and kept in the + * app's own storage. sherpa-onnx's author publishes them twice: as .tar.bz2 archives on GitHub and + * as plain files on Hugging Face. The plain files come first where they exist. Unpacking bzip2 on + * the phone pinned a core and held a download to 150 KB/s on Wi-Fi that does 19 MB/s. + */ +object SpeechModels { + + enum class Role { LIVE, FINAL } + + enum class Model( + val role: Role, + val id: String, + val label: String, + val detail: String, + val downloadMb: Int, + private val archive: String, + val encoder: String, + val decoder: String, + /** Empty for Canary, which decodes with attention and has no joiner. */ + val joiner: String, + val modelType: String = "", + /** Whether Hugging Face has the files unpacked under the archive's name. */ + val plainFiles: Boolean = true, + ) { + KROKO( + Role.LIVE, "kroko-en", "Kroko", "Banafo's Kroko ASR, CC BY-SA 4.0", 68, + "sherpa-onnx-streaming-zipformer-en-kroko-2025-08-06", + "encoder.onnx", "decoder.onnx", "joiner.onnx", modelType = "zipformer2", + ), + PARAKEET_SMALL( + Role.FINAL, "parakeet-110m-en", "Parakeet small", "Lighter and quicker, misses more names. NVIDIA NeMo, CC BY 4.0", 103, + "sherpa-onnx-nemo-parakeet_tdt_transducer_110m-en-36000-int8", + "encoder.int8.onnx", "decoder.int8.onnx", "joiner.int8.onnx", modelType = "nemo_transducer", + plainFiles = false, + ), + CANARY( + Role.FINAL, "canary-180m-en", "Canary", "Recommended. Quick, and hears names and places well. Knows English, Spanish, German and French; Gearslip listens in English for now. NVIDIA NeMo, CC BY 4.0", 198, + "sherpa-onnx-nemo-canary-180m-flash-en-es-de-fr-int8", + "encoder.int8.onnx", "decoder.int8.onnx", "", modelType = "canary", + ), + PARAKEET_LARGE( + Role.FINAL, "parakeet-0.6b-en", "Parakeet large", "Hears names best, but slower and three times the size. NVIDIA NeMo, CC BY 4.0", 631, + "sherpa-onnx-nemo-parakeet-tdt-0.6b-v2-int8", + "encoder.int8.onnx", "decoder.int8.onnx", "joiner.int8.onnx", modelType = "nemo_transducer", + ), + ; + + val url get() = "https://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models/$archive.tar.bz2" + fun fileUrl(name: String) = "https://huggingface.co/csukuangfj/$archive/resolve/main/$name" + val files get() = listOf(encoder, decoder, joiner, "tokens.txt").filter { it.isNotEmpty() } + } + + /** What setup offers in one tap: Kroko for the live words, Canary for the final ones. */ + val RECOMMENDED = listOf(Model.KROKO, Model.CANARY) + + fun downloadRecommended(context: Context) = RECOMMENDED.forEach { download(context, it) } + + sealed interface State { + data object Missing : State + data class Downloading(val percent: Int) : State + data object Ready : State + data class Failed(val reason: String) : State + } + + private const val PREFS = "speech" + private const val KEY_MODEL = "model" + private const val KEY_FINAL = "final_model" + private const val CHANNEL = "voice_download" + private const val NOTIFICATION_ID = 4_312 + + private val worker = Executors.newSingleThreadExecutor { Thread(it, "gearslip-speech-download") } + private val states = Model.entries.associateWith { MutableStateFlow(State.Missing) } + private val _selected = MutableStateFlow(Model.KROKO) + val selected: StateFlow = _selected.asStateFlow() + private val _final = MutableStateFlow(Model.CANARY) + + /** The model that rereads the whole phrase at the end; null to keep the live model's words. */ + val final: StateFlow = _final.asStateFlow() + @Volatile private var cancelled: Model? = null + + fun init(context: Context) { + val id = context.getSharedPreferences(PREFS, Context.MODE_PRIVATE).getString(KEY_MODEL, null) + _selected.value = Model.entries.firstOrNull { it.id == id && it.role == Role.LIVE } ?: Model.KROKO + // Canary is the default. A driver who never chose but already has Parakeet large keeps it. + val finalId = context.getSharedPreferences(PREFS, Context.MODE_PRIVATE).getString(KEY_FINAL, null) + ?: if (isReady(context, Model.PARAKEET_LARGE) && !isReady(context, Model.CANARY)) Model.PARAKEET_LARGE.id else Model.CANARY.id + _final.value = Model.entries.firstOrNull { it.id == finalId && it.role == Role.FINAL } + Model.entries.forEach { m -> if (isReady(context, m)) states.getValue(m).value = State.Ready } + // Models an older build offered and this one doesn't: hundreds of MB nobody can use. A + // known model's ".partial" folder is a download that may be running right now; [download] + // clears its own leftovers when it starts. + val known = Model.entries.map { it.id }.toSet() + File(context.filesDir, "speech").listFiles() + ?.filter { it.isDirectory && it.name.removeSuffix(".partial") !in known }?.forEach { + GearslipLog.i("speech: removing ${it.name}, no longer offered") + it.deleteRecursively() + } + } + + fun state(model: Model): StateFlow = states.getValue(model).asStateFlow() + + /** + * The models fetched since the last batch finished, so the percent covers all of them and + * doesn't jump back to 0 when the second one starts. + */ + private val batch = mutableSetOf() + private val _progress = MutableStateFlow(null) + + /** Percent through the current downloads, for Home and the notification; null when none run. */ + val progress: StateFlow = _progress.asStateFlow() + + private fun updateProgress(context: Context) { + val percent = synchronized(batch) { + if (batch.none { states.getValue(it).value is State.Downloading }) { + batch.clear() + null + } else { + val done = batch.sumOf { m -> + m.downloadMb * when (val st = states.getValue(m).value) { + is State.Downloading -> st.percent + State.Ready -> 100 + else -> 0 + } + } + done / batch.sumOf { it.downloadMb } + } + } + if (percent == _progress.value) return + _progress.value = percent + showProgress(context, percent) + } + + /** A silent notification with a progress bar, so the download is visible with the app closed. */ + private fun showProgress(context: Context, percent: Int?) { + val manager = NotificationManagerCompat.from(context) + if (percent == null) { + manager.cancel(NOTIFICATION_ID) + return + } + if (context.checkSelfPermission(Manifest.permission.POST_NOTIFICATIONS) != PackageManager.PERMISSION_GRANTED) return + context.getSystemService(NotificationManager::class.java).createNotificationChannel( + NotificationChannel(CHANNEL, "Voice download", NotificationManager.IMPORTANCE_LOW), + ) + val open = PendingIntent.getActivity( + context, NOTIFICATION_ID, + Intent(context, GearslipActivity::class.java).addFlags(Intent.FLAG_ACTIVITY_NEW_TASK or Intent.FLAG_ACTIVITY_CLEAR_TOP), + PendingIntent.FLAG_IMMUTABLE, + ) + val notification = NotificationCompat.Builder(context, CHANNEL) + .setSmallIcon(android.R.drawable.stat_sys_download) + .setContentTitle("Downloading voice") + .setContentText("$percent%") + .setProgress(100, percent, false) + .setOngoing(true) + .setOnlyAlertOnce(true) + .setSilent(true) + .setContentIntent(open) + .build() + runCatching { manager.notify(NOTIFICATION_ID, notification) } + } + + fun dir(context: Context, model: Model) = File(File(context.filesDir, "speech"), model.id) + + fun isReady(context: Context, model: Model) = model.files.all { File(dir(context, model), it).isFile } + + fun select(context: Context, model: Model) { + if (model.role == Role.FINAL) return selectFinal(context, model) + _selected.value = model + context.getSharedPreferences(PREFS, Context.MODE_PRIVATE).edit().putString(KEY_MODEL, model.id).apply() + SpeechEngine.unload() + } + + /** [model] null keeps the live words as they are. */ + fun selectFinal(context: Context, model: Model?) { + _final.value = model + context.getSharedPreferences(PREFS, Context.MODE_PRIVATE).edit().putString(KEY_FINAL, model?.id ?: "none").apply() + SpeechEngine.unload() + } + + fun delete(context: Context, model: Model) { + cancelled = model + dir(context, model).deleteRecursively() + states.getValue(model).value = State.Missing + updateProgress(context.applicationContext) + if (model == _selected.value || model == _final.value) SpeechEngine.unload() + } + + /** Fetches just the files the recognizer reads, plain where it can, else out of the archive. */ + fun download(context: Context, model: Model) { + val app = context.applicationContext + val state = states.getValue(model) + if (state.value is State.Downloading || state.value == State.Ready) return + state.value = State.Downloading(0) + synchronized(batch) { batch.add(model) } + updateProgress(app) + cancelled = null + worker.execute { + val target = dir(app, model) + val partial = File(target.parentFile, "${model.id}.partial").apply { deleteRecursively(); mkdirs() } + try { + val total = model.downloadMb * 1_048_576L + var done = 0L + fun progress(read: Long) { + if (cancelled == model) throw InterruptedException("cancelled") + state.value = State.Downloading((((done + read) * 100) / total).toInt().coerceIn(0, 99)) + updateProgress(app) + } + if (model.plainFiles) { + for (name in model.files) { + val conn = open(model.fileUrl(name)) + val length = conn.contentLengthLong + val file = File(partial, name) + CountingStream(conn.inputStream, ::progress).use { input -> + file.outputStream().use { input.copyTo(it, 256 * 1024) } + } + check(length <= 0 || file.length() == length) { "$name arrived cut short" } + done += file.length() + } + } else { + val conn = open(model.url) + val counting = CountingStream(conn.inputStream, ::progress) + TarArchiveInputStream(BZip2CompressorInputStream(counting.buffered())).use { tar -> + while (true) { + val entry = tar.nextEntry ?: break + val name = entry.name.substringAfterLast('/') + if (entry.isDirectory || name !in model.files) continue + File(partial, name).outputStream().use { tar.copyTo(it) } + } + } + } + check(model.files.all { File(partial, it).isFile }) { "the archive was missing files" } + target.deleteRecursively() + check(partial.renameTo(target)) { "couldn't store the model" } + GearslipLog.i("speech: downloaded ${model.label}") + state.value = State.Ready + updateProgress(app) + } catch (e: Exception) { + partial.deleteRecursively() + if (cancelled == model) { + state.value = State.Missing + } else { + GearslipLog.w("speech: download of ${model.label} failed: ${e.message}") + state.value = State.Failed(e.message ?: e.javaClass.simpleName) + } + updateProgress(app) + } + } + } + + private fun open(url: String): HttpURLConnection = + (URL(url).openConnection() as HttpURLConnection).apply { + connectTimeout = 15_000 + readTimeout = 30_000 + instanceFollowRedirects = true + }.also { check(it.responseCode == 200) { "the server answered ${it.responseCode}" } } + + /** Counts bytes as they pass, reporting about every megabyte. */ + private class CountingStream( + private val inner: java.io.InputStream, + private val onProgress: (Long) -> Unit, + ) : java.io.InputStream() { + private var read = 0L + private var reported = 0L + + override fun read(): Int = inner.read().also { if (it >= 0) count(1) } + override fun read(b: ByteArray, off: Int, len: Int): Int = inner.read(b, off, len).also { if (it > 0) count(it.toLong()) } + override fun close() = inner.close() + + private fun count(n: Long) { + read += n + if (read - reported >= 1_048_576) { + reported = read + onProgress(read) + } + } + } +} diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/speech/VoiceFocus.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/speech/VoiceFocus.kt index e79f2e5..1b16b8f 100644 --- a/gearslip/src/main/java/app/seb3thehacker/gearslip/speech/VoiceFocus.kt +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/speech/VoiceFocus.kt @@ -19,17 +19,35 @@ import app.seb3thehacker.gearslip.GearslipLog object VoiceFocus { private const val RELEASE_DELAY_MS = 1_500L + private const val AFTER_RELEASE_MS = 300L private val main = Handler(Looper.getMainLooper()) private var request: AudioFocusRequest? = null private var audio: AudioManager? = null private var holders = 0 + private val afterRelease = mutableListOf<() -> Unit>() private val releaseNow = Runnable { - val r = request ?: return@Runnable - audio?.abandonAudioFocusRequest(r) - request = null - GearslipLog.i("voice: gave audio focus back") + request?.let { r -> + audio?.abandonAudioFocusRequest(r) + request = null + GearslipLog.i("voice: gave audio focus back") + } + val pending = afterRelease.toList() + afterRelease.clear() + // Just after the music app has had its focus back and acted on it. + if (pending.isNotEmpty()) main.postDelayed({ pending.forEach { it() } }, AFTER_RELEASE_MS) + } + + /** + * Runs [action] once the music has its focus back, or now if Gearslip isn't holding it. + * Metrolist takes a pause sent while it's held as nothing and resumes when focus returns, + * so "pause" sends it again from here. + */ + fun afterRelease(action: () -> Unit) { + main.post { + if (request == null && holders == 0) action() else afterRelease += action + } } /** Pauses the music until a matching [release]. Safe from any thread. */ diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/stats/UsageStats.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/stats/UsageStats.kt index d03b176..55c6036 100644 --- a/gearslip/src/main/java/app/seb3thehacker/gearslip/stats/UsageStats.kt +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/stats/UsageStats.kt @@ -32,6 +32,8 @@ object UsageStats { val android: Boolean = true, val phone: Boolean = true, val cars: Boolean = true, + /** How each car session went: how far it got, how and when it ended. Off until ticked. */ + val failures: Boolean = false, ) private const val PREFS = "usage_stats" @@ -49,14 +51,18 @@ object UsageStats { /** The daily background job of earlier builds, cancelled wherever it's still scheduled. */ private const val OLD_JOB_ID = 4_207 private const val MAX_CARS = 10 + /** Each session for the "Where connections fail" share, waiting for the next note. */ + private const val KEY_SESSIONS = "sessions" + private const val KEY_LAST_SESSIONS = "last_sessions" + private const val MAX_SESSIONS = 20 private fun prefs(context: Context) = context.getSharedPreferences(PREFS, Context.MODE_PRIVATE) /** - * Dev builds show every switch and screen, so they can be tested, but never send anything: - * testing mustn't muddy the numbers. + * Debug builds send too, marked "debug". The server keeps those notes apart, so testing never + * muddies the numbers. */ - val sends: Boolean get() = !BuildConfig.DEBUG && BuildConfig.STATS_URL.isNotBlank() + val sends: Boolean get() = BuildConfig.STATS_URL.isNotBlank() fun enabled(context: Context): Boolean = prefs(context).getBoolean(KEY_ENABLED, false) @@ -69,20 +75,30 @@ object UsageStats { android = p.getBoolean("share_android", true), phone = p.getBoolean("share_phone", true), cars = p.getBoolean("share_cars", true), + failures = p.getBoolean("share_failures", false), ) } fun set(context: Context, enabled: Boolean, choices: Choices = choices(context)) { + val wasEnabled = enabled(context) prefs(context).edit() .putBoolean(KEY_ENABLED, enabled) .putBoolean(KEY_ASKED, true) .putBoolean("share_android", choices.android) .putBoolean("share_phone", choices.phone) .putBoolean("share_cars", choices.cars) + .putBoolean("share_failures", choices.failures) + // The server drops details from a box once it's unticked, so the next note sends + // them again rather than assuming the server still has them. + .remove(KEY_SENT_DETAILS) .apply() // Turning it on sends nothing yet: the driver may still be ticking boxes. The first note // waits for [choicesClosed]. - if (!enabled) prefs(context).edit().remove(KEY_CARS).remove(KEY_LAST_CARS).remove(KEY_SENT_DAY).apply() + if (!enabled) prefs(context).edit().remove(KEY_CARS).remove(KEY_LAST_CARS).remove(KEY_SENT_DAY) + .remove(KEY_SESSIONS).remove(KEY_LAST_SESSIONS).apply() + if (!choices.failures) prefs(context).edit().remove(KEY_SESSIONS).remove(KEY_LAST_SESSIONS).apply() + // Turned on anywhere - Settings, Home, or the ask itself - answers the one-time ask. + if (enabled && !wasEnabled) WorkedPrompt.onSharingOn(context) } /** The driver left the screen with the usage note choices: send today's note if it's due. */ @@ -90,10 +106,15 @@ object UsageStats { if (prefs(context).getString(KEY_SENT_DAY, null) != LocalDate.now().toString()) sendSoon(context) } - /** Gearslip was opened: send today's note unless one already went out. */ + /** + * Gearslip was opened: send today's note unless one already went out, or sooner if a session + * is still waiting, like one that ended in a crash before its note could go. + */ fun onAppOpened(context: Context) { context.getSystemService(JobScheduler::class.java)?.cancel(OLD_JOB_ID) - if (prefs(context).getString(KEY_SENT_DAY, null) != LocalDate.now().toString()) sendSoon(context) + val p = prefs(context) + val waiting = p.getString(KEY_CARS, "[]") != "[]" || p.getString(KEY_SESSIONS, "[]") != "[]" + if (waiting || p.getString(KEY_SENT_DAY, null) != LocalDate.now().toString()) sendSoon(context) } private val sending = AtomicBoolean(false) @@ -134,13 +155,40 @@ object UsageStats { outcome: String, screen: String = "", dpi: Int = 0, + session: JSONObject? = null, + link: String = "", + audio: String = "", ) { if (!enabled(context)) return val p = prefs(context) - synchronized(carsLock) { addCar(p, info, protocol, outcome, screen, dpi) } + synchronized(carsLock) { + addCar(p, info, protocol, outcome, screen, dpi, link, audio) + if (session != null && choices(context).failures) addSession(p, info, protocol, session, link, audio) + } sendSoon(context) } + private fun addSession( + p: android.content.SharedPreferences, + info: ServiceDiscovery.HeadUnitInfo?, + protocol: String, + session: JSONObject, + link: String, + audio: String, + ) { + val sessions = runCatching { JSONArray(p.getString(KEY_SESSIONS, "[]")) }.getOrDefault(JSONArray()) + if (sessions.length() >= MAX_SESSIONS) return + // The same car fields as the cars list, so the dashboard can line the two up. + session.put("name", info?.headUnitName.orEmpty()) + .put("car", info?.carModel.orEmpty()) + .put("year", info?.carYear.orEmpty()) + .put("protocol", protocol) + if (link.isNotEmpty()) session.put("link", link).put("audio", audio) + sessions.put(session) + // Written at once: after a crash the app is about to die, and a deferred write would be lost. + p.edit().putString(KEY_SESSIONS, sessions.toString()).commit() + } + private fun addCar( p: android.content.SharedPreferences, info: ServiceDiscovery.HeadUnitInfo?, @@ -148,6 +196,8 @@ object UsageStats { outcome: String, screen: String, dpi: Int, + link: String, + audio: String, ) { val cars = runCatching { JSONArray(p.getString(KEY_CARS, "[]")) }.getOrDefault(JSONArray()) val car = JSONObject() @@ -159,18 +209,45 @@ object UsageStats { .put("protocol", protocol) .put("result", outcome) if (screen.isNotEmpty()) car.put("screen", screen).put("dpi", dpi) + // USB or wireless, and whether music went over USB or Bluetooth: settings, not the person. + if (link.isNotEmpty()) car.put("link", link).put("audio", audio) // The same car and outcome twice in a day is one fact, not two. val seen = (0 until cars.length()).any { cars.optJSONObject(it)?.toString() == car.toString() } if (!seen && cars.length() < MAX_CARS) cars.put(car) - p.edit().putString(KEY_CARS, cars.toString()).apply() + p.edit().putString(KEY_CARS, cars.toString()).commit() } - // GrapheneOS reports the same build fields as stock Pixel firmware; its own apps give it away. - private fun isGrapheneOs(context: Context) = - runCatching { context.packageManager.getPackageInfo("app.grapheneos.setupwizard", 0) }.isSuccess + /** + * Which Android this is: a privacy OS by name, or "Stock" with the phone's brand. + * + * GrapheneOS reports the same build fields as stock Pixel firmware, so its own setup app gives + * it away; /e/OS is told apart from the LineageOS it's built on by its app store. The rest set + * build properties of their own. The iodéOS and DivestOS checks are best guesses from their + * build names, not confirmed on a real phone. + */ + fun osName(context: Context): String { + fun installed(pkg: String) = runCatching { context.packageManager.getPackageInfo(pkg, 0) }.isSuccess + val names = listOf("ro.build.flavor", "ro.build.display.id", "ro.lineage.version") + .joinToString(" ") { prop(it) } + " " + Build.DISPLAY + " " + Build.FINGERPRINT + return when { + installed("app.grapheneos.setupwizard") -> "GrapheneOS" + prop("ro.calyxos.version").isNotEmpty() -> "CalyxOS" + installed("foundation.e.apps") -> "/e/OS" + names.contains("iode", ignoreCase = true) -> "iodéOS" + names.contains("divest", ignoreCase = true) -> "DivestOS" + prop("ro.lineage.version").isNotEmpty() -> "LineageOS" + else -> "Stock (${Build.BRAND.replaceFirstChar { it.uppercase() }})" + } + } + + /** A system property, or "" when it isn't set or can't be read. */ + private fun prop(name: String): String = runCatching { + Class.forName("android.os.SystemProperties").getMethod("get", String::class.java) + .invoke(null, name) as String + }.getOrDefault("") private fun details(context: Context, c: Choices) = JSONObject().apply { - if (c.android) put("android", Build.VERSION.RELEASE).put("os", if (isGrapheneOs(context)) "GrapheneOS" else "other") + if (c.android) put("android", Build.VERSION.RELEASE).put("os", osName(context)) if (c.phone) put("phone", "${Build.MANUFACTURER} ${Build.MODEL}") } @@ -185,7 +262,15 @@ object UsageStats { .put("id", installId(context)) .put("version", BuildConfig.VERSION_NAME) .put("code", BuildConfig.VERSION_CODE) - .put("build", if (BuildConfig.DEBUG) "dev" else "release") + .put("build", if (BuildConfig.DEBUG) "debug" else "release") + // Which boxes are ticked, so the server knows what this phone shares and can drop + // anything it kept from a box since unticked. + .put("shares", JSONArray().apply { + if (c.android) put("android") + if (c.phone) put("phone") + if (c.cars) put("cars") + if (c.failures) put("sessions") + }) // Phone details change rarely, so they ride along only when they differ from what the // server last got, or once a month in case it lost them. Unticked ones are left out, and // the server keeps the last value it had. @@ -203,9 +288,35 @@ object UsageStats { } note.put("cars", cars) } + if (c.failures) { + var sessions = runCatching { JSONArray(p.getString(KEY_SESSIONS, "[]")) }.getOrDefault(JSONArray()) + if (everything && sessions.length() == 0) { + sessions = runCatching { JSONArray(p.getString(KEY_LAST_SESSIONS, "[]")) }.getOrDefault(JSONArray()) + } + note.put("sessions", sessions) + } return note } + /** + * A made-up session, for "What gets sent" to show before the first real one: a 25-minute + * drive that ended with the car switched off. + */ + val exampleSession: JSONObject + get() = JSONObject() + .put("stage", "streaming") + .put("ended", "byebye") + .put("code", "bye 1") + .put("failed_at", 1500) + .put("seconds", 1500) + .put("cert", "downloaded") + .put("name", "Head unit name") + .put("car", "Car model") + .put("year", "2021") + .put("protocol", "1.5") + .put("link", "usb") + .put("audio", "bluetooth") + /** Sends the note now; false when it didn't get through. */ private fun send(context: Context): Boolean { if (!sends || !enabled(context)) return true @@ -213,6 +324,7 @@ object UsageStats { val p = prefs(context) val note = synchronized(carsLock) { preview(context) } val sentCars = note.optJSONArray("cars")?.toString() + val sentSessions = note.optJSONArray("sessions")?.toString() val sentDetails = if (note.has("android") || note.has("phone")) details(context, choices(context)).toString() else null val body = note.toString().toByteArray() return runCatching { @@ -234,6 +346,16 @@ object UsageStats { if (sentDetails != null) edit.putString(KEY_SENT_DETAILS, sentDetails).putString(KEY_DETAILS_DAY, LocalDate.now().toString()) if (sentCars != null && sentCars != "[]") edit.putString(KEY_LAST_CARS, sentCars) if (sentCars == null || p.getString(KEY_CARS, "[]") == sentCars) edit.remove(KEY_CARS) + if (sentSessions != null && sentSessions != "[]") edit.putString(KEY_LAST_SESSIONS, sentSessions) + if (sentSessions == null || p.getString(KEY_SESSIONS, "[]") == sentSessions) { + edit.remove(KEY_SESSIONS) + } else if (sentSessions != "[]") { + // A session ended mid-send: keep only the ones this note didn't carry. + val now = JSONArray(p.getString(KEY_SESSIONS, "[]")) + val sent = JSONArray(sentSessions).length() + val left = JSONArray().also { for (i in sent until now.length()) it.put(now.get(i)) } + edit.putString(KEY_SESSIONS, left.toString()) + } edit.apply() } GearslipLog.i("stats: note sent") diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/stats/WorkedPrompt.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/stats/WorkedPrompt.kt new file mode 100644 index 0000000..ac4f487 --- /dev/null +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/stats/WorkedPrompt.kt @@ -0,0 +1,115 @@ +package app.seb3thehacker.gearslip.stats + +import android.content.Context +import app.seb3thehacker.gearslip.GearslipLog +import app.seb3thehacker.gearslip.ServiceDiscovery +import kotlinx.coroutines.flow.MutableStateFlow +import kotlinx.coroutines.flow.StateFlow +import org.json.JSONObject + +/** + * The one-time "Gearslip worked with your car. Share that?" ask, for drivers who never answered + * the usage notes question in setup, either way. It comes after the first drive that worked, when a driver is happiest to help, as + * a card on Home. Answering it clears it, and it never comes back. + * + * Saying yes turns on the same usage notes switch as Settings, and that drive's car goes in the + * first note. Until the driver answers, the car waits here on the phone and is sent nowhere. + */ +object WorkedPrompt { + + private const val PREFS = "worked_prompt" + private const val KEY_DONE = "done" + private const val KEY_CAR = "car" + + private fun prefs(context: Context) = context.getSharedPreferences(PREFS, Context.MODE_PRIVATE) + + private var loaded = false + private val flow = MutableStateFlow(null) + + /** The car's name while the ask is waiting for an answer, otherwise null. */ + fun pending(context: Context): StateFlow { + if (!loaded) { + loaded = true + flow.value = saved(context)?.takeUnless { UsageStats.asked(context) }?.let(::carName) + } + return flow + } + + /** A drive that reached the car's screen ended cleanly. Asks, if it's never asked before. */ + fun onDriveWorked(context: Context, info: ServiceDiscovery.HeadUnitInfo?, protocol: String, screen: String, dpi: Int) { + val p = prefs(context) + // Asked once only: a driver who answered in setup already said yes or no. + if (UsageStats.asked(context) || p.getBoolean(KEY_DONE, false) || p.contains(KEY_CAR)) return + val car = JSONObject() + .put("name", info?.headUnitName.orEmpty()) + .put("car", info?.carModel.orEmpty()) + .put("year", info?.carYear.orEmpty()) + .put("make", info?.headUnitMake.orEmpty()) + .put("model", info?.headUnitModel.orEmpty()) + .put("protocol", protocol) + .put("screen", screen) + .put("dpi", dpi) + p.edit().putString(KEY_CAR, car.toString()).apply() + loaded = true + flow.value = carName(car) + } + + /** True once a driver turns sharing on from the ask, until Home has thrown its confetti. */ + private val celebrateFlow = MutableStateFlow(false) + val celebrate: StateFlow = celebrateFlow + + fun celebrated() { celebrateFlow.value = false } + + /** The driver answered on Home. Either way it's never asked again. */ + fun answer(context: Context, share: Boolean) { + if (share) { + // Turning sharing on lands in [onSharingOn], which clears the ask. + UsageStats.set(context, true) + return + } + clear(context) + GearslipLog.i("worked prompt: not now") + } + + /** + * Usage notes were just turned on, from the ask or anywhere else. With the ask still waiting, + * that answers it, and the car it was about goes in the first note. + */ + internal fun onSharingOn(context: Context) { + val car = saved(context) ?: return + clear(context) + celebrateFlow.value = true + GearslipLog.i("worked prompt: usage notes turned on") + val info = ServiceDiscovery.HeadUnitInfo( + car.optString("name"), car.optString("car"), car.optString("year"), + car.optString("make"), car.optString("model"), + ) + UsageStats.recordSession( + context, info, car.optString("protocol"), "connected", + car.optString("screen"), car.optInt("dpi"), + ) + } + + private fun clear(context: Context) { + prefs(context).edit().remove(KEY_CAR).putBoolean(KEY_DONE, true).apply() + loaded = true + flow.value = null + } + + private fun saved(context: Context): JSONObject? = + prefs(context).getString(KEY_CAR, null)?.let { runCatching { JSONObject(it) }.getOrNull() } + + /** "your 2019 Clio", "your Renault Media Nav", or "your car" when it didn't say. */ + private fun carName(car: JSONObject): String { + val model = car.optString("car").trim() + val year = car.optString("year").trim().takeIf { it.isNotEmpty() && it != "0" } + val unit = "${car.optString("make")} ${car.optString("model")}".trim() + return when { + model.isNotEmpty() -> "your " + listOfNotNull(year, model).joinToString(" ") + unit.isNotEmpty() -> "your $unit" + else -> "your car" + } + } + + const val TITLE_START = "Gearslip worked with " +} diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/ui/AboutLinks.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/ui/AboutLinks.kt new file mode 100644 index 0000000..bd9898c --- /dev/null +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/ui/AboutLinks.kt @@ -0,0 +1,180 @@ +package app.seb3thehacker.gearslip.ui + +import android.content.ClipData +import android.content.ClipboardManager +import android.content.Context +import android.content.Intent +import android.net.Uri +import android.widget.Toast +import androidx.compose.foundation.clickable +import androidx.compose.material.icons.Icons +import androidx.compose.material.icons.automirrored.filled.KeyboardArrowRight +import androidx.compose.material.icons.automirrored.filled.Send +import androidx.compose.material.icons.filled.Build +import androidx.compose.material.icons.filled.Favorite +import androidx.compose.foundation.layout.Arrangement +import androidx.compose.foundation.layout.Column +import androidx.compose.foundation.layout.fillMaxWidth +import androidx.compose.foundation.text.selection.SelectionContainer +import androidx.compose.material3.AlertDialog +import androidx.compose.material3.Button +import androidx.compose.material3.Icon +import androidx.compose.material3.MaterialTheme +import androidx.compose.material3.OutlinedButton +import androidx.compose.material3.Text +import androidx.compose.material3.TextButton +import androidx.compose.runtime.Composable +import androidx.compose.runtime.getValue +import androidx.compose.runtime.mutableStateOf +import androidx.compose.runtime.remember +import androidx.compose.runtime.setValue +import androidx.compose.ui.Modifier +import androidx.compose.ui.text.font.FontFamily +import androidx.compose.ui.unit.dp +import androidx.compose.ui.platform.LocalContext + +/** Where Gearslip lives outside the app: support, the chat room, and the code. */ +internal object GearslipLinks { + const val LIBERAPAY = "https://liberapay.com/Seb3thehacker" + /** A card payment page for giving once, any amount. Its row stays hidden while this is empty. */ + const val PAYMENT_LINK = "" + /** Gearslip's Monero address. Its row stays hidden while this is empty. */ + const val MONERO = "44gTVxQJY3cirBJ41J5i1bPUVnSkTMhn9esSp1kg9MHwJ3TUz8tkkigUTKwiok5zA5RZvNxyM4pqkAfPs79pPLfaNG21pCn" + const val MATRIX = "https://matrix.to/#/!mf0_mgBd5KqKpc9FfX_wHp1jFCcwAtPrndIn_oGIyMk?via=matrix.org" + const val SOURCE = "https://github.com/Seb3thehacker/gearslip" + const val CARS = "$SOURCE/blob/main/CARS.md" + + fun open(context: Context, url: String) { + runCatching { + context.startActivity(Intent(Intent.ACTION_VIEW, Uri.parse(url)).addFlags(Intent.FLAG_ACTIVITY_NEW_TASK)) + } + } + + /** Opens a Monero wallet app on the address; says so when none is installed. */ + fun openMoneroWallet(context: Context) { + runCatching { + context.startActivity(Intent(Intent.ACTION_VIEW, Uri.parse("monero:$MONERO")).addFlags(Intent.FLAG_ACTIVITY_NEW_TASK)) + }.onFailure { + Toast.makeText(context, "No Monero wallet app found. Copy the address instead.", Toast.LENGTH_LONG).show() + } + } + + fun copyMonero(context: Context) { + context.getSystemService(ClipboardManager::class.java) + ?.setPrimaryClip(ClipData.newPlainText("Monero address", MONERO)) + Toast.makeText(context, "Monero address copied", Toast.LENGTH_SHORT).show() + } +} + +/** The ways to support Gearslip. Liberapay always; card and Monero once they're set up. */ +@Composable +internal fun DonateRows() { + val context = LocalContext.current + val arrow = @Composable { + Icon( + Icons.AutoMirrored.Filled.KeyboardArrowRight, + contentDescription = null, + tint = MaterialTheme.colorScheme.onSurfaceVariant, + ) + } + SettingsRow( + "Liberapay", + Modifier.clickable { GearslipLinks.open(context, GearslipLinks.LIBERAPAY) }, + subtitle = "Give weekly or monthly", + icon = Icons.Filled.Favorite, + trailing = arrow, + ) + if (GearslipLinks.PAYMENT_LINK.isNotEmpty()) { + SettingsDivider() + SettingsRow( + "Give once", + Modifier.clickable { GearslipLinks.open(context, GearslipLinks.PAYMENT_LINK) }, + subtitle = "Any amount, by card", + icon = Icons.Filled.Favorite, + trailing = arrow, + ) + } + if (GearslipLinks.MONERO.isNotEmpty()) { + var moneroOpen by remember { mutableStateOf(false) } + SettingsDivider() + SettingsRow( + "Monero", + Modifier.clickable { moneroOpen = true }, + subtitle = "Open your wallet or copy the address", + icon = Icons.Filled.Favorite, + trailing = arrow, + ) + if (moneroOpen) MoneroDialog { moneroOpen = false } + } +} + +/** The Monero address in full, with ways to pay it from this phone or another device. */ +@Composable +private fun MoneroDialog(onDismiss: () -> Unit) { + val context = LocalContext.current + AlertDialog( + onDismissRequest = onDismiss, + icon = { Icon(Icons.Filled.Favorite, contentDescription = null) }, + title = { Text("Support with Monero") }, + text = { + Column(verticalArrangement = Arrangement.spacedBy(12.dp)) { + Text("Send any amount to this address from your Monero wallet.") + SelectionContainer { + Text( + GearslipLinks.MONERO, + style = MaterialTheme.typography.bodySmall.copy(fontFamily = FontFamily.Monospace), + color = MaterialTheme.colorScheme.onSurfaceVariant, + ) + } + } + }, + confirmButton = { + Column(Modifier.fillMaxWidth(), verticalArrangement = Arrangement.spacedBy(8.dp)) { + Button(onClick = { GearslipLinks.openMoneroWallet(context) }, Modifier.fillMaxWidth()) { + Text("Open in wallet") + } + OutlinedButton(onClick = { GearslipLinks.copyMonero(context) }, Modifier.fillMaxWidth()) { + Text("Copy address") + } + TextButton(onClick = onDismiss, Modifier.fillMaxWidth()) { + Text("Close") + } + } + }, + ) +} + +/** The About section's links: the chat room, the source code, and credits. */ +@Composable +internal fun AboutLinkRows() { + val context = LocalContext.current + val arrow = @Composable { + Icon( + Icons.AutoMirrored.Filled.KeyboardArrowRight, + contentDescription = null, + tint = MaterialTheme.colorScheme.onSurfaceVariant, + ) + } + SettingsRow( + "Join the chat on Matrix", + Modifier.clickable { GearslipLinks.open(context, GearslipLinks.MATRIX) }, + subtitle = "Talk with other drivers about cars, apps, and Gearslip", + icon = Icons.AutoMirrored.Filled.Send, + trailing = arrow, + ) + SettingsDivider() + SettingsRow( + "Source code", + Modifier.clickable { GearslipLinks.open(context, GearslipLinks.SOURCE) }, + subtitle = "Read or change Gearslip's code under the AGPL", + icon = Icons.Filled.Build, + trailing = arrow, + ) + SettingsDivider() + SettingsRow( + "Voice credits", + subtitle = "Speech recognition by sherpa-onnx (Apache 2.0), with Kroko ASR by Banafo " + + "(CC BY-SA 4.0), and Parakeet and Canary by NVIDIA (CC BY 4.0)", + icon = Icons.Filled.Favorite, + ) +} diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/ui/AudioChoiceDialog.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/ui/AudioChoiceDialog.kt new file mode 100644 index 0000000..0314586 --- /dev/null +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/ui/AudioChoiceDialog.kt @@ -0,0 +1,71 @@ +package app.seb3thehacker.gearslip.ui + +import androidx.compose.foundation.layout.Arrangement +import androidx.compose.foundation.layout.Column +import androidx.compose.material3.AlertDialog +import androidx.compose.material3.MaterialTheme +import androidx.compose.material3.Text +import androidx.compose.material3.TextButton +import androidx.compose.runtime.Composable +import androidx.compose.runtime.getValue +import androidx.compose.runtime.mutableStateOf +import androidx.compose.runtime.remember +import androidx.compose.runtime.setValue +import androidx.compose.ui.unit.dp +import app.seb3thehacker.gearslip.BuildConfig + +/** + * Shown once, the first time the driver turns down Android's screen-sharing dialog: that dialog + * is the price of sending music over USB, and most people who refuse it would rather not see it + * again. Offers the two ways out: Bluetooth, or the one-time adb command. + */ +@Composable +fun AudioChoiceDialog(onUseBluetooth: () -> Unit, onKeepAsking: () -> Unit) { + var showCommand by remember { mutableStateOf(false) } + AlertDialog( + onDismissRequest = onKeepAsking, + title = { Text("Play car audio over Bluetooth?") }, + text = { + Column(verticalArrangement = Arrangement.spacedBy(12.dp)) { + Text( + "Android treats music sent over USB like sharing your screen, so it asks every " + + "drive. Over Bluetooth it never asks. You can change this later in Settings.", + ) + if (showCommand) { + Text( + "Or keep USB and stop the question for good: plug the phone into a " + + "computer with USB debugging on, and run this once.", + style = MaterialTheme.typography.bodyMedium, + ) + CopyableCommand("adb shell appops set ${BuildConfig.APPLICATION_ID} PROJECT_MEDIA allow") + } else { + TextButton(onClick = { showCommand = true }) { Text("Never ask, using a computer") } + } + } + }, + confirmButton = { TextButton(onClick = onUseBluetooth) { Text("Use Bluetooth") } }, + dismissButton = { TextButton(onClick = onKeepAsking) { Text("Keep asking") } }, + ) +} + +/** + * Android lets an app capture other apps' audio only while it holds the microphone permission. + * Once the driver has refused that twice, Android stops asking, and music silently stays on the + * phone; this says why and where to turn it back on. + */ +@Composable +fun MicBlockedDialog(onOpenSettings: () -> Unit, onDismiss: () -> Unit) { + AlertDialog( + onDismissRequest = onDismiss, + title = { Text("Car audio needs the microphone permission") }, + text = { + Text( + "Android lets an app send other apps' sound to the car only if it has the " + + "microphone permission. It's turned off for Gearslip, so music plays on the " + + "phone. Turn it on under Permissions, then Microphone.", + ) + }, + confirmButton = { TextButton(onClick = onOpenSettings) { Text("Open app settings") } }, + dismissButton = { TextButton(onClick = onDismiss) { Text("Not now") } }, + ) +} diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/ui/CertSetupScreen.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/ui/CertSetupScreen.kt index 870d7b6..37a2ad2 100644 --- a/gearslip/src/main/java/app/seb3thehacker/gearslip/ui/CertSetupScreen.kt +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/ui/CertSetupScreen.kt @@ -60,7 +60,6 @@ fun CertSetupScreen(onDone: () -> Unit) { val scope = rememberCoroutineScope() var downloadState by remember { mutableStateOf(DownloadState.Idle) } - var showImport by remember { mutableStateOf(false) } var pendingUri by remember { mutableStateOf(null) } var manualProgress by remember { mutableStateOf(false) } @@ -91,7 +90,7 @@ fun CertSetupScreen(onDone: () -> Unit) { if (installed) { "Cars accept only Google-issued certificates, and you have one loaded." } else { - "Cars accept only Google-issued certificates. Download one to connect." + "Cars accept only Google-issued certificates. Gearslip doesn't include one, so choose where yours comes from." }, style = MaterialTheme.typography.bodyLarge, textAlign = TextAlign.Center, @@ -131,10 +130,11 @@ fun CertSetupScreen(onDone: () -> Unit) { }, modifier = Modifier.fillMaxWidth().height(56.dp), ) { - Text("Download certificate") + Text("Download from aasdk") } Text( - "Works with most cars built before 2020. Newer cars reject it.", + "The open-source aasdk project publishes this certificate at github.com/opencardev/aasdk. " + + "Most cars accept it; some from around 2020 on don't.", style = MaterialTheme.typography.bodySmall, textAlign = TextAlign.Center, color = MaterialTheme.colorScheme.onSurfaceVariant, @@ -177,16 +177,18 @@ fun CertSetupScreen(onDone: () -> Unit) { Spacer(Modifier.height(16.dp)) - if (showImport) { - OutlinedButton( - onClick = { importLauncher.launch(arrayOf("*/*")) }, - modifier = Modifier.fillMaxWidth().height(52.dp), - ) { Text("Import your own .p12 cert") } - Spacer(Modifier.height(16.dp)) - } else { - TextButton(onClick = { showImport = true }) { Text("Import your own certificate instead") } - Spacer(Modifier.height(8.dp)) - } + OutlinedButton( + onClick = { importLauncher.launch(arrayOf("*/*")) }, + modifier = Modifier.fillMaxWidth().height(52.dp), + ) { Text("Import my own file") } + Text( + "A .p12 file with its private key, from wherever you choose.", + style = MaterialTheme.typography.bodySmall, + textAlign = TextAlign.Center, + color = MaterialTheme.colorScheme.onSurfaceVariant, + modifier = Modifier.padding(top = 8.dp), + ) + Spacer(Modifier.height(16.dp)) OutlinedButton( onClick = { diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/ui/CertSummary.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/ui/CertSummary.kt index f06c7f9..48716ca 100644 --- a/gearslip/src/main/java/app/seb3thehacker/gearslip/ui/CertSummary.kt +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/ui/CertSummary.kt @@ -14,8 +14,17 @@ data class CertSummary( val validUntil: String?, ) { companion object { + /** + * The last [read], so Settings draws the certificate row at once instead of a second + * later, which shifted everything under it. Null until the first read. + */ + @Volatile var cached: CertSummary? = null + private set + /** Blocking (parses a key store): call from a background dispatcher. */ - fun read(context: Context): CertSummary { + fun read(context: Context): CertSummary = readNow(context).also { cached = it } + + private fun readNow(context: Context): CertSummary { val identity = CertProvider.loadSupplied(context) ?: return CertSummary( CertProvider.Kind.SELF_SIGNED, diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/ui/Confetti.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/ui/Confetti.kt new file mode 100644 index 0000000..61ce7e4 --- /dev/null +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/ui/Confetti.kt @@ -0,0 +1,88 @@ +package app.seb3thehacker.gearslip.ui + +import androidx.compose.foundation.Canvas +import androidx.compose.foundation.layout.fillMaxSize +import androidx.compose.material3.MaterialTheme +import androidx.compose.runtime.Composable +import androidx.compose.runtime.LaunchedEffect +import androidx.compose.runtime.getValue +import androidx.compose.runtime.mutableFloatStateOf +import androidx.compose.runtime.remember +import androidx.compose.runtime.setValue +import androidx.compose.runtime.withFrameNanos +import androidx.compose.ui.Modifier +import androidx.compose.ui.geometry.Offset +import androidx.compose.ui.geometry.Size +import androidx.compose.ui.graphics.Color +import androidx.compose.ui.graphics.drawscope.rotate +import kotlin.math.sin +import kotlin.random.Random + +private class Piece( + val x: Float, // across the screen, 0..1 + val delay: Float, // seconds before it starts falling + val speed: Float, // screen heights per second + val sway: Float, // how far it drifts side to side, in screen widths + val swaySpeed: Float, + val spin: Float, // degrees per second + val width: Float, // dp + val height: Float, + val color: Color, +) + +/** + * Confetti falling from the top of the screen once, for about three seconds. Draws over whatever + * is under it and takes no touches; [onDone] fires when the last piece has left the screen. + */ +@Composable +internal fun ConfettiFall(onDone: () -> Unit, modifier: Modifier = Modifier) { + val scheme = MaterialTheme.colorScheme + val colors = listOf( + scheme.primary, scheme.tertiary, scheme.secondary, + Color(0xFFFFC107), Color(0xFFE91E63), Color(0xFF4CAF50), Color(0xFF2196F3), + ) + val pieces = remember { + List(PIECES) { + Piece( + x = Random.nextFloat(), + delay = Random.nextFloat() * 0.9f, + speed = 0.35f + Random.nextFloat() * 0.35f, + sway = 0.01f + Random.nextFloat() * 0.03f, + swaySpeed = 2f + Random.nextFloat() * 4f, + spin = (Random.nextFloat() - 0.5f) * 720f, + width = 4f + Random.nextFloat() * 4f, + height = 8f + Random.nextFloat() * 6f, + color = colors[Random.nextInt(colors.size)], + ) + } + } + var t by remember { mutableFloatStateOf(0f) } + LaunchedEffect(Unit) { + val start = withFrameNanos { it } + // The slowest piece, starting last, clears the screen by this time. + val end = pieces.maxOf { it.delay + 1.1f / it.speed } + while (t < end) { + withFrameNanos { t = (it - start) / 1e9f } + } + onDone() + } + Canvas(modifier.fillMaxSize()) { + val density = density + for (p in pieces) { + val age = t - p.delay + if (age < 0f) continue + val y = (age * p.speed - 0.05f) * size.height + if (y > size.height + 40f) continue + val x = (p.x + sin(age * p.swaySpeed) * p.sway) * size.width + rotate(age * p.spin, pivot = Offset(x, y)) { + drawRect( + p.color, + topLeft = Offset(x - p.width * density / 2, y - p.height * density / 2), + size = Size(p.width * density, p.height * density), + ) + } + } + } +} + +private const val PIECES = 140 diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/ui/DonatePrompt.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/ui/DonatePrompt.kt new file mode 100644 index 0000000..25c1c9e --- /dev/null +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/ui/DonatePrompt.kt @@ -0,0 +1,119 @@ +package app.seb3thehacker.gearslip.ui + +import android.content.Context +import androidx.compose.foundation.layout.Arrangement +import androidx.compose.foundation.layout.Column +import androidx.compose.foundation.layout.fillMaxWidth +import androidx.compose.material.icons.Icons +import androidx.compose.material.icons.filled.Favorite +import androidx.compose.material3.AlertDialog +import androidx.compose.material3.Button +import androidx.compose.material3.Icon +import androidx.compose.material3.OutlinedButton +import androidx.compose.material3.Text +import androidx.compose.material3.TextButton +import androidx.compose.runtime.Composable +import androidx.compose.runtime.getValue +import androidx.compose.runtime.remember +import androidx.compose.ui.Modifier +import androidx.compose.ui.platform.LocalContext +import androidx.compose.ui.unit.dp +import androidx.lifecycle.compose.collectAsStateWithLifecycle +import app.seb3thehacker.gearslip.GearslipLog +import kotlinx.coroutines.flow.MutableStateFlow +import kotlinx.coroutines.flow.StateFlow + +/** + * The donation ask on Home, after a drive that worked. A driver sees it first after their third + * such drive, so Gearslip has earned it, and then at most once every three months. Tapping either + * donate button ends it for good. + */ +object DonatePrompt { + + private const val PREFS = "donate_prompt" + private const val KEY_DRIVES = "drives" + private const val KEY_LAST = "last_shown" + private const val KEY_DONATED = "donated" + private const val KEY_WAITING = "waiting" + private const val FIRST_AFTER = 3 + private const val GAP_MS = 90L * 24 * 60 * 60 * 1000 + + private fun prefs(context: Context) = context.getSharedPreferences(PREFS, Context.MODE_PRIVATE) + + private var loaded = false + private val flow = MutableStateFlow(false) + + /** True while the ask is waiting for the driver to open Home. */ + fun pending(context: Context): StateFlow { + if (!loaded) { + loaded = true + flow.value = prefs(context).getBoolean(KEY_WAITING, false) + } + return flow + } + + /** A drive that reached the car's screen ended cleanly. Asks if it's time. */ + fun onDriveWorked(context: Context) { + val p = prefs(context) + if (p.getBoolean(KEY_DONATED, false)) return + val drives = p.getInt(KEY_DRIVES, 0) + 1 + val due = drives >= FIRST_AFTER && System.currentTimeMillis() - p.getLong(KEY_LAST, 0L) > GAP_MS + p.edit().putInt(KEY_DRIVES, drives).apply { + if (due) putBoolean(KEY_WAITING, true) + }.apply() + if (due) { + loaded = true + flow.value = true + } + } + + /** The driver closed the ask; [donated] if they picked a way to give. */ + fun answer(context: Context, donated: Boolean) { + prefs(context).edit() + .putBoolean(KEY_WAITING, false) + .putLong(KEY_LAST, System.currentTimeMillis()) + .apply { if (donated) putBoolean(KEY_DONATED, true) } + .apply() + loaded = true + flow.value = false + GearslipLog.i(if (donated) "donate prompt: opened a donation link" else "donate prompt: not now") + } +} + +/** Shows the donation ask over Home when one is waiting and the car isn't connected. */ +@Composable +internal fun DonateDialog(live: Boolean) { + val context = LocalContext.current + val waiting by remember { DonatePrompt.pending(context) }.collectAsStateWithLifecycle() + if (!waiting || live) return + fun give(url: String) { + DonatePrompt.answer(context, donated = true) + GearslipLinks.open(context, url) + } + AlertDialog( + onDismissRequest = { DonatePrompt.answer(context, donated = false) }, + icon = { Icon(Icons.Filled.Favorite, contentDescription = null) }, + title = { Text("Glad that drive worked") }, + text = { + Text( + "Gearslip is free, with no ads and no tracking. If it gets you on the road, your " + + "support keeps it going.", + ) + }, + confirmButton = { + Column(Modifier.fillMaxWidth(), verticalArrangement = Arrangement.spacedBy(8.dp)) { + Button(onClick = { give(GearslipLinks.LIBERAPAY) }, Modifier.fillMaxWidth()) { + Text("Support on Liberapay") + } + if (GearslipLinks.PAYMENT_LINK.isNotEmpty()) { + OutlinedButton(onClick = { give(GearslipLinks.PAYMENT_LINK) }, Modifier.fillMaxWidth()) { + Text("Give once by card") + } + } + TextButton(onClick = { DonatePrompt.answer(context, donated = false) }, Modifier.fillMaxWidth()) { + Text("Not now") + } + } + }, + ) +} diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/ui/GearslipApp.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/ui/GearslipApp.kt index 1a23888..78e894f 100644 --- a/gearslip/src/main/java/app/seb3thehacker/gearslip/ui/GearslipApp.kt +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/ui/GearslipApp.kt @@ -16,11 +16,9 @@ import androidx.compose.foundation.layout.fillMaxWidth import androidx.compose.foundation.layout.height import androidx.compose.foundation.layout.statusBars import androidx.compose.foundation.layout.windowInsetsPadding -import androidx.compose.material3.AlertDialog import androidx.compose.material3.MaterialTheme import androidx.compose.material3.Surface import androidx.compose.material3.Text -import androidx.compose.material3.TextButton import androidx.compose.runtime.Composable import androidx.compose.runtime.LaunchedEffect import androidx.compose.runtime.getValue @@ -39,7 +37,7 @@ import app.seb3thehacker.gearslip.BuildConfig import app.seb3thehacker.gearslip.CertProvider import app.seb3thehacker.gearslip.stats.UsageStats -private enum class Screen { HOME, LOGS, SETTINGS, CAR_PREVIEW, CERT_SETUP, SETUP_GUIDE, USAGE_NOTES, HELP, WHATS_NEW, RECOMMENDED_APPS } +private enum class Screen { INTRO, HOME, LOGS, SETTINGS, CAR_PREVIEW, CERT_SETUP, SETUP_GUIDE, USAGE_NOTES, HELP, WHATS_NEW, RECOMMENDED_APPS } /** Three screens and a back stack of depth one: no navigation library needed. */ @Composable @@ -49,19 +47,29 @@ fun GearslipApp(onDisconnect: () -> Unit, onRequestCallScreening: () -> Unit) { !AppSettings.hasSkippedCertSetup(context) && !CertProvider.hasAnyCert(context) } val needsSetupGuide = remember { !AppSettings.hasSeenPermissionsSetup(context) } - var screen by rememberSaveable { - mutableStateOf( - when { - needsCertSetup -> Screen.CERT_SETUP - needsSetupGuide -> Screen.SETUP_GUIDE - // Set up before usage notes existed: ask once, on the setup guide's own page. - !UsageStats.asked(context) -> Screen.USAGE_NOTES - else -> Screen.HOME - }, - ) + // Where launch goes once the intro (if any) is done. + val afterIntro = remember { + when { + needsCertSetup -> Screen.CERT_SETUP + needsSetupGuide -> Screen.SETUP_GUIDE + // Set up before usage notes existed: ask once, on the setup guide's own page. + !UsageStats.asked(context) -> Screen.USAGE_NOTES + else -> Screen.HOME + } } - BackHandler(enabled = screen != Screen.HOME) { - screen = if (screen == Screen.RECOMMENDED_APPS) Screen.SETTINGS else Screen.HOME + var screen by rememberSaveable { mutableStateOf(if (Intro.due(context)) Screen.INTRO else afterIntro) } + // A replay from About goes back to Settings; the launch one carries on into setup. + var introReturn by rememberSaveable { mutableStateOf(afterIntro) } + // Home's "it worked" card opens Settings at usage notes rather than at the top. + var settingsAtUsageNotes by rememberSaveable { mutableStateOf(false) } + // What's new and Connection help open from Home and from Settings; back returns to whichever. + var backTo by rememberSaveable { mutableStateOf(Screen.SETTINGS) } + BackHandler(enabled = screen != Screen.HOME && screen != Screen.INTRO) { + screen = when (screen) { + Screen.RECOMMENDED_APPS, Screen.LOGS, Screen.CAR_PREVIEW -> Screen.SETTINGS + Screen.HELP, Screen.WHATS_NEW -> backTo + else -> Screen.HOME + } } // Home stays upright; every other screen turns with the phone. @@ -75,6 +83,10 @@ fun GearslipApp(onDisconnect: () -> Unit, onRequestCallScreening: () -> Unit) { // The guide asks for this now, but anyone who finished it before that never saw the page. val notificationPermission = rememberLauncherForActivityResult(ActivityResultContracts.RequestPermission()) {} + // Reads the certificate once in the background, so Settings has it ready when opened. + LaunchedEffect(Unit) { + kotlinx.coroutines.withContext(kotlinx.coroutines.Dispatchers.Default) { runCatching { CertSummary.read(context) } } + } LaunchedEffect(Unit) { if (!needsSetupGuide && context.checkSelfPermission(Manifest.permission.POST_NOTIFICATIONS) != PackageManager.PERMISSION_GRANTED @@ -89,29 +101,35 @@ fun GearslipApp(onDisconnect: () -> Unit, onRequestCallScreening: () -> Unit) { .windowInsetsPadding(WindowInsets.statusBars) .consumeWindowInsets(WindowInsets.statusBars), ) { - if (BuildConfig.DEBUG && screen != Screen.CAR_PREVIEW) DevBuildBanner() + if (BuildConfig.DEBUG && screen != Screen.CAR_PREVIEW && screen != Screen.INTRO) DebugBuildBanner() Column(Modifier.weight(1f)) { when (screen) { + Screen.INTRO -> IntroScreen(onDone = { screen = introReturn }) Screen.HOME -> HomeScreen( - onOpenLogs = { screen = Screen.LOGS }, - onOpenSettings = { screen = Screen.SETTINGS }, - onOpenCarPreview = { screen = Screen.CAR_PREVIEW }, - onOpenHelp = { screen = Screen.HELP }, + onOpenSettings = { settingsAtUsageNotes = false; screen = Screen.SETTINGS }, + onOpenWhatsNew = { backTo = Screen.HOME; screen = Screen.WHATS_NEW }, + onOpenHelp = { backTo = Screen.HOME; screen = Screen.HELP }, + onOpenUsageNotes = { settingsAtUsageNotes = true; screen = Screen.SETTINGS }, onDisconnect = onDisconnect, ) - Screen.WHATS_NEW -> WhatsNewScreen(onBack = { screen = Screen.SETTINGS }) - Screen.LOGS -> LogsScreen(onBack = { screen = Screen.HOME }) + Screen.WHATS_NEW -> WhatsNewScreen(onBack = { screen = backTo }) + Screen.LOGS -> LogsScreen(onBack = { screen = Screen.SETTINGS }) Screen.SETTINGS -> SettingsScreen( onBack = { screen = Screen.HOME }, - onOpenRecommendedApps = { screen = Screen.RECOMMENDED_APPS }, - onOpenWhatsNew = { screen = Screen.WHATS_NEW }, + onOpenRecommendedApps = { settingsAtUsageNotes = false; screen = Screen.RECOMMENDED_APPS }, + onOpenWhatsNew = { settingsAtUsageNotes = false; backTo = Screen.SETTINGS; screen = Screen.WHATS_NEW }, + onOpenHelp = { settingsAtUsageNotes = false; backTo = Screen.SETTINGS; screen = Screen.HELP }, + onOpenCarPreview = { settingsAtUsageNotes = false; screen = Screen.CAR_PREVIEW }, + onOpenLogs = { settingsAtUsageNotes = false; screen = Screen.LOGS }, onReplayTutorial = { screen = Screen.SETUP_GUIDE }, + onReplayIntro = { Intro.replay(context); introReturn = Screen.SETTINGS; screen = Screen.INTRO }, + atUsageNotes = settingsAtUsageNotes, ) Screen.RECOMMENDED_APPS -> RecommendedAppsScreen(onBack = { screen = Screen.SETTINGS }) - Screen.CAR_PREVIEW -> CarPreviewScreen(onBack = { screen = Screen.HOME }) + Screen.CAR_PREVIEW -> CarPreviewScreen(onBack = { screen = Screen.SETTINGS }) Screen.HELP -> HelpScreen( - onBack = { screen = Screen.HOME }, + onBack = { screen = backTo }, onRequestCallScreening = onRequestCallScreening, ) Screen.CERT_SETUP -> CertSetupScreen( @@ -133,8 +151,6 @@ fun GearslipApp(onDisconnect: () -> Unit, onRequestCallScreening: () -> Unit) { // A car keyboard is open: offer the phone's own, for a passenger. if (screen != Screen.CAR_PREVIEW) PhoneTypingBar() } - - CompatWarning() } /** @@ -144,7 +160,7 @@ fun GearslipApp(onDisconnect: () -> Unit, onRequestCallScreening: () -> Unit) { * never mistaken for a release one. */ @Composable -private fun DevBuildBanner() { +private fun DebugBuildBanner() { Surface( modifier = Modifier.fillMaxWidth().height(28.dp), color = Color(0xFFF5C518), @@ -152,36 +168,10 @@ private fun DevBuildBanner() { ) { Box(Modifier.fillMaxSize(), contentAlignment = Alignment.Center) { Text( - "DEV BUILD", + "DEBUG BUILD", style = MaterialTheme.typography.labelSmall, fontWeight = FontWeight.Bold, ) } } } - -/** Shown once, on first run: the certificate is known to fail on head units newer than ~2020. */ -@Composable -private fun CompatWarning() { - val context = LocalContext.current - var visible by rememberSaveable { mutableStateOf(!AppSettings.hasSeenCompatWarning(context)) } - - if (visible) { - AlertDialog( - onDismissRequest = {}, - title = { Text("Before you plug in") }, - text = { - Text( - "Gearslip works with cars built before 2020. " + - "Newer firmware often blocks the connection.", - ) - }, - confirmButton = { - TextButton(onClick = { - AppSettings.setSeenCompatWarning(context) - visible = false - }) { Text("Got it") } - }, - ) - } -} diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/ui/HelpScreen.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/ui/HelpScreen.kt index 53ecf42..f2152b9 100644 --- a/gearslip/src/main/java/app/seb3thehacker/gearslip/ui/HelpScreen.kt +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/ui/HelpScreen.kt @@ -1,6 +1,7 @@ package app.seb3thehacker.gearslip.ui import app.seb3thehacker.gearslip.BuildConfig +import app.seb3thehacker.gearslip.car.CarSettings import androidx.compose.foundation.layout.Arrangement import androidx.compose.foundation.layout.Column import androidx.compose.foundation.layout.Row @@ -23,13 +24,16 @@ import androidx.compose.material3.Scaffold import androidx.compose.material3.Text import androidx.compose.material3.TopAppBar import androidx.compose.runtime.Composable +import androidx.compose.runtime.collectAsState +import androidx.compose.runtime.getValue import androidx.compose.runtime.remember import androidx.compose.ui.Modifier +import androidx.compose.ui.platform.LocalContext import androidx.compose.ui.text.font.FontWeight import androidx.compose.ui.unit.dp /** - * One step of the setup list: a sentence that says what to do, and optionally how. + * One step: a sentence that says what to do, and optionally how. * * [command] is for a step that cannot be done on the phone; it is shown as something to copy * rather than to read, because a mistyped package name fails silently. [onClick] is for a step @@ -43,51 +47,58 @@ private class Step( val onClick: (() -> Unit)? = null, ) -private fun setupSteps(onRequestCallScreening: () -> Unit) = listOf( - Step("Use a car from 2020 or earlier.", "Newer head units reject the certificate."), +private val FIRST_DRIVE = listOf( + Step("Turn off Android Auto.", "Disable or uninstall the Android Auto app, so the car talks to Gearslip instead."), + Step("Load a certificate.", "In Settings, download one or import your own."), Step( - "On GrapheneOS, exempt Gearslip from exploit protections.", - "Open Settings, then Apps, then Gearslip, and turn on Exploit protection compatibility mode.", + "Allow the microphone.", + "Android won't let an app pass music to the car without it. Gearslip listens only when " + + "you tap the mic or press the talk button on your wheel.", ), - Step("Disable or uninstall the Android Auto app."), - Step("Load a certificate.", "Open Settings, then download or import a certificate."), +) + +private fun inTheCar(onRequestCallScreening: () -> Unit) = listOf( Step( - "Allow the microphone, so music can reach the car.", - "Android asks for the microphone before it will let any app pass music along. " + - "Gearslip listens only when you tap Reply or press the voice button. The rest of the " + - "time it copies what a music app is playing, and nothing else.", + "Plug in, or connect wirelessly.", + "Gearslip opens by itself when the car connects. For wireless, pair the phone with the " + + "car first, then tap Connect wirelessly on Home.", ), Step( - "Optional: stop Android asking about audio every drive.", - "Connect the phone to a computer with USB debugging turned on, and run this command. " + - "It tells Android to trust Gearslip with audio from now on. Skip it and everything " + - "still works, but you have to tap Start once each time you set off.", - command = "adb shell appops set ${BuildConfig.APPLICATION_ID} PROJECT_MEDIA allow", + "Pair over Bluetooth for calls.", + "Calls play through the car once your phone and the car are paired. Gearslip offers to " + + "pair the first time you plug in.", ), Step( - "Optional: keep notifications readable while music plays.", - "Android treats sending audio to the car like sharing your screen, and hides what your " + - "notifications say. Run this command too, and Gearslip switches that off only while " + - "music is going to the car, then puts it back.", - command = "adb shell pm grant ${BuildConfig.APPLICATION_ID} android.permission.WRITE_SECURE_SETTINGS", - ), - Step( - "Optional: let the car screen answer and decline calls.", - "This sets Gearslip as your phone's caller ID and spam-blocking app, the same kind of " + - "consent a call-blocking app asks for. Without it, calls still ring, but only the " + - "phone itself can answer or decline them.", + "Answer calls from the car screen.", + "Set Gearslip as your caller ID and spam app. Without it, calls still ring, but only " + + "the phone can answer them.", buttonLabel = "Set up", onClick = onRequestCallScreening, ), - Step("Plug the phone into the car.", "Gearslip opens by itself when the head unit connects."), ) -/** The full setup reference, behind the help button: what the one-time guide walks through, kept - * here too so it can be reread without repeating the guide itself. */ +private val USB_AUDIO = listOf( + Step( + "Stop the audio prompt each drive.", + "Connect the phone to a computer with USB debugging on, and run this once. Skip it and " + + "you tap Start each time you set off.", + command = "adb shell appops set ${BuildConfig.APPLICATION_ID} PROJECT_MEDIA allow", + ), + Step( + "Keep notifications readable.", + "Android hides what notifications say while it shares audio. Run this too, and Gearslip " + + "lifts that only while music goes to the car.", + command = "adb shell pm grant ${BuildConfig.APPLICATION_ID} android.permission.WRITE_SECURE_SETTINGS", + ), +) + +/** Which cars work, and the setup the one-time guide walks through, to reread at any time. */ @OptIn(ExperimentalMaterial3Api::class) @Composable fun HelpScreen(onBack: () -> Unit, onRequestCallScreening: () -> Unit) { - val steps = remember(onRequestCallScreening) { setupSteps(onRequestCallScreening) } + val car = remember(onRequestCallScreening) { inTheCar(onRequestCallScreening) } + val usb by CarSettings.pipeAudio.collectAsState() + val context = LocalContext.current Scaffold( containerColor = MaterialTheme.colorScheme.background, @@ -108,39 +119,59 @@ fun HelpScreen(onBack: () -> Unit, onRequestCallScreening: () -> Unit) { .padding(padding) .padding(horizontal = 24.dp) .verticalScroll(rememberScrollState()), + verticalArrangement = Arrangement.spacedBy(24.dp), ) { - Spacer(Modifier.height(8.dp)) + Spacer(Modifier.height(0.dp)) + SettingsSection("Which cars work") { + Text( + "Most cars work. Starting around 2020, some cars added newer security " + + "standards that reject Gearslip's certificate, and Gearslip can't run in those cars.", + style = MaterialTheme.typography.bodyLarge, + modifier = Modifier.padding(start = 16.dp, end = 16.dp, top = 16.dp), + ) + FilledTonalButton( + onClick = { GearslipLinks.open(context, GearslipLinks.CARS) }, + modifier = Modifier.padding(16.dp), + ) { Text("See the full list") } + } + Steps("Before your first drive", FIRST_DRIVE) + Steps("In the car", car) + if (usb) Steps("USB audio", USB_AUDIO) Spacer(Modifier.height(16.dp)) - Column(verticalArrangement = Arrangement.spacedBy(18.dp)) { - steps.forEachIndexed { index, step -> - Row(horizontalArrangement = Arrangement.spacedBy(16.dp)) { - Text( - "${index + 1}", - style = MaterialTheme.typography.titleMedium, - fontWeight = FontWeight.Bold, - color = MaterialTheme.colorScheme.primary, - modifier = Modifier.width(16.dp), - ) - Column(verticalArrangement = Arrangement.spacedBy(4.dp)) { - Text(step.action, style = MaterialTheme.typography.bodyLarge, fontWeight = FontWeight.SemiBold) - step.detail?.let { - Text( - it, - style = MaterialTheme.typography.bodyMedium, - color = MaterialTheme.colorScheme.onSurfaceVariant, - ) - } - step.command?.let { CopyableCommand(it) } - if (step.onClick != null) { - FilledTonalButton(onClick = step.onClick, modifier = Modifier.padding(top = 4.dp)) { - Text(step.buttonLabel ?: "Open") - } - } + } + } +} + +/** A titled card of numbered steps. */ +@Composable +private fun Steps(title: String, steps: List) { + SettingsSection(title) { + steps.forEachIndexed { index, step -> + if (index > 0) SettingsDivider() + Row( + Modifier.fillMaxWidth().padding(16.dp), + horizontalArrangement = Arrangement.spacedBy(16.dp), + ) { + Text( + "${index + 1}", + style = MaterialTheme.typography.titleMedium, + fontWeight = FontWeight.Bold, + color = MaterialTheme.colorScheme.primary, + modifier = Modifier.width(16.dp), + ) + Column(verticalArrangement = Arrangement.spacedBy(4.dp)) { + Text(step.action, style = MaterialTheme.typography.bodyLarge, fontWeight = FontWeight.SemiBold) + step.detail?.let { + Text(it, style = MaterialTheme.typography.bodyMedium, color = MaterialTheme.colorScheme.onSurfaceVariant) + } + step.command?.let { CopyableCommand(it) } + if (step.onClick != null) { + FilledTonalButton(onClick = step.onClick, modifier = Modifier.padding(top = 4.dp)) { + Text(step.buttonLabel ?: "Open") } } } } - Spacer(Modifier.height(32.dp)) } } } diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/ui/HomeScreen.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/ui/HomeScreen.kt index a13d01f..4abde51 100644 --- a/gearslip/src/main/java/app/seb3thehacker/gearslip/ui/HomeScreen.kt +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/ui/HomeScreen.kt @@ -1,7 +1,12 @@ package app.seb3thehacker.gearslip.ui +import app.seb3thehacker.gearslip.speech.SpeechModels + +import androidx.compose.material3.LinearProgressIndicator + import androidx.compose.foundation.clickable import androidx.compose.foundation.layout.Arrangement +import androidx.compose.foundation.layout.Box import androidx.compose.foundation.layout.Column import androidx.compose.foundation.layout.Row import androidx.compose.foundation.layout.Spacer @@ -11,6 +16,13 @@ import androidx.compose.foundation.layout.height import androidx.compose.foundation.layout.padding import androidx.compose.foundation.layout.size import android.content.Context +import androidx.compose.material3.ButtonDefaults +import androidx.compose.ui.graphics.Color +import androidx.compose.foundation.isSystemInDarkTheme +import app.seb3thehacker.gearslip.car.CarSettings +import app.seb3thehacker.gearslip.car.BluetoothAudioCheck +import android.provider.Settings +import android.content.Intent import androidx.compose.foundation.rememberScrollState import androidx.compose.foundation.verticalScroll import androidx.compose.material.icons.Icons @@ -20,6 +32,7 @@ import androidx.compose.material.icons.filled.Send import androidx.compose.material.icons.filled.List import androidx.compose.material.icons.filled.PlayArrow import androidx.compose.material.icons.filled.Settings +import androidx.compose.material.icons.filled.Star import androidx.compose.material3.Button import androidx.compose.material3.CircularProgressIndicator import androidx.compose.material3.HorizontalDivider @@ -31,6 +44,7 @@ import androidx.compose.material3.Surface import androidx.compose.material3.Text import androidx.compose.runtime.Composable import androidx.compose.runtime.getValue +import androidx.compose.runtime.remember import androidx.compose.ui.Alignment import androidx.compose.ui.Modifier import androidx.compose.ui.graphics.vector.ImageVector @@ -41,117 +55,163 @@ import androidx.lifecycle.compose.collectAsStateWithLifecycle import app.seb3thehacker.gearslip.BuildConfig import app.seb3thehacker.gearslip.SessionStatus import app.seb3thehacker.gearslip.notify.TestMessage +import app.seb3thehacker.gearslip.stats.WorkedPrompt @Composable fun HomeScreen( - onOpenLogs: () -> Unit, onOpenSettings: () -> Unit, - onOpenCarPreview: () -> Unit, + onOpenWhatsNew: () -> Unit, onOpenHelp: () -> Unit, + onOpenUsageNotes: () -> Unit, onDisconnect: () -> Unit, ) { val status by SessionStatus.state.collectAsStateWithLifecycle() + val context = LocalContext.current + val workedCar by remember { WorkedPrompt.pending(context) }.collectAsStateWithLifecycle() + val live = status.phase == SessionStatus.Phase.CONNECTING || + status.phase == SessionStatus.Phase.PROJECTING - Scaffold(containerColor = MaterialTheme.colorScheme.background) { padding -> - Column( - Modifier - .fillMaxSize() - .padding(padding) - .padding(horizontal = 24.dp), - ) { + val celebrate by WorkedPrompt.celebrate.collectAsStateWithLifecycle() + Box { + Scaffold(containerColor = MaterialTheme.colorScheme.background) { padding -> Column( Modifier - .weight(1f) - .verticalScroll(rememberScrollState()), + .fillMaxSize() + .padding(padding) + .padding(horizontal = 24.dp), ) { - Spacer(Modifier.height(24.dp)) - Row( - Modifier.fillMaxWidth(), - verticalAlignment = Alignment.CenterVertically, - horizontalArrangement = Arrangement.SpaceBetween, + Column( + Modifier + .weight(1f) + .verticalScroll(rememberScrollState()), ) { - Text( - "Gearslip", - style = MaterialTheme.typography.displayLarge, - fontWeight = FontWeight.Bold, - color = MaterialTheme.colorScheme.onBackground, - ) - IconButton( - onClick = onOpenSettings, - modifier = Modifier.size(56.dp), + Spacer(Modifier.height(24.dp)) + Row( + Modifier.fillMaxWidth(), + verticalAlignment = Alignment.CenterVertically, + horizontalArrangement = Arrangement.SpaceBetween, ) { - Icon(Icons.Filled.Settings, contentDescription = "Settings", modifier = Modifier.size(40.dp)) + Text( + "Gearslip", + style = MaterialTheme.typography.displayLarge, + fontWeight = FontWeight.Bold, + color = MaterialTheme.colorScheme.onBackground, + ) + IconButton( + onClick = onOpenSettings, + modifier = Modifier.size(56.dp), + ) { + Icon(Icons.Filled.Settings, contentDescription = "Settings", modifier = Modifier.size(40.dp)) + } } - } - Text( - "v${LocalContext.current.appVersionName()}", - style = MaterialTheme.typography.labelLarge, - color = MaterialTheme.colorScheme.onSurfaceVariant, - ) - Spacer(Modifier.height(16.dp)) - StatusCard(status) - Spacer(Modifier.height(16.dp)) - } - - // Sits at the bottom, where a thumb reaches it. - Column(Modifier.padding(vertical = 16.dp)) { - val live = status.phase == SessionStatus.Phase.CONNECTING || - status.phase == SessionStatus.Phase.PROJECTING - if (live) { - Button(onClick = onDisconnect, modifier = Modifier.fillMaxWidth().height(56.dp)) { - Text("Disconnect") + Text( + "v${LocalContext.current.appVersionName()}", + style = MaterialTheme.typography.labelLarge, + color = MaterialTheme.colorScheme.onSurfaceVariant, + ) + Spacer(Modifier.height(16.dp)) + StatusCard(status) + if (live) BluetoothMissingCard() + VoiceDownloadCard() + // The first drive that worked asks once to share it, right under the result. + val car = workedCar + if (car != null && !live) { + Spacer(Modifier.height(16.dp)) + WorkedCard(car, onSeeWhatsSent = onOpenUsageNotes) } Spacer(Modifier.height(16.dp)) + UpdateCard() } - Surface(color = MaterialTheme.colorScheme.surfaceContainerHigh, shape = MaterialTheme.shapes.extraLarge) { - Column { - if (!live) { - ActionRow(Icons.Filled.PlayArrow, "Car preview", onOpenCarPreview) - RowDivider() - } - ActionRow(Icons.Filled.List, "Live logs", onOpenLogs) - RowDivider() - ActionRow(Icons.Filled.Info, "Connection help", onOpenHelp) - if (BuildConfig.DEBUG) { - val context = LocalContext.current - RowDivider() - ActionRow(Icons.Filled.Send, "Send a test message") { TestMessage.post(context) } + // Sits at the bottom, where a thumb reaches it. + Column(Modifier.padding(vertical = 16.dp)) { + SettingsCard { + SettingsRow( + "What's new in ${Intro.release}", + Modifier.clickable(onClick = onOpenWhatsNew), + icon = Icons.Filled.Star, + trailing = { Arrow() }, + ) + SettingsDivider() + SettingsRow( + "Which cars work", + Modifier.clickable(onClick = onOpenHelp), + subtitle = "Most do. See which ones, and what to set up first", + icon = Icons.Filled.Info, + trailing = { Arrow() }, + ) + } + if (live) { + Spacer(Modifier.height(16.dp)) + Button(onClick = onDisconnect, modifier = Modifier.fillMaxWidth().height(56.dp)) { + Text("Disconnect") } } } } } + // Sharing just went on from the "it worked" ask: a little thanks. + if (celebrate) ConfettiFall(onDone = WorkedPrompt::celebrated) + // The usage notes ask comes first; the donation ask waits until that's answered. + if (workedCar == null) DonateDialog(live) + } +} + +/** Bluetooth audio is chosen, but the phone isn't connected to the car. See [BluetoothAudioCheck]. */ +@Composable +private fun BluetoothMissingCard() { + val missing by BluetoothAudioCheck.missing.collectAsStateWithLifecycle() + val usb by CarSettings.pipeAudio.collectAsStateWithLifecycle() + val warn by CarSettings.bluetoothWarning.collectAsStateWithLifecycle() + if (!missing || usb || !warn) return + val context = LocalContext.current + // Yellow, not the theme's error red: music is missing, but nothing is broken. Material has no + // warning colour, so these are fixed amber tones for light and dark. + val dark = isSystemInDarkTheme() + val container = if (dark) Color(0xFF4A3B00) else Color(0xFFFFE08A) + val content = if (dark) Color(0xFFFFE08A) else Color(0xFF3A2E00) + Spacer(Modifier.height(16.dp)) + Surface(color = container, contentColor = content, shape = MaterialTheme.shapes.extraLarge) { + Column(Modifier.fillMaxWidth().padding(20.dp), verticalArrangement = Arrangement.spacedBy(8.dp)) { + Text("Bluetooth isn't connected", style = MaterialTheme.typography.titleLarge, fontWeight = FontWeight.SemiBold) + Text( + "Music plays over Bluetooth, but your phone isn't connected to the car. Connect it " + + "here or from the car's Bluetooth menu.", + style = MaterialTheme.typography.bodyMedium, + ) + Button( + colors = ButtonDefaults.buttonColors(containerColor = content, contentColor = container), + onClick = { + runCatching { + context.startActivity(Intent(Settings.ACTION_BLUETOOTH_SETTINGS).addFlags(Intent.FLAG_ACTIVITY_NEW_TASK)) + } + }, + modifier = Modifier.padding(top = 4.dp), + ) { Text("Bluetooth settings") } + } + } +} + +/** While the voice models download, how far along they are; nothing otherwise. */ +@Composable +private fun VoiceDownloadCard() { + val progress by SpeechModels.progress.collectAsStateWithLifecycle() + val percent = progress ?: return + Spacer(Modifier.height(16.dp)) + SettingsCard { + Column(Modifier.padding(horizontal = 16.dp, vertical = 14.dp), verticalArrangement = Arrangement.spacedBy(10.dp)) { + Text("Downloading voice, $percent%", style = MaterialTheme.typography.bodyLarge) + LinearProgressIndicator(progress = { percent / 100f }, modifier = Modifier.fillMaxWidth()) + } } } @Composable -private fun ActionRow(icon: ImageVector, label: String, onClick: () -> Unit) { - Row( - Modifier - .fillMaxWidth() - .clickable(onClick = onClick) - .padding(horizontal = 20.dp, vertical = 18.dp), - verticalAlignment = Alignment.CenterVertically, - horizontalArrangement = Arrangement.spacedBy(16.dp), - ) { - Icon(icon, contentDescription = null, tint = MaterialTheme.colorScheme.primary) - Text(label, style = MaterialTheme.typography.titleMedium, modifier = Modifier.weight(1f)) - Icon( - Icons.AutoMirrored.Filled.KeyboardArrowRight, - contentDescription = null, - tint = MaterialTheme.colorScheme.onSurfaceVariant, - ) - } -} - -@Composable -private fun RowDivider() { - HorizontalDivider( - modifier = Modifier.padding(horizontal = 20.dp), - color = MaterialTheme.colorScheme.outlineVariant, - ) -} +private fun Arrow() = Icon( + Icons.AutoMirrored.Filled.KeyboardArrowRight, + contentDescription = null, + tint = MaterialTheme.colorScheme.onSurfaceVariant, +) @Composable private fun StatusCard(status: SessionStatus.Snapshot) { @@ -164,11 +224,11 @@ private fun StatusCard(status: SessionStatus.Snapshot) { scheme.surfaceContainerHigh to scheme.onSurface } + val live = status.phase == SessionStatus.Phase.CONNECTING || status.phase == SessionStatus.Phase.PROJECTING Surface(color = container, contentColor = content, shape = MaterialTheme.shapes.extraLarge) { + Column(Modifier.fillMaxWidth().padding(24.dp)) { Row( - Modifier - .fillMaxWidth() - .padding(20.dp), + Modifier.fillMaxWidth(), verticalAlignment = Alignment.CenterVertically, horizontalArrangement = Arrangement.spacedBy(16.dp), ) { @@ -184,11 +244,17 @@ private fun StatusCard(status: SessionStatus.Snapshot) { ) {} } Column { - Text(status.headline, style = MaterialTheme.typography.titleLarge, fontWeight = FontWeight.SemiBold) + Text(status.headline, style = MaterialTheme.typography.headlineSmall, fontWeight = FontWeight.SemiBold) Spacer(Modifier.height(4.dp)) Text(status.detail, style = MaterialTheme.typography.bodyMedium) } } + // The other way in: wireless, right under the cable's instructions. + if (!live) { + Spacer(Modifier.height(20.dp)) + WirelessButton() + } + } } } diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/ui/IntroScreen.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/ui/IntroScreen.kt new file mode 100644 index 0000000..2417baf --- /dev/null +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/ui/IntroScreen.kt @@ -0,0 +1,877 @@ +package app.seb3thehacker.gearslip.ui + +import android.content.Context +import android.content.Intent +import android.net.Uri +import android.provider.Settings +import androidx.activity.compose.BackHandler +import androidx.compose.animation.core.Animatable +import androidx.compose.animation.core.CubicBezierEasing +import androidx.compose.animation.core.FastOutSlowInEasing +import androidx.compose.animation.core.LinearEasing +import androidx.compose.animation.core.RepeatMode +import androidx.compose.animation.core.animateDpAsState +import androidx.compose.animation.core.animateFloat +import androidx.compose.animation.core.infiniteRepeatable +import androidx.compose.animation.core.keyframes +import androidx.compose.animation.core.rememberInfiniteTransition +import androidx.compose.animation.core.tween +import androidx.compose.foundation.Canvas +import androidx.compose.foundation.BorderStroke +import androidx.compose.foundation.background +import androidx.compose.foundation.border +import androidx.compose.foundation.clickable +import androidx.compose.foundation.layout.Arrangement +import androidx.compose.foundation.layout.Box +import androidx.compose.foundation.layout.BoxWithConstraints +import androidx.compose.foundation.layout.Column +import androidx.compose.foundation.layout.Row +import androidx.compose.foundation.layout.Spacer +import androidx.compose.foundation.layout.fillMaxSize +import androidx.compose.foundation.layout.fillMaxWidth +import androidx.compose.foundation.layout.height +import androidx.compose.foundation.layout.offset +import androidx.compose.foundation.layout.padding +import androidx.compose.foundation.layout.size +import androidx.compose.foundation.layout.width +import androidx.compose.foundation.layout.wrapContentWidth +import androidx.compose.foundation.shape.RoundedCornerShape +import androidx.compose.material3.MaterialTheme +import androidx.compose.material3.Text +import androidx.compose.runtime.Composable +import androidx.compose.runtime.LaunchedEffect +import androidx.compose.runtime.getValue +import androidx.compose.runtime.mutableIntStateOf +import androidx.compose.runtime.mutableStateOf +import androidx.compose.runtime.remember +import androidx.compose.runtime.setValue +import androidx.compose.ui.Alignment +import androidx.compose.ui.Modifier +import androidx.compose.ui.draw.clip +import androidx.compose.ui.geometry.CornerRadius +import androidx.compose.ui.geometry.Offset +import androidx.compose.ui.geometry.Size +import androidx.compose.ui.graphics.Color +import androidx.compose.ui.graphics.Path +import androidx.compose.ui.graphics.StrokeCap +import androidx.compose.ui.graphics.StrokeJoin +import androidx.compose.ui.graphics.drawscope.DrawScope +import androidx.compose.ui.graphics.drawscope.Stroke +import androidx.compose.ui.graphics.drawscope.rotate +import androidx.compose.ui.graphics.drawscope.scale +import androidx.compose.ui.graphics.graphicsLayer +import androidx.compose.ui.platform.LocalContext +import androidx.compose.ui.platform.LocalDensity +import androidx.compose.ui.text.TextStyle +import androidx.compose.ui.text.font.FontWeight +import androidx.compose.ui.unit.Dp +import androidx.compose.ui.unit.TextUnit +import androidx.compose.ui.unit.dp +import app.seb3thehacker.gearslip.BuildConfig +import app.seb3thehacker.gearslip.CertProvider +import app.seb3thehacker.gearslip.AppSettings +import androidx.compose.runtime.collectAsState +import androidx.compose.ui.draw.clipToBounds +import androidx.compose.ui.draw.drawWithContent +import androidx.compose.ui.graphics.BlendMode +import androidx.compose.ui.graphics.Brush +import androidx.compose.ui.graphics.CompositingStrategy +import androidx.compose.foundation.layout.wrapContentHeight +import androidx.compose.foundation.layout.PaddingValues +import androidx.compose.material.icons.Icons +import androidx.compose.material.icons.filled.PlayArrow +import androidx.compose.material.icons.filled.Search +import androidx.compose.runtime.CompositionLocalProvider +import app.seb3thehacker.gearslip.car.CarSettings +import app.seb3thehacker.gearslip.car.theme.GsButton +import app.seb3thehacker.gearslip.car.theme.GsIconButton +import app.seb3thehacker.gearslip.car.theme.GsThemes +import app.seb3thehacker.gearslip.car.theme.GsTone +import app.seb3thehacker.gearslip.car.theme.LocalGsTheme +import app.seb3thehacker.gearslip.speech.SpeechModels +import app.seb3thehacker.gearslip.stats.UsageStats +import kotlin.math.PI +import kotlin.math.cos +import kotlin.math.atan2 +import kotlin.math.sqrt +import kotlin.math.sin +import kotlin.random.Random +import kotlinx.coroutines.delay + +/** + * The release intro: a few screens shown once after updating to a new release, before setup. New + * installs skip it. Its look follows the 0.3 mockup: Material shapes in the phone's own colours, + * with every size taken from the screen's width so it scales from small phones to tablets. + */ +object Intro { + private const val PREFS = "intro" + private const val KEY_SEEN = "seen_release" + private const val KEY_STEP = "step" + + /** "0.3" from "0.3.00-alpha": the intro belongs to a release, not to each patch. */ + val release: String = BuildConfig.VERSION_NAME.split('.').take(2).joinToString(".") + + /** + * True once per release for people who used Gearslip before. A new install, or one with its + * data cleared, gets the setup guide instead: there's nothing "new" yet, so this release + * counts as seen and the next one shows. Install dates can't tell those apart, since + * clearing data keeps them, so this goes by what setup left behind; 0.2 saved the same. + */ + fun due(context: Context): Boolean { + if (context.getSharedPreferences(PREFS, Context.MODE_PRIVATE).getString(KEY_SEEN, null) == release) return false + val usedBefore = AppSettings.hasSeenPermissionsSetup(context) || + AppSettings.hasSkippedCertSetup(context) || CertProvider.hasAnyCert(context) + if (!usedBefore) { + markSeen(context) + return false + } + return true + } + + fun markSeen(context: Context) { + context.getSharedPreferences(PREFS, Context.MODE_PRIVATE).edit() + .putString(KEY_SEEN, release) + .remove(KEY_STEP) + .apply() + } + + /** A replay from Settings starts at the first page. */ + fun replay(context: Context) { + context.getSharedPreferences(PREFS, Context.MODE_PRIVATE).edit().remove(KEY_STEP).apply() + } + + /** The page the intro was on, so an app restart part-way through resumes there. */ + internal fun savedStep(context: Context): String? = + context.getSharedPreferences(PREFS, Context.MODE_PRIVATE).getString(KEY_STEP, null) + + internal fun saveStep(context: Context, step: String) { + context.getSharedPreferences(PREFS, Context.MODE_PRIVATE).edit().putString(KEY_STEP, step).apply() + } +} + +/** The intro's tones, taken from the phone's own Material You colours like the rest of the app. */ +internal class IntroColors( + val surface: Color, val onSurface: Color, val onVariant: Color, val outline: Color, + val primary: Color, val onPrimary: Color, val primaryC: Color, val onPrimaryC: Color, + val tertiaryC: Color, val onTertiaryC: Color, val secondaryC: Color, val onSecondaryC: Color, +) + +@Composable +internal fun introColors(): IntroColors { + val m = MaterialTheme.colorScheme + return IntroColors( + surface = m.surface, onSurface = m.onSurface, onVariant = m.onSurfaceVariant, outline = m.outline, + primary = m.primary, onPrimary = m.onPrimary, primaryC = m.primaryContainer, onPrimaryC = m.onPrimaryContainer, + tertiaryC = m.tertiaryContainer, onTertiaryC = m.onTertiaryContainer, + secondaryC = m.secondaryContainer, onSecondaryC = m.onSecondaryContainer, + ) +} + +/** In the order shown: what's new, what to pick, housekeeping, then the people behind it. */ +private enum class IntroStep { WELCOME, WIRELESS, KNOBS, VOICE, THEME, APPS, GRAPHENE, CHAT, THANKS } + +private val Pop = CubicBezierEasing(0.3f, 1.6f, 0.5f, 1f) +private val Rise = CubicBezierEasing(0.2f, 0.8f, 0.2f, 1f) + +@Composable +fun IntroScreen(onDone: () -> Unit) { + val context = LocalContext.current + val c = introColors() + // The GrapheneOS screen is only news to people running it. + val steps = remember { + IntroStep.entries.filter { it != IntroStep.GRAPHENE || UsageStats.osName(context) == "GrapheneOS" } + } + // Turning exploit protection back on restarts the app, so the intro picks up where it was. + var index by remember { + mutableIntStateOf(steps.indexOfFirst { it.name == Intro.savedStep(context) }.coerceAtLeast(0)) + } + val step = steps[index] + LaunchedEffect(step) { Intro.saveStep(context, step.name) } + val finish = { + Intro.markSeen(context) + onDone() + } + BackHandler(enabled = index > 0) { index-- } + + BoxWithConstraints(Modifier.fillMaxSize().background(c.surface)) { + val w = maxWidth + val h = maxHeight + // The mockup's cqw: one percent of the screen's width. + fun cq(n: Float): Dp = w * (n / 100f) + val density = LocalDensity.current + fun cqSp(n: Float): TextUnit = with(density) { cq(n).toSp() } + + // The step on top, Back and Next under it, the same as the setup guide. + Column(Modifier.fillMaxSize()) { + Box(Modifier.weight(1f).fillMaxWidth()) { + when (step) { + IntroStep.WELCOME -> Welcome(c, ::cq, ::cqSp, h) + IntroStep.WIRELESS -> BadgeStep(c, ::cq, ::cqSp, c.secondaryC, "Wireless.", + "If your car does wireless Android Auto, pair once and leave the cable at home.", step) { WifiIcon(c.onSecondaryC) } + IntroStep.KNOBS -> BadgeStep(c, ::cq, ::cqSp, c.tertiaryC, "Knobs work.", + "Turn, push, done. No touchscreen? Gearslip doesn't need one anymore.", step) { DialIcon(c.onTertiaryC) } + IntroStep.VOICE -> VoiceStep(c, ::cq, ::cqSp, step) + IntroStep.GRAPHENE -> BadgeStep(c, ::cq, ::cqSp, c.secondaryC, "Protections back on.", + "On GrapheneOS, Gearslip no longer needs exploit protection compatibility mode. If you " + + "turned it on for Gearslip, turn it off to get full protection back.", step, + action = "Open app settings" to { + runCatching { + context.startActivity( + Intent(Settings.ACTION_APPLICATION_DETAILS_SETTINGS, Uri.fromParts("package", context.packageName, null)), + ) + } + }, + outlined = true, + ) { ShieldIcon(c.onSecondaryC, c.secondaryC) } + IntroStep.THANKS -> BadgeStep(c, ::cq, ::cqSp, c.tertiaryC, "Thank you.", + "Thanks to your usage notes, Gearslip is confirmed to work in cars newer than 2020. " + + "Each note shows which cars work and which still fail, so I know what to fix next.", step, + action = "See which cars work" to { GearslipLinks.open(context, GearslipLinks.CARS) }, + outlined = true, + secondAction = "Support on Liberapay" to { GearslipLinks.open(context, GearslipLinks.LIBERAPAY) }, + ) { HeartIcon(c.onTertiaryC) } + IntroStep.CHAT -> BadgeStep(c, ::cq, ::cqSp, c.primaryC, "Come say hi.", + "Gearslip has a chat room on Matrix. Talk with other drivers about cars, apps, and " + + "where Gearslip is headed.", step, + action = "Join on Matrix" to { GearslipLinks.open(context, GearslipLinks.MATRIX) }, + outlined = true, + ) { ChatIcon(c.onPrimaryC, c.primaryC) } + IntroStep.APPS -> BadgeStep(c, ::cq, ::cqSp, c.secondaryC, "Good company.", + "Free, open-source apps that work well in the car. The first in each row works best. Tap one to see it on GitHub.", step, + extra = { AppPicks(c, ::cq, ::cqSp, step) }) { StarIcon(c.onSecondaryC) } + IntroStep.THEME -> BadgeStep(c, ::cq, ::cqSp, c.tertiaryC, "Your buttons.", + "Pick how the car's buttons look. Change it any time in the car's settings.", step, + extra = { ThemePicker(c, ::cq, ::cqSp, step) }) { ButtonsIcon(c.onTertiaryC) } + } + } + Column(Modifier.fillMaxWidth().padding(horizontal = cq(8f))) { + Dots(c, steps.size, index, ::cq, Modifier.align(Alignment.CenterHorizontally)) + Spacer(Modifier.height(cq(5f))) + PillRow( + c, ::cq, ::cqSp, + onBack = if (index > 0) ({ index-- }) else null, + label = if (index == steps.lastIndex) "Start" else "Next", + ) { if (index < steps.lastIndex) index++ else finish() } + Spacer(Modifier.height(cq(6f))) + } + } + } +} + +/** 0 to 1 once each time [key] changes, after [delayMs]: drives the entrance of one element. */ +@Composable +internal fun entrance(key: Any, delayMs: Int, durationMs: Int = 650): Float { + val a = remember(key) { Animatable(0f) } + LaunchedEffect(key) { + delay(delayMs.toLong()) + a.animateTo(1f, tween(durationMs, easing = LinearEasing)) + } + return a.value +} + +/** Slides up and fades in, like the mockup's `up`. */ +internal fun Modifier.up(p: Float, distance: Dp) = graphicsLayer { + val e = Rise.transform(p) + alpha = e + translationY = (1 - e) * distance.toPx() +} + +/** Grows in with a bounce, like the mockup's `bounce`. */ +internal fun Modifier.pop(p: Float) = graphicsLayer { + val e = Pop.transform(p) + alpha = p.coerceAtMost(1f) + scaleX = 0.5f + 0.5f * e + scaleY = 0.5f + 0.5f * e +} + +@Composable +private fun Welcome(c: IntroColors, cq: (Float) -> Dp, cqSp: (Float) -> TextUnit, height: Dp) { + val key = IntroStep.WELCOME + val spin by rememberInfiniteTransition(label = "spin").animateFloat( + 0f, 360f, infiniteRepeatable(tween(18_000, easing = LinearEasing)), label = "spin", + ) + Box(Modifier.fillMaxSize()) { + Cookie(c.primaryC, Modifier.size(cq(120f)).offset(x = cq(28f), y = -cq(22f)).align(Alignment.TopEnd).graphicsLayer { rotationZ = spin; alpha = 0.9f }) + Ticker(c.secondaryC, c.onSecondaryC, "OPEN SOURCE ✦ NO ACCOUNT ✦ YOUR CAR, YOUR PHONE ✦ ", cq, cqSp, reverse = true, tilt = 3f, + key = key, modifier = Modifier.align(Alignment.TopCenter).padding(top = height * 0.11f)) + + Column(Modifier.align(Alignment.CenterStart).padding(horizontal = cq(8f))) { + Text("Welcome to", color = c.onVariant, style = TextStyle(fontSize = cqSp(4.2f), fontWeight = FontWeight.SemiBold), + modifier = Modifier.up(entrance(key, 100), cq(7f))) + Text( + Intro.release, color = c.primary, + style = TextStyle(fontSize = cqSp(44f), fontWeight = FontWeight.Black, letterSpacing = cqSp(-2.4f), lineHeight = cqSp(38f)), + modifier = Modifier.offset(x = -cq(1.5f)).pop(entrance(key, 250, 900)), + ) + Text( + "alpha · the big one", color = c.onTertiaryC, + style = TextStyle(fontSize = cqSp(4f), fontWeight = FontWeight.Bold), + modifier = Modifier.padding(top = cq(4f)).up(entrance(key, 700), cq(7f)) + .clip(RoundedCornerShape(50)).background(c.tertiaryC).padding(horizontal = cq(4.4f), vertical = cq(2.2f)), + ) + } + + Confetti(c, key, Modifier.fillMaxSize()) + } +} + +/** A tilted band of words scrolling past, slid in from the side when the step opens. */ +@Composable +private fun Ticker( + bg: Color, fg: Color, words: String, cq: (Float) -> Dp, cqSp: (Float) -> TextUnit, + reverse: Boolean, tilt: Float, key: Any, modifier: Modifier, +) { + val enter = entrance(key, 400, 800) + val scroll by rememberInfiniteTransition(label = "ticker").animateFloat( + 0f, 1f, infiniteRepeatable(tween(if (reverse) 20_000 else 16_000, easing = LinearEasing)), label = "ticker", + ) + Box( + modifier + .fillMaxWidth() + .graphicsLayer { + rotationZ = tilt + scaleX = 1.2f + val e = Rise.transform(enter) + alpha = e + translationX = (1 - e) * -cq(30f).toPx() + } + .background(bg) + .padding(vertical = cq(2.6f)), + ) { + Text( + words.repeat(4), + color = fg, maxLines = 1, softWrap = false, + style = TextStyle(fontSize = cqSp(5f), fontWeight = FontWeight.ExtraBold), + modifier = Modifier + .wrapContentWidth(Alignment.Start, unbounded = true) + .graphicsLayer { + // The text holds the words four times; sliding by half loops cleanly. + val shift = size.width / 2f * (if (reverse) 1 - scroll else scroll) + translationX = -shift + }, + ) + } +} + +/** Material shapes thrown once from behind the version number. */ +@Composable +private fun Confetti(c: IntroColors, key: Any, modifier: Modifier) { + val p = entrance(key, 450, 1300) + val bits = remember { + val r = Random(7) + List(22) { i -> + val a = i / 22f * 2 * PI.toFloat() + r.nextFloat() * 0.4f + val d = 30f + r.nextFloat() * 45f + Bit(cos(a) * d, sin(a) * d * 1.2f, r.nextInt(-270, 270).toFloat(), i % 4, i % 3) + } + } + if (p <= 0f || p >= 1f) return + val tones = listOf(c.primary, c.primaryC, c.tertiaryC, c.secondaryC) + Canvas(modifier) { + val unit = size.width / 100f + val e = CubicBezierEasing(0.15f, 0.7f, 0.3f, 1f).transform(p) + val alpha = if (p < 0.7f) 1f else 1f - (p - 0.7f) / 0.3f + val origin = Offset(size.width * 0.3f, size.height * 0.42f) + bits.forEach { b -> + val at = origin + Offset(b.dx * unit * e, b.dy * unit * e) + val s = 3.6f * unit * (0.3f + 0.7f * e) + rotate(b.spin * e, at) { + val color = tones[b.tone].copy(alpha = alpha) + when (b.shape) { + 0 -> drawCircle(color, s / 2, at) + 1 -> drawRoundRect(color, at - Offset(s / 2, s / 2), Size(s, s), androidx.compose.ui.geometry.CornerRadius(unit)) + else -> drawRoundRect(color, at - Offset(s / 2, s / 4), Size(s, s / 2), androidx.compose.ui.geometry.CornerRadius(s / 4)) + } + } + } + } +} + +private class Bit(val dx: Float, val dy: Float, val spin: Float, val tone: Int, val shape: Int) + +/** One idea per screen: a spinning cookie with an icon, a headline, a line of text, maybe a button. */ +@Composable +private fun BadgeStep( + c: IntroColors, cq: (Float) -> Dp, cqSp: (Float) -> TextUnit, badge: Color, + title: String, body: String, key: Any, + action: Pair Unit>? = null, outlined: Boolean = false, + secondAction: Pair Unit>? = null, + extra: (@Composable () -> Unit)? = null, + icon: @Composable () -> Unit, +) { + val spin by rememberInfiniteTransition(label = "spin").animateFloat( + 0f, 360f, infiniteRepeatable(tween(18_000, easing = LinearEasing)), label = "spin", + ) + Column( + Modifier.fillMaxSize().padding(horizontal = cq(8f)), + verticalArrangement = Arrangement.Center, + ) { + Box(Modifier.size(cq(34f)).pop(entrance(key, 50, 800)), contentAlignment = Alignment.Center) { + Cookie(badge, Modifier.fillMaxSize().graphicsLayer { rotationZ = spin }) + Box(Modifier.size(cq(15f))) { icon() } + } + Spacer(Modifier.height(cq(8f))) + Text( + title, color = c.onSurface, + style = TextStyle(fontSize = cqSp(13.5f), fontWeight = FontWeight.Black, letterSpacing = cqSp(-0.6f), lineHeight = cqSp(12.5f)), + modifier = Modifier.up(entrance(key, 200, 700), cq(7f)), + ) + Spacer(Modifier.height(cq(4f))) + Text( + body, color = c.onVariant, + style = TextStyle(fontSize = cqSp(4.5f), lineHeight = cqSp(6.5f)), + modifier = Modifier.up(entrance(key, 350, 700), cq(7f)), + ) + extra?.invoke() + listOfNotNull(action, secondAction).forEachIndexed { i, (label, onClick) -> + Spacer(Modifier.height(if (i > 0) cq(3f) else if (outlined) cq(7f) else cq(9f))) + Text( + label, + color = if (outlined) c.primary else c.onPrimary, + style = TextStyle(fontSize = cqSp(if (outlined) 4.4f else 4.6f), fontWeight = FontWeight.Bold), + modifier = Modifier + .up(entrance(key, 500 + i * 150, 700), cq(7f)) + .clip(RoundedCornerShape(50)) + .then( + if (outlined) Modifier.border(BorderStroke(2.dp, c.outline), RoundedCornerShape(50)) + else Modifier.background(c.primary), + ) + .clickable(onClick = onClick) + .padding(horizontal = if (outlined) cq(7f) else cq(9f), vertical = if (outlined) cq(3.6f) else cq(4.2f)), + ) + } + } +} + +/** + * Voice: a few things to say, drifting past, and a button that fetches both speech models so the + * assistant works the first time someone tries it in the car. + */ +@Composable +private fun VoiceStep(c: IntroColors, cq: (Float) -> Dp, cqSp: (Float) -> TextUnit, key: IntroStep) { + val context = LocalContext.current + remember { SpeechModels.init(context) } + val needed = SpeechModels.RECOMMENDED + val states = needed.map { SpeechModels.state(it).collectAsState().value } + val ready = states.all { it == SpeechModels.State.Ready } + val label = when { + states.any { it is SpeechModels.State.Downloading } -> + "Downloading, ${needed.zip(states).sumOf { (m, st) -> m.downloadMb * percentOf(st) } / needed.sumOf { it.downloadMb }}%" + states.any { it is SpeechModels.State.Failed } -> "Download failed. Try again" + else -> "Download voice, ${needed.sumOf { it.downloadMb }} MB" + } + BadgeStep( + c, cq, cqSp, c.primaryC, "Just say it.", + "Tap the mic, or press the talk button on your wheel, and speak. Your phone turns speech " + + "into text by itself, and nothing you say leaves it. Say \"help\" in the car for the full list.", + key, + action = if (ready) null else label to { SpeechModels.downloadRecommended(context) }, + outlined = true, + extra = { CommandDrift(c, cq, cqSp, key) }, + ) { MicIcon(c.onPrimaryC) } +} + +private fun percentOf(state: SpeechModels.State) = when (state) { + SpeechModels.State.Ready -> 100 + is SpeechModels.State.Downloading -> state.percent + else -> 0 +} + +private val SAMPLE_COMMANDS = listOf( + "Play Daft Punk", "What's playing?", "Next song", + "Call Mom", "Open Organic Maps", "Pause", "Help", +) + +/** Example commands rising slowly through a window that fades at both edges. */ +@Composable +private fun CommandDrift(c: IntroColors, cq: (Float) -> Dp, cqSp: (Float) -> TextUnit, key: Any) { + val scroll by rememberInfiniteTransition(label = "drift").animateFloat( + 0f, 1f, infiniteRepeatable(tween(SAMPLE_COMMANDS.size * 2_200, easing = LinearEasing)), label = "drift", + ) + Box( + Modifier + .padding(top = cq(6f)) + .fillMaxWidth() + .height(cq(34f)) + .up(entrance(key, 500, 700), cq(7f)) + .graphicsLayer { compositingStrategy = CompositingStrategy.Offscreen } + .drawWithContent { + drawContent() + drawRect( + Brush.verticalGradient(0f to Color.Transparent, 0.25f to Color.Black, 0.75f to Color.Black, 1f to Color.Transparent), + blendMode = BlendMode.DstIn, + ) + } + .clipToBounds(), + ) { + Column( + Modifier + .wrapContentHeight(Alignment.Top, unbounded = true) + // The list is there twice; rising by half loops cleanly. + .graphicsLayer { translationY = -size.height / 2f * scroll }, + verticalArrangement = Arrangement.spacedBy(cq(2.4f)), + ) { + repeat(2) { + SAMPLE_COMMANDS.forEach { say -> + Text( + "\u201c$say\u201d", color = c.onSecondaryC, + style = TextStyle(fontSize = cqSp(4.4f), fontWeight = FontWeight.SemiBold), + modifier = Modifier + .clip(RoundedCornerShape(50)) + .background(c.secondaryC) + .padding(horizontal = cq(4.4f), vertical = cq(2f)), + ) + } + } + } + } +} + +/** Each kind of app with its picks, the first recommended most. */ +private val APP_PICKS = listOf( + "Maps" to listOf( + "Organic Maps" to "https://github.com/organicmaps/organicmaps", + "Vela" to "https://github.com/PimpinPumpkin/Vela", + ), + "Music" to listOf( + "ViVi Music" to "https://github.com/vivizzz007/vivi-music", + "Metrolist" to "https://github.com/MetrolistGroup/Metrolist", + ), + "Podcasts" to listOf( + "AntennaPod" to "https://github.com/AntennaPod/AntennaPod", + ), +) + +/** One row per kind, with a chip per app opening its GitHub page; the rows arrive one after another. */ +@Composable +private fun AppPicks(c: IntroColors, cq: (Float) -> Dp, cqSp: (Float) -> TextUnit, key: IntroStep) { + val context = LocalContext.current + Column(Modifier.padding(top = cq(6f)), verticalArrangement = Arrangement.spacedBy(cq(2.6f))) { + APP_PICKS.forEachIndexed { i, (kind, apps) -> + Column( + Modifier + .fillMaxWidth() + .up(entrance(key, 500 + i * 120, 700), cq(7f)) + .clip(RoundedCornerShape(cq(5f))) + .background(c.secondaryC) + .padding(horizontal = cq(4f), vertical = cq(3f)), + ) { + Text(kind.uppercase(), color = c.onSecondaryC.copy(alpha = 0.7f), + style = TextStyle(fontSize = cqSp(3.2f), fontWeight = FontWeight.Bold, letterSpacing = cqSp(0.3f)), + modifier = Modifier.padding(start = cq(1f), bottom = cq(1.6f))) + Row(horizontalArrangement = Arrangement.spacedBy(cq(2f))) { + apps.forEach { (name, url) -> + Text( + "$name ↗", color = c.onSecondaryC, + style = TextStyle(fontSize = cqSp(4.4f), fontWeight = FontWeight.ExtraBold), + maxLines = 1, + modifier = Modifier + .clip(RoundedCornerShape(50)) + .border(BorderStroke(1.5.dp, c.onSecondaryC.copy(alpha = 0.35f)), RoundedCornerShape(50)) + .clickable { + runCatching { + context.startActivity(Intent(Intent.ACTION_VIEW, Uri.parse(url)).addFlags(Intent.FLAG_ACTIVITY_NEW_TASK)) + } + } + .padding(horizontal = cq(3.6f), vertical = cq(2f)), + ) + } + } + } + } + } +} + +/** + * The car's button styles side by side, each drawn in its own style, and a row of real car + * controls below that redraws in whichever is picked. Picking saves it, so the car shows it too. + */ +@Composable +private fun ThemePicker(c: IntroColors, cq: (Float) -> Dp, cqSp: (Float) -> TextUnit, key: IntroStep) { + val chosenId by CarSettings.uiTheme.collectAsState() + val chosen = GsThemes.byId(chosenId) + // Flat first: it's the default. + val themes = listOf(GsThemes.Flat) + GsThemes.builtIn.filter { it != GsThemes.Flat } + Column( + Modifier.padding(top = cq(7f)).fillMaxWidth().up(entrance(key, 500, 700), cq(7f)), + verticalArrangement = Arrangement.spacedBy(cq(6f)), + ) { + Row(Modifier.fillMaxWidth(), horizontalArrangement = Arrangement.spacedBy(cq(3f))) { + themes.forEach { theme -> + CompositionLocalProvider(LocalGsTheme provides theme) { + GsButton( + onClick = { CarSettings.setUiTheme(theme.id) }, + modifier = Modifier.weight(1f), + tone = if (theme == chosen) GsTone.Primary else GsTone.Neutral, + contentPadding = PaddingValues(horizontal = cq(1f), vertical = cq(3f)), + ) { + Text( + if (theme == chosen) "✓ ${theme.name}" else theme.name, + style = TextStyle(fontSize = cqSp(4.2f), fontWeight = FontWeight.Bold), + maxLines = 1, + ) + } + } + } + } + // What the car shows: the player's controls and a full-width action. + CompositionLocalProvider(LocalGsTheme provides chosen) { + Row( + Modifier.fillMaxWidth(), + horizontalArrangement = Arrangement.spacedBy(cq(3f)), + verticalAlignment = Alignment.CenterVertically, + ) { + GsIconButton(Icons.Filled.Search, "Search", onClick = {}, size = cq(15f), iconSize = cq(7f)) + GsIconButton(Icons.Filled.PlayArrow, "Play", onClick = {}, tone = GsTone.Primary, size = cq(15f), iconSize = cq(8f)) + GsButton( + onClick = {}, + modifier = Modifier.weight(1f).height(cq(15f)), + tone = GsTone.Tonal, + ) { Text("Navigate", style = TextStyle(fontSize = cqSp(4.6f), fontWeight = FontWeight.Bold)) } + } + } + } +} + +/** Three keys of a keypad, pressing one after another. */ +@Composable +private fun ButtonsIcon(color: Color) { + val t by rememberInfiniteTransition(label = "keys").animateFloat( + 0f, 3f, infiniteRepeatable(tween(2400, easing = LinearEasing)), label = "keys", + ) + Canvas(Modifier.fillMaxSize()) { + inIconBox { + listOf(6f, 19f, 32f).forEachIndexed { i, x -> + // Each key dips for a moment while its turn comes round. + val dip = (1f - kotlin.math.abs(t - i - 0.5f) * 2f).coerceIn(0f, 1f) * 4f + drawRoundRect(color.copy(alpha = 0.45f), Offset(x, 16f + 4f), Size(10f, 16f), + androidx.compose.ui.geometry.CornerRadius(3f)) + drawRoundRect(color, Offset(x, 16f + dip), Size(10f, 16f), androidx.compose.ui.geometry.CornerRadius(3f)) + } + } + } +} + +/** Material's scalloped "cookie": a circle whose radius waves around its edge. */ +@Composable +internal fun Cookie(color: Color, modifier: Modifier) { + Canvas(modifier) { + val unit = size.minDimension / 100f + val path = Path() + var deg = 0 + while (deg <= 360) { + val a = deg * PI / 180 + val r = (44 + 4.5 * cos(9 * a)) * unit + val x = center.x + (r * cos(a)).toFloat() + val y = center.y + (r * sin(a)).toFloat() + if (deg == 0) path.moveTo(x, y) else path.lineTo(x, y) + deg += 3 + } + path.close() + drawPath(path, color) + } +} + +/** Draws in the mockup's 48-unit icon box. */ +private inline fun DrawScope.inIconBox(block: DrawScope.() -> Unit) { + val s = size.minDimension / 48f + scale(s, s, Offset.Zero) { block() } +} + +@Composable +internal fun WifiIcon(color: Color) { + val t = rememberInfiniteTransition(label = "wifi") + val waves = (0..2).map { i -> + t.animateFloat( + 0.35f, 1f, + infiniteRepeatable(tween(800, delayMillis = 0, easing = FastOutSlowInEasing), RepeatMode.Reverse, + initialStartOffset = androidx.compose.animation.core.StartOffset(i * 200)), + label = "wave$i", + ) + } + Canvas(Modifier.fillMaxSize()) { + inIconBox { + val stroke = Stroke(5f, cap = StrokeCap.Round) + // The mockup's three arcs, inner to outer: radius r through (24 ± half, y). + listOf(Triple(8f, 6f, 34f), Triple(18f, 13f, 26f), Triple(28f, 20f, 18f)).forEachIndexed { i, (r, half, y) -> + val cy = y + sqrt(r * r - half * half) + val start = Math.toDegrees(atan2((y - cy).toDouble(), (-half).toDouble())).toFloat() + val end = Math.toDegrees(atan2((y - cy).toDouble(), half.toDouble())).toFloat() + drawArc(color.copy(alpha = waves[i].value), start, (end - start + 360f) % 360f, false, + Offset(24f - r, cy - r), Size(2 * r, 2 * r), style = stroke) + } + drawCircle(color, 3.6f, Offset(24f, 41f)) + } + } +} + +@Composable +private fun DialIcon(color: Color) { + val angle by rememberInfiniteTransition(label = "dial").animateFloat( + -70f, -70f, + infiniteRepeatable(keyframes { + durationMillis = 2600 + -70f at 0 using CubicBezierEasing(0.5f, 0f, 0.3f, 1f) + 70f at 1040 + 70f at 1430 using CubicBezierEasing(0.5f, 0f, 0.3f, 1f) + -70f at 2470 + }), + label = "dial", + ) + Canvas(Modifier.fillMaxSize()) { + inIconBox { + drawCircle(color, 19f, Offset(24f, 24f), style = Stroke(5f)) + rotate(angle, Offset(24f, 24f)) { + drawRoundRect(color, Offset(21.5f, 7f), Size(5f, 13f), androidx.compose.ui.geometry.CornerRadius(2.5f)) + } + } + } +} + +@Composable +internal fun MicIcon(color: Color) { + val t = rememberInfiniteTransition(label = "mic") + val waves = (0..1).map { i -> + t.animateFloat( + 0.25f, 1f, + infiniteRepeatable(tween(700, easing = FastOutSlowInEasing), RepeatMode.Reverse, + initialStartOffset = androidx.compose.animation.core.StartOffset(i * 250)), + label = "mic$i", + ) + } + Canvas(Modifier.fillMaxSize()) { + inIconBox { + drawRoundRect(color, Offset(18f, 5f), Size(12f, 22f), androidx.compose.ui.geometry.CornerRadius(6f)) + val stroke = Stroke(4f, cap = StrokeCap.Round) + drawArc(color, 0f, 180f, false, Offset(12f, 14f), Size(24f, 22f), style = stroke) + drawLine(color, Offset(24f, 36f), Offset(24f, 42f), 4f, StrokeCap.Round) + // Sound waves either side, pulsing as if someone's talking. + listOf(1f to 0, -1f to 0, 1f to 1, -1f to 1).forEach { (side, i) -> + val r = if (i == 0) 15f else 21f + drawArc(color.copy(alpha = waves[i].value), if (side > 0) -35f else 145f, 70f, false, + Offset(24f - r, 16f - r), Size(2 * r, 2 * r), style = stroke) + } + } + } +} + +@Composable +internal fun StarIcon(color: Color) { + val turn by rememberInfiniteTransition(label = "star").animateFloat( + -8f, 8f, infiniteRepeatable(tween(1600, easing = FastOutSlowInEasing), RepeatMode.Reverse), label = "star", + ) + Canvas(Modifier.fillMaxSize()) { + inIconBox { + val star = Path() + for (i in 0 until 10) { + val r = if (i % 2 == 0) 20f else 8.5f + val a = (-90 + i * 36) * PI / 180 + val x = 24f + (r * cos(a)).toFloat() + val y = 25f + (r * sin(a)).toFloat() + if (i == 0) star.moveTo(x, y) else star.lineTo(x, y) + } + star.close() + rotate(turn, Offset(24f, 25f)) { drawPath(star, color) } + } + } +} + +@Composable +internal fun ShieldIcon(color: Color, tick: Color) { + val p = entrance(IntroStep.GRAPHENE, 800, 500) + Canvas(Modifier.fillMaxSize()) { + inIconBox { + val shield = Path().apply { + moveTo(24f, 4f); lineTo(40f, 10f); lineTo(40f, 22f) + cubicTo(40f, 32f, 33f, 40f, 24f, 44f) + cubicTo(15f, 40f, 8f, 32f, 8f, 22f) + lineTo(8f, 10f); close() + } + drawPath(shield, color) + drawPartialStroke(listOf(Offset(16f, 24f), Offset(22f, 30f), Offset(33f, 18f)), p, tick, 4.5f) + } + } +} + +/** A speech bubble with three dots that pulse, like someone typing. */ +@Composable +private fun ChatIcon(color: Color, dots: Color) { + val t = rememberInfiniteTransition(label = "typing") + val pulse = (0..2).map { i -> + t.animateFloat( + 0.35f, 1f, + infiniteRepeatable(tween(500, easing = FastOutSlowInEasing), RepeatMode.Reverse, + initialStartOffset = androidx.compose.animation.core.StartOffset(i * 170)), + label = "dot$i", + ) + } + Canvas(Modifier.fillMaxSize()) { + inIconBox { + drawRoundRect(color, Offset(4f, 7f), Size(40f, 28f), CornerRadius(10f, 10f)) + drawPath(Path().apply { moveTo(12f, 33f); lineTo(10f, 43f); lineTo(22f, 33f); close() }, color) + pulse.forEachIndexed { i, a -> drawCircle(dots.copy(alpha = a.value), 3.2f, Offset(15f + i * 9f, 21f)) } + } + } +} + +@Composable +internal fun HeartIcon(color: Color) { + val beat by rememberInfiniteTransition(label = "beat").animateFloat( + 1f, 1f, + infiniteRepeatable(keyframes { + durationMillis = 1400 + 1f at 0; 1.16f at 210; 1f at 420; 1.1f at 630; 1f at 840 + }), + label = "beat", + ) + Canvas(Modifier.fillMaxSize()) { + inIconBox { + scale(beat, beat, Offset(24f, 24f)) { + val heart = Path().apply { + moveTo(24f, 41f) + cubicTo(10f, 31f, 5f, 23f, 5f, 16f) + cubicTo(5f, 10f, 10f, 6f, 15f, 6f) + cubicTo(19f, 6f, 22f, 8f, 24f, 12f) + cubicTo(26f, 8f, 29f, 6f, 33f, 6f) + cubicTo(38f, 6f, 43f, 10f, 43f, 16f) + cubicTo(43f, 23f, 38f, 31f, 24f, 41f) + close() + } + drawPath(heart, color) + } + } + } +} + +/** A polyline drawn up to [progress] of its length, so a tick can sweep in. */ +private fun DrawScope.drawPartialStroke(points: List, progress: Float, color: Color, width: Float) { + if (progress <= 0f) return + val lengths = points.zipWithNext { a, b -> (b - a).getDistance() } + var left = lengths.sum() * FastOutSlowInEasing.transform(progress.coerceAtMost(1f)) + val path = Path().apply { moveTo(points[0].x, points[0].y) } + for (i in lengths.indices) { + if (left <= 0f) break + val f = (left / lengths[i]).coerceAtMost(1f) + val to = points[i] + (points[i + 1] - points[i]) * f + path.lineTo(to.x, to.y) + left -= lengths[i] + } + drawPath(path, color, style = Stroke(width, cap = StrokeCap.Round, join = StrokeJoin.Round)) +} + +@Composable +internal fun Dots(c: IntroColors, count: Int, current: Int, cq: (Float) -> Dp, modifier: Modifier) { + Row(modifier, horizontalArrangement = Arrangement.spacedBy(cq(2f))) { + repeat(count) { i -> + val on = i == current + val width by animateDpAsState(if (on) cq(7f) else cq(2f), tween(300), label = "dot") + Box( + Modifier + .size(width, cq(2f)) + .clip(RoundedCornerShape(50)) + .background(if (on) c.primary else c.outline.copy(alpha = 0.45f)), + ) + } + } +} + diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/ui/RecommendedApps.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/ui/RecommendedApps.kt index 2a84106..2e87798 100644 --- a/gearslip/src/main/java/app/seb3thehacker/gearslip/ui/RecommendedApps.kt +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/ui/RecommendedApps.kt @@ -47,13 +47,16 @@ private class RecommendedApp(val name: String, val packageName: String, val url: private val recommended = listOf( "Maps" to listOf( - RecommendedApp("Vela Maps", "app.vela", "https://github.com/PimpinPumpkin/Vela", "Turn-by-turn navigation without Google."), RecommendedApp("Organic Maps", "app.organicmaps", "https://github.com/organicmaps/organicmaps", "Offline maps and routing."), + RecommendedApp("Vela Maps", "app.vela", "https://github.com/PimpinPumpkin/Vela", "Turn-by-turn navigation without Google."), ), "Music" to listOf( RecommendedApp("ViVi Music", "com.vivi.vivimusic", "https://github.com/vivizzz007/vivi-music", "A YouTube Music player."), RecommendedApp("Metrolist", "com.metrolist.music", "https://github.com/MetrolistGroup/Metrolist", "A YouTube Music player."), ), + "Podcasts" to listOf( + RecommendedApp("AntennaPod", "de.danoeh.antennapod", "https://github.com/AntennaPod/AntennaPod", "Podcasts, chapters and all."), + ), ) /** Opened from Settings: the same list the setup guide ends with. */ diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/ui/SettingsScreen.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/ui/SettingsScreen.kt index f6bba10..bee694f 100644 --- a/gearslip/src/main/java/app/seb3thehacker/gearslip/ui/SettingsScreen.kt +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/ui/SettingsScreen.kt @@ -12,13 +12,18 @@ import androidx.compose.foundation.layout.fillMaxSize import androidx.compose.foundation.layout.height import androidx.compose.foundation.layout.padding import androidx.compose.foundation.rememberScrollState +import androidx.compose.foundation.selection.selectable import androidx.compose.foundation.selection.toggleable import androidx.compose.foundation.text.KeyboardOptions import androidx.compose.foundation.verticalScroll import androidx.compose.material.icons.Icons import androidx.compose.material.icons.automirrored.filled.ArrowBack import androidx.compose.material.icons.automirrored.filled.KeyboardArrowRight +import androidx.compose.material.icons.automirrored.filled.List +import androidx.compose.material.icons.automirrored.filled.Send import androidx.compose.material.icons.filled.Add +import androidx.compose.material.icons.filled.Info +import androidx.compose.material.icons.filled.PlayArrow import androidx.compose.material.icons.filled.Delete import androidx.compose.material.icons.filled.KeyboardArrowDown import androidx.compose.material.icons.filled.KeyboardArrowUp @@ -33,12 +38,14 @@ import androidx.compose.material3.Icon import androidx.compose.material3.IconButton import androidx.compose.material3.MaterialTheme import androidx.compose.material3.OutlinedTextField +import androidx.compose.material3.RadioButton import androidx.compose.material3.Scaffold import androidx.compose.material3.Switch import androidx.compose.material3.Text import androidx.compose.material3.TextButton import androidx.compose.material3.TopAppBar import androidx.compose.runtime.Composable +import androidx.compose.runtime.collectAsState import androidx.compose.runtime.getValue import androidx.compose.runtime.mutableIntStateOf import androidx.compose.runtime.mutableStateOf @@ -47,6 +54,8 @@ import androidx.compose.runtime.remember import androidx.compose.runtime.rememberCoroutineScope import androidx.compose.runtime.setValue import androidx.compose.ui.Modifier +import androidx.compose.ui.layout.onGloballyPositioned +import androidx.compose.ui.layout.positionInParent import androidx.compose.ui.platform.LocalContext import androidx.compose.ui.semantics.Role import androidx.compose.ui.text.input.KeyboardType @@ -55,8 +64,14 @@ import androidx.compose.ui.unit.dp import app.seb3thehacker.gearslip.AppSettings import app.seb3thehacker.gearslip.BuildConfig import app.seb3thehacker.gearslip.CertProvider +import app.seb3thehacker.gearslip.SessionStatus +import app.seb3thehacker.gearslip.notify.TestMessage +import app.seb3thehacker.gearslip.car.CarSettings import kotlinx.coroutines.Dispatchers import kotlinx.coroutines.launch +import androidx.compose.runtime.LaunchedEffect +import androidx.compose.runtime.snapshotFlow +import kotlinx.coroutines.flow.first import kotlinx.coroutines.withContext @OptIn(ExperimentalMaterial3Api::class) @@ -65,8 +80,23 @@ fun SettingsScreen( onBack: () -> Unit, onOpenRecommendedApps: () -> Unit, onOpenWhatsNew: () -> Unit, + onOpenHelp: () -> Unit, + onOpenCarPreview: () -> Unit, + onOpenLogs: () -> Unit, onReplayTutorial: () -> Unit, + onReplayIntro: () -> Unit, + atUsageNotes: Boolean = false, ) { + val scroll = rememberScrollState() + // Opened from Home's "it worked" card: start at usage notes. + var usageNotesY by remember { mutableIntStateOf(-1) } + if (atUsageNotes) { + LaunchedEffect(Unit) { + val y = snapshotFlow { usageNotesY }.first { it >= 0 } + snapshotFlow { scroll.maxValue }.first { it > 0 && it < Int.MAX_VALUE } + scroll.scrollTo(y) + } + } Scaffold( containerColor = MaterialTheme.colorScheme.background, topBar = { @@ -85,15 +115,27 @@ fun SettingsScreen( .fillMaxSize() .padding(padding) .padding(horizontal = 24.dp) - .verticalScroll(rememberScrollState()), + .verticalScroll(scroll), verticalArrangement = Arrangement.spacedBy(24.dp), ) { Spacer(Modifier.height(0.dp)) - AboutSection(onOpenWhatsNew, onReplayTutorial) + SettingsSection( + "Support Gearslip", + footer = "Gearslip is free, with no ads and no tracking. Your support keeps it going.", + ) { DonateRows() } ConnectionSection() + HelpSection(onOpenHelp, onOpenCarPreview, onOpenLogs) + CarAudioSection() + VoiceModelSection() if (SHOW_RECOMMENDED_APPS) AppsSection(onOpenRecommendedApps) + SettingsSection( + "Usage notes", + Modifier.onGloballyPositioned { usageNotesY = it.positionInParent().y.toInt() }, + header = USAGE_STATS_INTRO, + ) { UsageStatsRows() } + SettingsSection("Updates") { UpdateRow() } CertificateSection() - SettingsSection("Usage notes", header = USAGE_STATS_INTRO) { UsageStatsRows() } + AboutSection(onOpenWhatsNew, onReplayTutorial, onReplayIntro) Spacer(Modifier.height(16.dp)) } } @@ -106,10 +148,10 @@ private fun Arrow() = Icon( tint = MaterialTheme.colorScheme.onSurfaceVariant, ) -/** What's new and the setup guide, one tap from the top. */ +/** What's new, the replays, and where Gearslip lives outside the app. */ @Composable -private fun AboutSection(onOpenWhatsNew: () -> Unit, onReplayTutorial: () -> Unit) { - SettingsSection { +private fun AboutSection(onOpenWhatsNew: () -> Unit, onReplayTutorial: () -> Unit, onReplayIntro: () -> Unit) { + SettingsSection("About") { SettingsRow( "What's new", Modifier.clickable(onClick = onOpenWhatsNew), @@ -118,6 +160,14 @@ private fun AboutSection(onOpenWhatsNew: () -> Unit, onReplayTutorial: () -> Uni trailing = { Arrow() }, ) SettingsDivider() + SettingsRow( + "Replay the ${Intro.release} intro", + Modifier.clickable(onClick = onReplayIntro), + subtitle = "What's in this release, in a few taps", + icon = Icons.Filled.Star, + trailing = { Arrow() }, + ) + SettingsDivider() SettingsRow( "Replay the setup guide", Modifier.clickable(onClick = onReplayTutorial), @@ -125,6 +175,8 @@ private fun AboutSection(onOpenWhatsNew: () -> Unit, onReplayTutorial: () -> Uni icon = Icons.Filled.Refresh, trailing = { Arrow() }, ) + SettingsDivider() + AboutLinkRows() } } @@ -147,6 +199,75 @@ private fun ConnectionSection() { } } +/** Connection help, and the tools for checking what Gearslip does. */ +@Composable +private fun HelpSection(onOpenHelp: () -> Unit, onOpenCarPreview: () -> Unit, onOpenLogs: () -> Unit) { + val context = LocalContext.current + val status by SessionStatus.state.collectAsState() + val live = status.phase == SessionStatus.Phase.CONNECTING || status.phase == SessionStatus.Phase.PROJECTING + SettingsSection("Help and tools") { + SettingsRow( + "Connection help", + Modifier.clickable(onClick = onOpenHelp), + subtitle = "Which cars work, and what to set up first", + icon = Icons.Filled.Info, + trailing = { Arrow() }, + ) + // The preview draws the car screen on the phone, so it waits until no car is using it. + if (!live) { + SettingsDivider() + SettingsRow( + "Car preview", + Modifier.clickable(onClick = onOpenCarPreview), + subtitle = "See the car screen on your phone", + icon = Icons.Filled.PlayArrow, + trailing = { Arrow() }, + ) + } + SettingsDivider() + SettingsRow( + "Live logs", + Modifier.clickable(onClick = onOpenLogs), + subtitle = "What Gearslip and the car are saying to each other", + icon = Icons.AutoMirrored.Filled.List, + trailing = { Arrow() }, + ) + if (BuildConfig.DEBUG) { + SettingsDivider() + SettingsRow( + "Send a test message", + Modifier.clickable { TestMessage.post(context) }, + subtitle = "Debug builds only. Posts a message notification to read on the car", + icon = Icons.AutoMirrored.Filled.Send, + ) + } + } +} + +/** How music reaches the car: by Bluetooth as usual, or through Gearslip over USB. Same choice as setup. */ +@Composable +private fun CarAudioSection() { + val usb by CarSettings.pipeAudio.collectAsState() + SettingsSection( + "Car audio", + footer = if (usb) "Run the command in Help once, and USB stops asking." else "Calls go over Bluetooth either way.", + ) { + SettingsRow( + "Bluetooth (Recommended)", + Modifier.selectable(!usb, role = Role.RadioButton) { CarSettings.setPipeAudio(false) }, + subtitle = "No prompt each drive, and your map app's voice reaches the car", + trailing = { RadioButton(selected = !usb, onClick = null) }, + ) + SettingsDivider() + SettingsRow( + "USB", + Modifier.selectable(usb, role = Role.RadioButton) { CarSettings.setPipeAudio(true) }, + subtitle = "Clearer sound with no delay. Android asks to share audio once each drive", + trailing = { RadioButton(selected = usb, onClick = null) }, + ) + } +} + @Composable private fun AppsSection(onOpenRecommendedApps: () -> Unit) { SettingsSection("Apps") { @@ -165,7 +286,8 @@ private fun CertificateSection() { val scope = rememberCoroutineScope() var refresh by remember { mutableIntStateOf(0) } - val summary by produceState(null, refresh) { + // The cached summary shows at once; the read still runs, in case the cert changed since. + val summary by produceState(CertSummary.cached, refresh) { value = withContext(Dispatchers.Default) { CertSummary.read(context) } } @@ -208,7 +330,7 @@ private fun CertificateSection() { } SettingsDivider() SettingsRow( - if (downloading) "Downloading…" else "Download certificate", + if (downloading) "Downloading…" else "Download from aasdk", Modifier.clickable(enabled = !downloading) { downloading = true scope.launch { @@ -222,13 +344,14 @@ private fun CertificateSection() { } } }, + subtitle = "The certificate the open-source aasdk project publishes on GitHub. Gearslip doesn't include one", icon = Icons.Filled.Refresh, ) SettingsDivider() SettingsRow( - "Import a certificate", + "Import my own file", Modifier.clickable { picker.launch(arrayOf("*/*")) }, - subtitle = "A .p12 file with its private key", + subtitle = "A .p12 file with its private key, from wherever you choose", icon = Icons.Filled.Add, ) val removable = when (s?.kind) { diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/ui/SetupGuideScreen.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/ui/SetupGuideScreen.kt index 08bd3ce..3fc8e8d 100644 --- a/gearslip/src/main/java/app/seb3thehacker/gearslip/ui/SetupGuideScreen.kt +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/ui/SetupGuideScreen.kt @@ -1,6 +1,44 @@ package app.seb3thehacker.gearslip.ui +import app.seb3thehacker.gearslip.speech.SpeechModels + +import androidx.compose.material3.MaterialTheme + +import androidx.compose.material3.LinearProgressIndicator + import androidx.compose.foundation.layout.BoxWithConstraints +import androidx.compose.animation.core.LinearEasing +import androidx.compose.animation.core.animateFloat +import androidx.compose.animation.core.infiniteRepeatable +import androidx.compose.animation.core.rememberInfiniteTransition +import androidx.compose.animation.core.tween +import androidx.compose.foundation.BorderStroke +import androidx.compose.foundation.background +import androidx.compose.foundation.border +import androidx.compose.foundation.clickable +import androidx.compose.foundation.layout.Box +import androidx.compose.foundation.layout.size +import androidx.compose.foundation.layout.systemBarsPadding +import androidx.compose.foundation.shape.RoundedCornerShape +import androidx.compose.material.icons.Icons +import androidx.compose.material.icons.filled.Build +import androidx.compose.material.icons.filled.Call +import androidx.compose.material.icons.filled.Check +import androidx.compose.material.icons.filled.Close +import androidx.compose.material.icons.filled.DateRange +import androidx.compose.material.icons.filled.Email +import androidx.compose.material.icons.filled.LocationOn +import androidx.compose.material.icons.filled.Notifications +import androidx.compose.material.icons.filled.Phone +import androidx.compose.material.icons.filled.Warning +import androidx.compose.material3.Icon +import androidx.compose.ui.graphics.Color +import androidx.compose.ui.graphics.graphicsLayer +import androidx.compose.ui.graphics.vector.ImageVector +import androidx.compose.ui.platform.LocalDensity +import androidx.compose.ui.text.TextStyle +import androidx.compose.ui.unit.Dp +import androidx.compose.ui.unit.TextUnit import androidx.compose.foundation.layout.heightIn import androidx.compose.foundation.rememberScrollState import androidx.compose.foundation.verticalScroll @@ -27,15 +65,7 @@ import androidx.compose.foundation.layout.fillMaxSize import androidx.compose.foundation.layout.fillMaxWidth import androidx.compose.foundation.layout.height import androidx.compose.foundation.layout.padding -import androidx.compose.material3.Button -import androidx.compose.material3.FilledTonalButton -import androidx.compose.material3.LinearProgressIndicator -import androidx.compose.material3.MaterialTheme -import androidx.compose.material3.OutlinedButton -import androidx.compose.material3.Scaffold -import androidx.compose.material3.Surface import androidx.compose.material3.Text -import androidx.compose.material3.TextButton import androidx.compose.runtime.Composable import androidx.compose.runtime.getValue import androidx.compose.runtime.mutableIntStateOf @@ -49,6 +79,14 @@ import androidx.compose.ui.text.AnnotatedString import androidx.compose.ui.text.font.FontWeight import androidx.compose.ui.unit.dp import app.seb3thehacker.gearslip.AppSettings +import app.seb3thehacker.gearslip.car.CarSettings +import androidx.compose.foundation.selection.selectable +import androidx.compose.material.icons.filled.PlayArrow +import androidx.compose.material3.RadioButton +import androidx.compose.runtime.LaunchedEffect +import androidx.compose.runtime.collectAsState +import androidx.compose.ui.semantics.Role +import androidx.compose.ui.unit.sp import app.seb3thehacker.gearslip.call.CarCalls import app.seb3thehacker.gearslip.car.Prefetch import app.seb3thehacker.gearslip.host.AndroidAuto @@ -66,14 +104,25 @@ private class GuidePage( val title: String, /** Plain text, or an [AnnotatedString] when part of it needs to stand out. */ val explanation: CharSequence, + /** Drawn inside the page's badge, in the badge's content colour. */ + val icon: @Composable (Color) -> Unit, + /** Marked "Optional" above the title: everything works without it. */ + val optional: Boolean = false, val permissions: Array? = null, val command: String? = null, val actionLabel: String? = null, val onAction: (() -> Unit)? = null, - /** Drawn under [explanation], inside the card. */ + /** Drawn under [explanation]. */ val content: (@Composable () -> Unit)? = null, + /** Only shown when the driver picked USB audio on the car audio page. */ + val usbOnly: Boolean = false, ) +/** A plain Material icon, sized to fill the badge's icon box. */ +private fun glyph(icon: ImageVector): @Composable (Color) -> Unit = { tint -> + Icon(icon, contentDescription = null, tint = tint, modifier = Modifier.fillMaxSize()) +} + /** True once every permission in [permissions] is already granted - nothing left for this page to do. */ private fun granted(context: Context, permissions: Array): Boolean = permissions.all { context.checkSelfPermission(it) == PackageManager.PERMISSION_GRANTED } @@ -105,8 +154,9 @@ private fun hasNotificationAccess(context: Context): Boolean { * who set up before the notes existed see once. Leaving it without turning them on is a no. */ private fun usageNotesPage(context: Context) = GuidePage( - title = USAGE_STATS_TITLE, + title = "Help other drivers.", explanation = USAGE_STATS_INTRO, + icon = { HeartIcon(it) }, content = { UsageStatsChoices() DisposableEffect(Unit) { @@ -119,11 +169,10 @@ private fun pages(context: Context, onRequestCallScreening: () -> Unit): List Unit): List Unit): List Unit): List Unit): List Unit): List Unit): List Unit): List Unit): List Unit): List Unit): List Unit): List "Downloading, $progress%. You can carry on; it keeps going." + states.all { it == SpeechModels.State.Ready } -> "Voice is ready." + states.any { it is SpeechModels.State.Failed } -> "The download didn't finish. Tap Download voice to try again." + else -> null + } + LaunchedEffect(Unit) { SpeechModels.init(context) } + Column(verticalArrangement = Arrangement.spacedBy(8.dp)) { + progress?.let { LinearProgressIndicator(progress = { it / 100f }, modifier = Modifier.fillMaxWidth()) } + line?.let { Text(it, style = MaterialTheme.typography.bodyMedium, color = MaterialTheme.colorScheme.onSurfaceVariant) } + } +} + +@Composable +private fun AudioChoice() { + val usb by CarSettings.pipeAudio.collectAsState() + LaunchedEffect(Unit) { if (!CarSettings.pipeAudioChosen) CarSettings.setPipeAudio(false) } + Column(verticalArrangement = Arrangement.spacedBy(12.dp)) { + AudioCard( + "Bluetooth", "Recommended", + "No prompt each drive, and your map app's voice reaches the car.", + selected = !usb, + ) { CarSettings.setPipeAudio(false) } + AudioCard( + "USB", null, + "Clearer sound with no delay. Android asks to share audio once each drive.", + selected = usb, + ) { CarSettings.setPipeAudio(true) } + } +} + +@Composable +private fun AudioCard(title: String, tag: String?, body: String, selected: Boolean, onClick: () -> Unit) { + val c = introColors() + val shape = RoundedCornerShape(24.dp) + Row( + Modifier + .fillMaxWidth() + .clip(shape) + .background(if (selected) c.primaryC else Color.Transparent) + .border(BorderStroke(2.dp, if (selected) c.primary else c.outline), shape) + .selectable(selected, role = Role.RadioButton, onClick = onClick) + .padding(16.dp), + verticalAlignment = Alignment.Top, + ) { + RadioButton(selected = selected, onClick = null) + Column(Modifier.padding(start = 12.dp)) { + Text(title, color = if (selected) c.onPrimaryC else c.onSurface, style = TextStyle(fontSize = 18.sp, fontWeight = FontWeight.Bold)) + tag?.let { Text(it, color = c.primary, style = TextStyle(fontSize = 13.sp, fontWeight = FontWeight.SemiBold)) } + Text( + body, color = if (selected) c.onPrimaryC else c.onVariant, + style = TextStyle(fontSize = 15.sp, lineHeight = 21.sp), + modifier = Modifier.padding(top = 4.dp), + ) + } + } +} + +/** + * The first-run guide, one page at a time: permissions, then removing Android Auto if it is still installed, then the optional extras. Nothing * here can be verified as done except a permission grant, so every other page is a "Next" a * driver takes on trust, with "Skip" sitting right next to it. */ @Composable fun SetupGuideScreen(onRequestCallScreening: () -> Unit, onDone: () -> Unit, onlyUsageNotes: Boolean = false) { val context = LocalContext.current - val pages = remember { if (onlyUsageNotes) listOf(usageNotesPage(context)) else pages(context, onRequestCallScreening) } + val allPages = remember { if (onlyUsageNotes) listOf(usageNotesPage(context)) else pages(context, onRequestCallScreening) } + // The USB-only pages all come after the car audio page, so dropping them never moves the + // page the driver is on. + val usbAudio by CarSettings.pipeAudio.collectAsState() + val pages = remember(usbAudio) { allPages.filter { usbAudio || !it.usbOnly } } var index by remember { mutableIntStateOf(0) } // Only ours to handle once there is a previous page to return to; at the first page, the @@ -342,31 +490,23 @@ fun SetupGuideScreen(onRequestCallScreening: () -> Unit, onDone: () -> Unit, onl context.checkSelfPermission(it) == PackageManager.PERMISSION_GRANTED } ?: false - Scaffold(containerColor = MaterialTheme.colorScheme.background) { padding -> - Column( - Modifier - .fillMaxSize() - .padding(padding) - .padding(horizontal = 24.dp), - horizontalAlignment = Alignment.CenterHorizontally, - ) { - Spacer(Modifier.height(24.dp)) - // A single page has no progress to show. - if (pages.size > 1) { - LinearProgressIndicator( - progress = { (index + 1f) / pages.size }, - modifier = Modifier.fillMaxWidth().height(10.dp).clip(MaterialTheme.shapes.extraLarge), - ) - Spacer(Modifier.height(12.dp)) - Text( - "${index + 1} of ${pages.size}", - style = MaterialTheme.typography.titleMedium, - fontWeight = FontWeight.SemiBold, - color = MaterialTheme.colorScheme.onSurfaceVariant, - ) - } - // The page scrolls between the progress bar and the buttons, so a tall page (the usage - // note's choices, say) never pushes Next off the screen. Short pages stay centred. + val c = introColors() + // The badges take turns through the phone's three accent colours, as the release intro does. + val (badge, onBadge) = listOf(c.primaryC to c.onPrimaryC, c.secondaryC to c.onSecondaryC, c.tertiaryC to c.onTertiaryC)[index % 3] + val spin by rememberInfiniteTransition(label = "spin").animateFloat( + 0f, 360f, infiniteRepeatable(tween(18_000, easing = LinearEasing)), label = "spin", + ) + + BoxWithConstraints(Modifier.fillMaxSize().background(c.surface).systemBarsPadding()) { + val w = maxWidth + // One percent of the screen's width, as in the intro, so it scales the same way. + fun cq(n: Float): Dp = w * (n / 100f) + val density = LocalDensity.current + fun cqSp(n: Float): TextUnit = with(density) { cq(n).toSp() } + + Column(Modifier.fillMaxSize().padding(horizontal = cq(8f))) { + // The page scrolls above the buttons, so a tall one (the usage notes' choices) never + // pushes them off the screen. Short pages sit in the middle. BoxWithConstraints(Modifier.weight(1f).fillMaxWidth()) { key(index) { Column( @@ -374,88 +514,114 @@ fun SetupGuideScreen(onRequestCallScreening: () -> Unit, onDone: () -> Unit, onl .fillMaxWidth() .verticalScroll(rememberScrollState()) .heightIn(min = maxHeight) - .padding(vertical = 24.dp), + .padding(vertical = cq(6f)), verticalArrangement = Arrangement.Center, ) { - Surface(color = MaterialTheme.colorScheme.surfaceContainerLow, shape = MaterialTheme.shapes.extraLarge) { - Column(Modifier.fillMaxWidth().padding(24.dp), verticalArrangement = Arrangement.spacedBy(12.dp)) { + Box(Modifier.size(cq(28f)).pop(entrance(index, 50, 800)), contentAlignment = Alignment.Center) { + Cookie(badge, Modifier.fillMaxSize().graphicsLayer { rotationZ = spin }) + Box(Modifier.size(cq(12f))) { page.icon(onBadge) } + } + Spacer(Modifier.height(cq(7f))) + if (page.optional) { + Text( + "OPTIONAL", color = c.primary, + style = TextStyle(fontSize = cqSp(3.4f), fontWeight = FontWeight.Bold, letterSpacing = cqSp(0.4f)), + modifier = Modifier.up(entrance(index, 150, 700), cq(7f)), + ) + Spacer(Modifier.height(cq(1.5f))) + } + Text( + page.title, color = c.onSurface, + // Long titles step down a size so they stay on two lines. + style = TextStyle( + fontSize = cqSp(if (page.title.length > 16) 8f else 10f), fontWeight = FontWeight.Black, + letterSpacing = cqSp(-0.4f), lineHeight = cqSp(if (page.title.length > 16) 8.6f else 10.5f), + ), + modifier = Modifier.up(entrance(index, 200, 700), cq(7f)), + ) + Spacer(Modifier.height(cq(4f))) + Column( + Modifier.up(entrance(index, 350, 700), cq(7f)), + verticalArrangement = Arrangement.spacedBy(cq(3f)), + ) { + Text( + page.explanation as? AnnotatedString ?: AnnotatedString(page.explanation.toString()), + color = c.onVariant, + style = TextStyle(fontSize = cqSp(4.5f), lineHeight = cqSp(6.5f)), + ) + page.command?.let { CopyableCommand(it) } + page.content?.invoke() + if (page.permissions != null && alreadyGranted) { Text( - page.title, - style = MaterialTheme.typography.headlineSmall, - fontWeight = FontWeight.Bold, - color = MaterialTheme.colorScheme.onSurface, + "Already allowed.", color = c.primary, + style = TextStyle(fontSize = cqSp(4.2f), fontWeight = FontWeight.SemiBold), ) - Text( - page.explanation as? AnnotatedString ?: AnnotatedString(page.explanation.toString()), - style = MaterialTheme.typography.bodyLarge, - color = MaterialTheme.colorScheme.onSurfaceVariant, - ) - page.command?.let { CopyableCommand(it) } - page.content?.invoke() } } } } } + // A single page has nothing to count. + if (pages.size > 1) { + Dots(c, pages.size, index, ::cq, Modifier.align(Alignment.CenterHorizontally)) + Spacer(Modifier.height(cq(5f))) + } + val back = if (index > 0) ({ index -= 1 }) else null + val next = { index += 1 } when { - page.permissions != null && alreadyGranted -> { - Text( - "Already allowed.", - style = MaterialTheme.typography.bodyMedium, - color = MaterialTheme.colorScheme.primary, - ) - Spacer(Modifier.height(16.dp)) - PrimaryRow(showBack = index > 0, onBack = { index -= 1 }) { - Button(onClick = { index += 1 }, modifier = it) { Text("Next") } - } - } - page.permissions != null -> { - PrimaryRow(showBack = index > 0, onBack = { index -= 1 }) { - Button(onClick = { launcher.launch(page.permissions) }, modifier = it) { Text("Allow") } - } - Spacer(Modifier.height(8.dp)) - TextButton(onClick = { index += 1 }) { Text("Skip for now") } + page.permissions != null && !alreadyGranted -> { + PillRow(c, ::cq, ::cqSp, back, "Allow") { launcher.launch(page.permissions) } + Pill(c, ::cqSp, "Skip for now", PillStyle.TEXT, Modifier.align(Alignment.CenterHorizontally), next) } page.onAction != null -> { - FilledTonalButton( - onClick = page.onAction, - modifier = Modifier.fillMaxWidth().height(52.dp), - ) { Text(page.actionLabel ?: "Open") } - Spacer(Modifier.height(8.dp)) - PrimaryRow(showBack = index > 0, onBack = { index -= 1 }) { - Button(onClick = { index += 1 }, modifier = it) { Text("Next") } - } - } - else -> { - PrimaryRow(showBack = index > 0, onBack = { index -= 1 }) { - Button(onClick = { index += 1 }, modifier = it) { - Text(if (index == pages.lastIndex) "Done" else "Next") - } - } + Pill(c, ::cqSp, page.actionLabel ?: "Open", PillStyle.OUTLINED, Modifier.fillMaxWidth().padding(bottom = cq(3f)), page.onAction) + PillRow(c, ::cq, ::cqSp, back, "Next", next) } + else -> PillRow(c, ::cq, ::cqSp, back, if (index == pages.lastIndex) "Done" else "Next", next) } - Spacer(Modifier.height(24.dp)) + Spacer(Modifier.height(cq(6f))) } } } -/** - * The bottom action row: a lower-emphasis Back on the left once there is a previous page, and the - * page's own primary button - passed [content] so the caller keeps the button's own label and - * click behavior - filling the rest of the width. - */ +internal enum class PillStyle { FILLED, OUTLINED, TEXT } + +/** The intro's rounded buttons: filled for the step's main action, outlined beside it, bare text to skip. */ @Composable -private fun PrimaryRow(showBack: Boolean, onBack: () -> Unit, content: @Composable (Modifier) -> Unit) { - if (showBack) { - Row(Modifier.fillMaxWidth(), horizontalArrangement = Arrangement.spacedBy(12.dp)) { - OutlinedButton( - onClick = onBack, - modifier = Modifier.weight(1f).height(52.dp), - ) { Text("Back") } - content(Modifier.weight(1f).height(52.dp)) - } - } else { - content(Modifier.fillMaxWidth().height(52.dp)) +internal fun Pill(c: IntroColors, cqSp: (Float) -> TextUnit, label: String, style: PillStyle, modifier: Modifier = Modifier, onClick: () -> Unit) { + val shape = RoundedCornerShape(50) + Box( + modifier + .clip(shape) + .then( + when (style) { + PillStyle.FILLED -> Modifier.background(c.primary) + PillStyle.OUTLINED -> Modifier.border(BorderStroke(2.dp, c.outline), shape) + PillStyle.TEXT -> Modifier + }, + ) + .clickable(onClick = onClick) + .heightIn(min = 52.dp) + .padding(horizontal = 20.dp, vertical = 12.dp), + contentAlignment = Alignment.Center, + ) { + Text( + label, + color = if (style == PillStyle.FILLED) c.onPrimary else c.primary, + style = TextStyle(fontSize = cqSp(4.4f), fontWeight = FontWeight.Bold), + ) + } +} + +/** Back on the left once there's a page to go back to, and the page's main action filling the rest. */ +@Composable +internal fun PillRow( + c: IntroColors, cq: (Float) -> Dp, cqSp: (Float) -> TextUnit, + onBack: (() -> Unit)?, label: String, onClick: () -> Unit, +) { + Row(Modifier.fillMaxWidth(), horizontalArrangement = Arrangement.spacedBy(cq(3f))) { + if (onBack != null) Pill(c, cqSp, "Back", PillStyle.OUTLINED, Modifier.weight(1f), onBack) + Pill(c, cqSp, label, PillStyle.FILLED, Modifier.weight(if (onBack != null) 1.4f else 1f), onClick) } } diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/ui/UpdateViews.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/ui/UpdateViews.kt new file mode 100644 index 0000000..c4b8774 --- /dev/null +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/ui/UpdateViews.kt @@ -0,0 +1,237 @@ +package app.seb3thehacker.gearslip.ui + +import android.content.Intent +import android.net.Uri +import androidx.compose.foundation.clickable +import androidx.compose.foundation.layout.Arrangement +import androidx.compose.foundation.layout.Column +import androidx.compose.foundation.layout.Row +import androidx.compose.foundation.layout.Spacer +import androidx.compose.foundation.layout.fillMaxWidth +import androidx.compose.foundation.layout.height +import androidx.compose.foundation.layout.padding +import androidx.compose.foundation.selection.selectable +import androidx.compose.foundation.selection.toggleable +import androidx.compose.material.icons.Icons +import androidx.compose.material.icons.automirrored.filled.KeyboardArrowRight +import androidx.compose.material.icons.filled.CheckCircle +import androidx.compose.material3.AlertDialog +import androidx.compose.material3.Button +import androidx.compose.material3.Icon +import androidx.compose.material3.LinearProgressIndicator +import androidx.compose.material3.MaterialTheme +import androidx.compose.material3.RadioButton +import androidx.compose.material3.Switch +import androidx.compose.material3.Surface +import androidx.compose.material3.Text +import androidx.compose.material3.TextButton +import androidx.compose.runtime.Composable +import androidx.compose.runtime.getValue +import androidx.compose.runtime.mutableStateOf +import androidx.compose.runtime.remember +import androidx.compose.runtime.setValue +import androidx.compose.ui.Alignment +import androidx.compose.ui.Modifier +import androidx.compose.ui.platform.LocalContext +import androidx.compose.ui.semantics.Role +import androidx.compose.ui.text.font.FontWeight +import androidx.compose.ui.unit.dp +import androidx.lifecycle.compose.collectAsStateWithLifecycle +import app.seb3thehacker.gearslip.BuildConfig +import app.seb3thehacker.gearslip.SessionStatus +import app.seb3thehacker.gearslip.update.UpdateChecker +import app.seb3thehacker.gearslip.update.UpdateChecker.State + +private const val UNPLUG_FIRST = "Unplug from the car to install. Installing closes Gearslip." + +/** Shown on Home once a newer version is found; hidden otherwise. */ +@Composable +fun UpdateCard() { + val context = LocalContext.current + val state by UpdateChecker.state.collectAsStateWithLifecycle() + SessionStatus.state.collectAsStateWithLifecycle().value // redraw when the car comes or goes + val release = when (val s = state) { + is State.Available -> s.release + is State.Downloading -> s.release + is State.Ready -> s.release + is State.Failed -> s.release + else -> null + } ?: return + + Surface( + color = MaterialTheme.colorScheme.tertiaryContainer, + contentColor = MaterialTheme.colorScheme.onTertiaryContainer, + shape = MaterialTheme.shapes.extraLarge, + ) { + Column(Modifier.fillMaxWidth().padding(20.dp)) { + Text("Gearslip ${release.version} is out", style = MaterialTheme.typography.titleLarge, fontWeight = FontWeight.SemiBold) + Spacer(Modifier.height(4.dp)) + when (val s = state) { + is State.Downloading -> { + Text("Downloading… ${(s.fraction * 100).toInt()}%", style = MaterialTheme.typography.bodyMedium) + Spacer(Modifier.height(12.dp)) + LinearProgressIndicator(progress = { s.fraction }, modifier = Modifier.fillMaxWidth()) + } + is State.Ready -> { + val connected = UpdateChecker.carConnected + Text( + if (connected) UNPLUG_FIRST else "Downloaded. Android asks before it installs.", + style = MaterialTheme.typography.bodyMedium, + ) + Spacer(Modifier.height(12.dp)) + Button(onClick = { UpdateChecker.install(context) }, enabled = !connected) { Text("Install") } + } + else -> { + Text( + (s as? State.Failed)?.message ?: "Download it now. Android asks before it installs.", + style = MaterialTheme.typography.bodyMedium, + ) + Spacer(Modifier.height(12.dp)) + Row(horizontalArrangement = Arrangement.spacedBy(8.dp)) { + Button(onClick = { UpdateChecker.download(context, release) }) { + Text(if (s is State.Failed) "Try again" else "Update") + } + if (release.pageUrl.isNotBlank()) { + TextButton(onClick = { + context.startActivity(Intent(Intent.ACTION_VIEW, Uri.parse(release.pageUrl))) + }) { Text("What's new") } + } + } + } + } + } + } + // Room before whatever Home shows next; nothing when the card is hidden. + Spacer(Modifier.height(16.dp)) +} + +/** What the update check found, in a line. */ +private fun statusLine(state: State): String = when (state) { + State.Idle -> if (BuildConfig.DEBUG) "Debug builds check only when you tap Check now" else "Gearslip checks once a day" + State.Checking -> "Checking…" + State.UpToDate -> "You have the latest version" + is State.Available -> "Version ${state.release.version} is out" + is State.Downloading -> "Downloading… ${(state.fraction * 100).toInt()}%" + is State.Ready -> if (UpdateChecker.carConnected) UNPLUG_FIRST else "Version ${state.release.version} is ready to install" + is State.Failed -> state.message +} + +/** + * The channels the dialog offers. Empty for now: every release so far goes out as GitHub's latest, + * so Beta and Alpha would offer the same as Stable. Add them back once a pre-release ships: + * Triple(UpdateChecker.Channel.STABLE, "Stable", "Finished releases only"), + * Triple(UpdateChecker.Channel.BETA, "Beta", "Test versions that are nearly done"), + * Triple(UpdateChecker.Channel.ALPHA, "Alpha", "Every test version, first. Expect bugs"), + */ +private val CHANNELS = emptyList>() + +/** The one Settings row for updates. Checking, the channel and automatic checks all sit behind it. */ +@Composable +internal fun UpdateRow() { + val context = LocalContext.current + val state by UpdateChecker.state.collectAsStateWithLifecycle() + SessionStatus.state.collectAsStateWithLifecycle().value + var channel by remember { mutableStateOf(UpdateChecker.channel(context)) } + var open by remember { mutableStateOf(false) } + val channelName = CHANNELS.firstOrNull { it.first == channel }?.second + SettingsRow( + "Updates", + Modifier.clickable { open = true }, + subtitle = listOfNotNull(channelName, statusLine(state)).joinToString(" · "), + icon = Icons.Filled.CheckCircle, + trailing = { + Icon( + Icons.AutoMirrored.Filled.KeyboardArrowRight, + contentDescription = null, + tint = MaterialTheme.colorScheme.onSurfaceVariant, + ) + }, + ) + if (open) { + UpdateDialog( + state = state, + channel = channel, + onChannel = { + channel = it + UpdateChecker.setChannel(context, it) + }, + onDismiss = { open = false }, + ) + } +} + +@Composable +private fun UpdateDialog( + state: State, + channel: UpdateChecker.Channel, + onChannel: (UpdateChecker.Channel) -> Unit, + onDismiss: () -> Unit, +) { + val context = LocalContext.current + var auto by remember { mutableStateOf(UpdateChecker.autoCheckEnabled(context)) } + val action: Pair Unit>? = when (state) { + State.Checking, is State.Downloading -> null + is State.Available -> "Download" to { UpdateChecker.download(context, state.release) } + is State.Ready -> if (UpdateChecker.carConnected) null else "Install" to { UpdateChecker.install(context) } + is State.Failed -> "Try again" to { + state.release?.let { UpdateChecker.download(context, it) } ?: UpdateChecker.check(context) + } + else -> "Check now" to { UpdateChecker.check(context) } + } + AlertDialog( + onDismissRequest = onDismiss, + title = { Text("Updates") }, + text = { + Column { + Text( + "Version ${BuildConfig.VERSION_NAME}. ${statusLine(state)}.", + style = MaterialTheme.typography.bodyMedium, + modifier = Modifier.padding(bottom = 12.dp), + ) + CHANNELS.forEach { (value, label, description) -> + Row( + Modifier + .fillMaxWidth() + .selectable(channel == value, role = Role.RadioButton) { onChannel(value) } + .padding(vertical = 8.dp), + verticalAlignment = Alignment.CenterVertically, + ) { + RadioButton(selected = channel == value, onClick = null) + Column(Modifier.padding(start = 16.dp)) { + Text(label, style = MaterialTheme.typography.bodyLarge) + Text( + description, + style = MaterialTheme.typography.bodyMedium, + color = MaterialTheme.colorScheme.onSurfaceVariant, + ) + } + } + } + Row( + Modifier + .fillMaxWidth() + .toggleable(auto, role = Role.Switch) { + auto = it + UpdateChecker.setAutoCheckEnabled(context, it) + } + .padding(top = 12.dp, bottom = 4.dp), + verticalAlignment = Alignment.CenterVertically, + ) { + Column(Modifier.weight(1f)) { + Text("Check automatically", style = MaterialTheme.typography.bodyLarge) + Text( + if (BuildConfig.DEBUG) "Debug builds check only when you tap Check now" else "Once a day", + style = MaterialTheme.typography.bodyMedium, + color = MaterialTheme.colorScheme.onSurfaceVariant, + ) + } + Switch(checked = auto, onCheckedChange = null) + } + } + }, + confirmButton = { + action?.let { (label, run) -> TextButton(onClick = run) { Text(label) } } + }, + dismissButton = { TextButton(onClick = onDismiss) { Text("Done") } }, + ) +} diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/ui/UsageStatsChoices.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/ui/UsageStatsChoices.kt index 70fbc61..c030d99 100644 --- a/gearslip/src/main/java/app/seb3thehacker/gearslip/ui/UsageStatsChoices.kt +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/ui/UsageStatsChoices.kt @@ -1,14 +1,24 @@ package app.seb3thehacker.gearslip.ui import androidx.compose.animation.AnimatedVisibility +import android.content.Context import androidx.compose.foundation.clickable import androidx.compose.foundation.layout.Column import androidx.compose.foundation.layout.ColumnScope +import androidx.compose.foundation.layout.Arrangement +import androidx.compose.foundation.layout.PaddingValues +import androidx.compose.foundation.layout.Spacer +import androidx.compose.material3.Button +import androidx.compose.material3.OutlinedButton +import androidx.compose.material3.TextButton +import app.seb3thehacker.gearslip.stats.WorkedPrompt import androidx.compose.foundation.layout.Row import androidx.compose.foundation.layout.fillMaxWidth import androidx.compose.foundation.layout.padding import androidx.compose.foundation.selection.toggleable import androidx.compose.material.icons.Icons +import androidx.compose.material.icons.automirrored.filled.KeyboardArrowRight +import androidx.compose.material.icons.filled.Check import androidx.compose.material.icons.filled.Info import androidx.compose.material.icons.filled.KeyboardArrowDown import androidx.compose.material.icons.filled.KeyboardArrowUp @@ -27,6 +37,10 @@ import androidx.compose.runtime.remember import androidx.compose.runtime.setValue import androidx.compose.ui.Alignment import androidx.compose.ui.Modifier +import androidx.compose.foundation.rememberScrollState +import androidx.compose.foundation.verticalScroll +import androidx.compose.material3.AlertDialog +import androidx.compose.material3.TextButton import androidx.compose.ui.platform.LocalContext import androidx.compose.ui.semantics.Role import androidx.compose.ui.text.AnnotatedString @@ -42,18 +56,13 @@ import app.seb3thehacker.gearslip.stats.UsageStats internal const val USAGE_STATS_TITLE = "Help show which cars work" private const val INTRO_START = - "Gearslip can't tell which cars it works in unless drivers say so. Turn this on, and Gearslip " + - "sends a short note when you open it and after each drive, never in the background. The " + - "note carries the Gearslip version and whatever you choose to share. " + "Turn this on, and Gearslip sends a short note when you open it and after each drive. " private const val INTRO_PRIVACY = - "It holds no name, account, or location, only a random number so your phone counts once. " + - "Cloudflare carries the note, and the server keeps no IP address." + "The note holds no name, account, or location. A random number counts your phone once, and the server keeps no IP address." private const val INTRO_END = - " Nothing helps the project more. The notes show which cars work, so the next driver knows " + - "before installing. When a car fails, the note says where the connection broke, which is " + - "often enough to fix a car no one working on Gearslip owns." + " The notes show which cars work and where the others fail." -/** The pitch for the usage notes, with the privacy promise in bold so it isn't missed. */ +/** The pitch for the usage notes, in setup and Settings, with the privacy promise in bold so it isn't missed. */ internal val USAGE_STATS_INTRO: AnnotatedString = buildAnnotatedString { append(INTRO_START) withStyle(SpanStyle(fontWeight = FontWeight.Bold)) { append(INTRO_PRIVACY) } @@ -83,11 +92,7 @@ internal fun ColumnScope.UsageStatsRows() { SettingsRow( "Send usage notes", Modifier.toggleable(on, role = Role.Switch) { on = it; save() }, - subtitle = when { - BuildConfig.DEBUG -> "Dev build: nothing is ever sent" - BuildConfig.STATS_URL.isBlank() -> "This build has no stats server, so it sends nothing" - else -> null - }, + subtitle = buildNote(), icon = Icons.Filled.Send, trailing = { Switch(checked = on, onCheckedChange = null) }, ) @@ -100,17 +105,21 @@ internal fun ColumnScope.UsageStatsRows() { color = MaterialTheme.colorScheme.primary, modifier = Modifier.padding(start = 56.dp, end = 16.dp, top = 12.dp, bottom = 4.dp), ) - Choice("Android version, and whether it's GrapheneOS", choices.android) { choices = choices.copy(android = it); save() } + Choice("Android version and system", choices.android) { choices = choices.copy(android = it); save() } Choice("Phone model", choices.phone) { choices = choices.copy(phone = it); save() } - Choice("Each car's head unit, model, year, and screen size, and whether it connected", choices.cars) { + Choice("Each car and whether it connected", choices.cars) { choices = choices.copy(cars = it) save() } + Choice("Each drive and how it ended", choices.failures) { + choices = choices.copy(failures = it) + save() + } SettingsDivider() SettingsRow( "What gets sent", Modifier.clickable { noteOpen = !noteOpen }, - subtitle = "Each car goes after the drive ends. The Android version and phone model go only when they change, and once a month", + subtitle = "Every field, exactly as it goes out", icon = Icons.Filled.Info, trailing = { Icon( @@ -122,7 +131,7 @@ internal fun ColumnScope.UsageStatsRows() { ) AnimatedVisibility(noteOpen) { // Rebuilt from the saved choices each time one changes, so it always matches. - val note = remember(choices) { UsageStats.preview(context, everything = true).toString(2) } + val note = remember(choices) { notePreview(context) } Surface( color = MaterialTheme.colorScheme.surfaceContainerLowest, shape = MaterialTheme.shapes.medium, @@ -152,3 +161,68 @@ private fun Choice(label: String, checked: Boolean, onChange: (Boolean) -> Unit) Checkbox(checked = checked, onCheckedChange = null, modifier = Modifier.padding(vertical = 8.dp)) } } + +/** The note as it would go out now, with an example drive until a real one is in it. */ +private fun notePreview(context: Context): String { + val preview = UsageStats.preview(context, everything = true) + // Until the first drive the list is empty, so show what one entry looks like. + val example = if (preview.optJSONArray("sessions")?.length() == 0) { + "\n\nExample, not sent: one drive adds this to \"sessions\"\n" + UsageStats.exampleSession.toString(2) + } else "" + return preview.toString(2) + example +} + +/** What's different about where this build's notes go; null for a normal release. */ +private fun buildNote(): String? = when { + BuildConfig.STATS_URL.isBlank() -> "This build has no stats server, so it sends nothing" + BuildConfig.DEBUG -> "Debug build: notes are kept apart and never counted in the stats" + else -> null +} + +/** + * The one-time ask after the first drive that worked, in the status card's place on Home. Share + * turns on usage notes like the switch does; the link shows exactly what that sends first. + */ +@Composable +internal fun WorkedCard(car: String, onSeeWhatsSent: () -> Unit, modifier: Modifier = Modifier) { + val context = LocalContext.current + val scheme = MaterialTheme.colorScheme + Surface( + modifier, + color = scheme.primaryContainer, + contentColor = scheme.onPrimaryContainer, + shape = MaterialTheme.shapes.extraLarge, + ) { + Column(Modifier.fillMaxWidth().padding(20.dp)) { + Row(verticalAlignment = Alignment.CenterVertically) { + Icon(Icons.Filled.Check, contentDescription = null) + Text( + WorkedPrompt.TITLE_START + car, + style = MaterialTheme.typography.titleLarge, + fontWeight = FontWeight.SemiBold, + modifier = Modifier.padding(start = 12.dp), + ) + } + Text( + "Share that so the next driver knows it works? No name, account, or location.", + style = MaterialTheme.typography.bodyMedium, + modifier = Modifier.padding(top = 8.dp), + ) + buildNote()?.let { + Text(it, style = MaterialTheme.typography.bodySmall, modifier = Modifier.padding(top = 4.dp)) + } + Row( + Modifier.fillMaxWidth().padding(top = 12.dp), + verticalAlignment = Alignment.CenterVertically, + horizontalArrangement = Arrangement.spacedBy(8.dp), + ) { + TextButton(onClick = onSeeWhatsSent, contentPadding = PaddingValues(horizontal = 0.dp)) { + Text("What gets sent") + } + Spacer(Modifier.weight(1f)) + OutlinedButton(onClick = { WorkedPrompt.answer(context, share = false) }) { Text("Not now") } + Button(onClick = { WorkedPrompt.answer(context, share = true) }) { Text("Share") } + } + } + } +} diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/ui/VoiceModelViews.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/ui/VoiceModelViews.kt new file mode 100644 index 0000000..da48810 --- /dev/null +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/ui/VoiceModelViews.kt @@ -0,0 +1,125 @@ +package app.seb3thehacker.gearslip.ui + +import androidx.compose.foundation.layout.Arrangement +import androidx.compose.foundation.layout.Column +import androidx.compose.foundation.layout.Row +import androidx.compose.foundation.layout.fillMaxWidth +import androidx.compose.foundation.layout.padding +import androidx.compose.foundation.selection.selectable +import androidx.compose.material3.LinearProgressIndicator +import androidx.compose.material3.MaterialTheme +import androidx.compose.material3.OutlinedButton +import androidx.compose.material3.RadioButton +import androidx.compose.material3.Text +import androidx.compose.material3.TextButton +import androidx.compose.runtime.Composable +import androidx.compose.runtime.collectAsState +import androidx.compose.runtime.getValue +import androidx.compose.runtime.remember +import androidx.compose.ui.Alignment +import androidx.compose.ui.Modifier +import androidx.compose.ui.platform.LocalContext +import androidx.compose.ui.semantics.Role +import androidx.compose.ui.unit.dp +import app.seb3thehacker.gearslip.speech.SpeechModels +import app.seb3thehacker.gearslip.speech.SpeechModels.Model +import app.seb3thehacker.gearslip.speech.SpeechModels.State + +/** + * Which speech model the voice assistant listens with, and downloading it. Each is fetched once + * from sherpa-onnx's author ([SpeechModels] says where); nothing is sent anywhere while listening. + */ +@Composable +internal fun VoiceModelSection() { + val context = LocalContext.current + remember { SpeechModels.init(context) } + val selected by SpeechModels.selected.collectAsState() + val final by SpeechModels.final.collectAsState() + SettingsSection(title = "Voice: live words") { + Text( + "The voice assistant turns speech into text on this phone, and nothing you say leaves it. " + + "It needs two models, about 270 MB together, downloaded once. " + + "This one writes your words as you speak.", + style = MaterialTheme.typography.bodyMedium, + color = MaterialTheme.colorScheme.onSurfaceVariant, + modifier = Modifier.padding(horizontal = 16.dp, vertical = 12.dp), + ) + Model.entries.filter { it.role == SpeechModels.Role.LIVE }.forEachIndexed { i, model -> + if (i > 0) SettingsDivider() + ModelRow(model, model == selected) + } + } + SettingsSection(title = "Voice: final words") { + Text( + "When you stop talking, this model reads the whole phrase again. It catches names and " + + "places the live model misses.", + style = MaterialTheme.typography.bodyMedium, + color = MaterialTheme.colorScheme.onSurfaceVariant, + modifier = Modifier.padding(horizontal = 16.dp, vertical = 12.dp), + ) + val context = LocalContext.current + Row( + Modifier + .fillMaxWidth() + .selectable(final == null, role = Role.RadioButton) { SpeechModels.selectFinal(context, null) } + .padding(horizontal = 8.dp, vertical = 8.dp), + verticalAlignment = Alignment.CenterVertically, + ) { + RadioButton(final == null, onClick = null, modifier = Modifier.padding(8.dp)) + Column { + Text("None", style = MaterialTheme.typography.titleMedium) + Text( + "Keep the live words. Fastest, least accurate", + style = MaterialTheme.typography.bodySmall, + color = MaterialTheme.colorScheme.onSurfaceVariant, + ) + } + } + Model.entries.filter { it.role == SpeechModels.Role.FINAL }.forEach { model -> + SettingsDivider() + ModelRow(model, model == final) + } + } +} + +@Composable +private fun ModelRow(model: Model, chosen: Boolean) { + val context = LocalContext.current + val state by SpeechModels.state(model).collectAsState() + Column( + Modifier + .fillMaxWidth() + .selectable(chosen, role = Role.RadioButton) { SpeechModels.select(context, model) } + .padding(horizontal = 8.dp, vertical = 8.dp), + ) { + Row(verticalAlignment = Alignment.CenterVertically) { + RadioButton(chosen, onClick = null, modifier = Modifier.padding(8.dp)) + Column(Modifier.weight(1f)) { + Text("${model.label} · ${model.downloadMb} MB", style = MaterialTheme.typography.titleMedium) + Text( + when (val s = state) { + State.Ready -> "Downloaded. ${model.detail}" + is State.Downloading -> "Downloading, ${s.percent}%" + is State.Failed -> "Download failed: ${s.reason}" + State.Missing -> model.detail + }, + style = MaterialTheme.typography.bodySmall, + color = MaterialTheme.colorScheme.onSurfaceVariant, + ) + } + Row(horizontalArrangement = Arrangement.spacedBy(4.dp)) { + when (state) { + State.Ready -> TextButton(onClick = { SpeechModels.delete(context, model) }) { Text("Delete") } + is State.Downloading -> TextButton(onClick = { SpeechModels.delete(context, model) }) { Text("Cancel") } + else -> OutlinedButton(onClick = { SpeechModels.download(context, model) }) { Text("Download") } + } + } + } + (state as? State.Downloading)?.let { + LinearProgressIndicator( + progress = { it.percent / 100f }, + modifier = Modifier.fillMaxWidth().padding(start = 48.dp, end = 8.dp, top = 4.dp), + ) + } + } +} diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/ui/WirelessButton.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/ui/WirelessButton.kt new file mode 100644 index 0000000..109569d --- /dev/null +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/ui/WirelessButton.kt @@ -0,0 +1,127 @@ +package app.seb3thehacker.gearslip.ui + +import android.Manifest +import android.content.Intent +import android.provider.Settings +import android.bluetooth.BluetoothDevice +import android.content.pm.PackageManager +import android.os.Build +import androidx.activity.compose.rememberLauncherForActivityResult +import androidx.activity.result.contract.ActivityResultContracts +import androidx.compose.foundation.clickable +import androidx.compose.foundation.layout.Box +import androidx.compose.foundation.layout.Column +import androidx.compose.foundation.layout.Spacer +import androidx.compose.foundation.layout.size +import androidx.compose.foundation.layout.width +import androidx.compose.foundation.layout.fillMaxWidth +import androidx.compose.foundation.layout.height +import androidx.compose.foundation.layout.padding +import androidx.compose.foundation.rememberScrollState +import androidx.compose.foundation.verticalScroll +import androidx.compose.material3.AlertDialog +import androidx.compose.material3.Button +import androidx.compose.material3.MaterialTheme +import androidx.compose.material3.Text +import androidx.compose.material3.TextButton +import androidx.compose.runtime.Composable +import androidx.compose.runtime.getValue +import androidx.compose.runtime.mutableStateOf +import androidx.compose.runtime.remember +import androidx.compose.runtime.setValue +import androidx.compose.ui.Modifier +import androidx.compose.ui.platform.LocalContext +import androidx.compose.ui.text.font.FontWeight +import androidx.compose.ui.unit.dp +import androidx.core.content.ContextCompat +import app.seb3thehacker.gearslip.wireless.WirelessLink + +/** What the wireless start-up needs: Bluetooth to reach the car, nearby Wi-Fi to join its network, local network to talk to it. */ +private val WIRELESS_PERMISSIONS = buildList { + add(Manifest.permission.BLUETOOTH_CONNECT) + if (Build.VERSION.SDK_INT >= 33) add(Manifest.permission.NEARBY_WIFI_DEVICES) + // Android 17 silently drops traffic to local addresses, the car included, without it. + if (Build.VERSION.SDK_INT >= 37) add("android.permission.ACCESS_LOCAL_NETWORK") + add(Manifest.permission.ACCESS_FINE_LOCATION) +}.toTypedArray() + +/** + * Wireless Android Auto, in Home's status card: pick the paired car, and Gearslip connects to + * it over Bluetooth and Wi-Fi. + */ +@Composable +internal fun WirelessButton() { + val context = LocalContext.current + var devices by remember { mutableStateOf?>(null) } + var denied by remember { mutableStateOf(false) } + fun granted() = WIRELESS_PERMISSIONS.all { + ContextCompat.checkSelfPermission(context, it) == PackageManager.PERMISSION_GRANTED + } + val ask = rememberLauncherForActivityResult(ActivityResultContracts.RequestMultiplePermissions()) { + if (granted()) devices = WirelessLink.pairedDevices(context) else denied = true + } + + if (denied) { + Text( + "Gearslip needs Bluetooth and nearby Wi-Fi access for this.", + style = MaterialTheme.typography.bodyMedium, + modifier = Modifier.padding(bottom = 8.dp), + ) + } + Button( + onClick = { + denied = false + if (granted()) devices = WirelessLink.pairedDevices(context) else ask.launch(WIRELESS_PERMISSIONS) + }, + modifier = Modifier.fillMaxWidth().height(64.dp), + ) { + Box(Modifier.size(28.dp)) { WifiIcon(MaterialTheme.colorScheme.onPrimary) } + Spacer(Modifier.width(12.dp)) + Text("Connect wirelessly", style = MaterialTheme.typography.titleMedium, fontWeight = FontWeight.SemiBold) + } + + devices?.let { list -> + AlertDialog( + onDismissRequest = { devices = null }, + title = { Text("Which car?") }, + text = { + Column(Modifier.verticalScroll(rememberScrollState())) { + if (list.isEmpty()) { + Text("Your phone isn't paired with anything yet. Pair it with the car first.") + } + list.forEach { device -> + Text( + WirelessLink.name(device), + style = MaterialTheme.typography.bodyLarge, + modifier = Modifier + .fillMaxWidth() + .clickable { + devices = null + WirelessLink.start(context, device) + } + .padding(vertical = 14.dp), + ) + } + // The car has to be paired before it shows up here, and pairing happens in Android's settings. + if (list.isNotEmpty()) { + Text( + "Car not listed? Pair it with your phone first.", + style = MaterialTheme.typography.bodyMedium, + color = MaterialTheme.colorScheme.onSurfaceVariant, + modifier = Modifier.padding(top = 8.dp), + ) + } + } + }, + dismissButton = { + TextButton(onClick = { + devices = null + runCatching { + context.startActivity(Intent(Settings.ACTION_BLUETOOTH_SETTINGS).addFlags(Intent.FLAG_ACTIVITY_NEW_TASK)) + } + }) { Text("Bluetooth settings") } + }, + confirmButton = { TextButton(onClick = { devices = null }) { Text("Cancel") } }, + ) + } +} diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/update/UpdateChecker.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/update/UpdateChecker.kt new file mode 100644 index 0000000..6877498 --- /dev/null +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/update/UpdateChecker.kt @@ -0,0 +1,437 @@ +package app.seb3thehacker.gearslip.update + +import android.Manifest +import android.app.NotificationChannel +import android.app.NotificationManager +import android.app.PendingIntent +import android.content.Context +import android.content.Intent +import android.content.pm.PackageManager +import android.net.Uri +import android.provider.Settings +import androidx.core.app.NotificationCompat +import androidx.core.app.NotificationManagerCompat +import androidx.core.content.FileProvider +import androidx.core.graphics.drawable.toBitmap +import app.seb3thehacker.gearslip.BuildConfig +import app.seb3thehacker.gearslip.GearslipActivity +import app.seb3thehacker.gearslip.GearslipLog +import app.seb3thehacker.gearslip.SessionStatus +import app.seb3thehacker.gearslip.notify.CarMotion +import app.seb3thehacker.gearslip.notify.CarNotification +import app.seb3thehacker.gearslip.notify.CarNotifications +import kotlinx.coroutines.flow.MutableStateFlow +import kotlinx.coroutines.flow.StateFlow +import kotlinx.coroutines.flow.asStateFlow +import org.json.JSONObject +import java.io.File +import java.net.HttpURLConnection +import java.net.URL +import java.security.MessageDigest + +/** + * Looks for a newer Gearslip on GitHub and installs it through Android's own installer. + * + * Which releases count depends on the driver's channel. Stable takes only GitHub's "latest" + * release, which leaves out pre-releases and drafts. Beta adds pre-releases tagged "beta", and + * Alpha adds every pre-release. The app checks at most once a day: when + * it opens, and when a drive ends. With automatic checks off it never asks GitHub on its own, + * only when the driver taps Check for updates. After a drive a new version also gets + * a notification, once per version, since the driver is parked and the app is likely closed. The + * car screen shows the same news once per version, at plug-in and only while parked. + * Nothing runs in the background otherwise, and nothing else is sent: the request is the same one + * a browser makes for the release page. Debug builds never check on their own. + * + * The APK is checked against the size and SHA-256 GitHub lists for it, and Android refuses an + * update signed with a different key, so a bad download can't replace the app. + */ +object UpdateChecker { + + data class Release(val version: String, val apkUrl: String, val size: Long, val sha256: String?, val pageUrl: String) + + sealed interface State { + data object Idle : State + data object Checking : State + data object UpToDate : State + data class Available(val release: Release) : State + data class Downloading(val release: Release, val fraction: Float) : State + data class Ready(val release: Release, val apk: File) : State + data class Failed(val message: String, val release: Release? = null) : State + } + + private val flow = MutableStateFlow(State.Idle) + val state: StateFlow = flow.asStateFlow() + + private const val PREFS = "updates" + private const val KEY_CHECKED_AT = "checked_at" + private const val KEY_FOUND = "found" + /** Named for the notifications toggle it grew out of; it now covers every automatic check. */ + private const val KEY_AUTO = "notify" + /** The old on/off switch for pre-releases. On meant every pre-release, which is now Alpha. */ + private const val KEY_BETA = "beta" + private const val KEY_CHANNEL = "channel" + private const val KEY_NOTIFIED = "notified_version" + private const val KEY_CAR_NOTIFIED = "car_notified_version" + /** Lets the car screen settle after plug-in, so the card isn't lost behind the first frames. */ + private const val CAR_DELAY_MS = 8_000L + private const val CHANNEL = "updates" + private const val NOTIFICATION_ID = 4100 + private const val DAY_MS = 24 * 60 * 60 * 1000L + private const val LATEST = "https://api.github.com/repos/Seb3thehacker/gearslip/releases/latest" + /** Newest first, pre-releases included. Ten covers any run of betas between two releases. */ + private const val ALL = "https://api.github.com/repos/Seb3thehacker/gearslip/releases?per_page=10" + + /** Set when the driver tapped Install but first had to allow installs from Gearslip. */ + @Volatile private var installWhenAllowed = false + + private fun prefs(context: Context) = context.applicationContext.getSharedPreferences(PREFS, Context.MODE_PRIVATE) + + /** + * Whether Gearslip checks on its own, and tells the driver what it finds on the phone and the + * car screen. On by default: the check sends nothing about the phone. + */ + fun autoCheckEnabled(context: Context): Boolean = prefs(context).getBoolean(KEY_AUTO, true) + + fun setAutoCheckEnabled(context: Context, on: Boolean) { + prefs(context).edit().putBoolean(KEY_AUTO, on).apply() + if (!on) NotificationManagerCompat.from(context).cancel(NOTIFICATION_ID) + } + + /** Which releases count as updates. Each channel also takes everything the one before it does. */ + enum class Channel { STABLE, BETA, ALPHA } + + fun channel(context: Context): Channel { + val p = prefs(context) + p.getString(KEY_CHANNEL, null)?.let { saved -> Channel.entries.firstOrNull { it.name == saved }?.let { return it } } + return if (p.getBoolean(KEY_BETA, false)) Channel.ALPHA else Channel.STABLE + } + + /** + * Switching channels forgets what the old one found: an alpha found earlier shouldn't still be + * offered on Stable. Checks again right away if automatic checks are on. + */ + fun setChannel(context: Context, channel: Channel) { + prefs(context).edit().putString(KEY_CHANNEL, channel.name).remove(KEY_BETA).remove(KEY_FOUND).remove(KEY_CHECKED_AT).apply() + val current = flow.value + if (current is State.Checking || current is State.Downloading) return + updatesDir(context).deleteRecursively() + flow.value = State.Idle + if (autoCheckEnabled(context) && !BuildConfig.DEBUG) check(context) + } + + private fun dueForCheck(context: Context) = + !BuildConfig.DEBUG && autoCheckEnabled(context) && System.currentTimeMillis() - prefs(context).getLong(KEY_CHECKED_AT, 0) >= DAY_MS + + /** Gearslip came to the front: check if a day has passed, and finish an install that waited on permission. */ + fun onAppOpened(context: Context) { + val current = flow.value + if (installWhenAllowed && current is State.Ready && context.packageManager.canRequestPackageInstalls()) { + installWhenAllowed = false + install(context) + return + } + if (current != State.Idle) return + // A release found earlier still shows after Android closed the app. + savedRelease(context)?.let { flow.value = State.Available(it) } + if (dueForCheck(context)) check(context) + } + + /** + * A car session ended, so the driver is parked. Checks if a day has passed, then posts a + * notification for a new version they haven't been told about yet. + */ + fun onDriveEnded(context: Context) { + if (!autoCheckEnabled(context)) return + val app = context.applicationContext + Thread({ + val current = flow.value + if (dueForCheck(app) && current !is State.Checking && current !is State.Downloading) { + flow.value = State.Checking + flow.value = checkNow(app) + } + val release = foundRelease(app) ?: return@Thread + if (prefs(app).getString(KEY_NOTIFIED, null) == release.version) return@Thread + if (notify(app, release)) prefs(app).edit().putString(KEY_NOTIFIED, release.version).apply() + }, "gearslip-update").start() + } + + /** + * The car screen just came up. A new version gets a card in the top right, like a message, + * once per version. Skipped while moving; the next plug-in tries again. + */ + fun onCarConnected(context: Context) { + if (!autoCheckEnabled(context)) return + val app = context.applicationContext + Thread({ + val current = flow.value + if (dueForCheck(app) && current !is State.Checking && current !is State.Downloading) { + flow.value = State.Checking + flow.value = checkNow(app) + } + val release = foundRelease(app) ?: return@Thread + if (prefs(app).getString(KEY_CAR_NOTIFIED, null) == release.version) return@Thread + Thread.sleep(CAR_DELAY_MS) + if (!carConnected || CarMotion.moving) return@Thread + CarNotifications.post( + CarNotification( + key = "gearslip-update", + id = CarNotifications.nextId(), + packageName = app.packageName, + appLabel = "Gearslip", + title = "Gearslip ${release.version} is out", + text = "Update in the Gearslip app or from GitHub", + postedAt = System.currentTimeMillis(), + icon = runCatching { app.packageManager.getApplicationIcon(app.packageName).toBitmap(96, 96) }.getOrNull(), + reply = null, + ), + quiet = false, + ) + prefs(app).edit().putString(KEY_CAR_NOTIFIED, release.version).apply() + GearslipLog.i("update: showed ${release.version} on the car screen") + }, "gearslip-update").start() + } + + private fun foundRelease(app: Context): Release? = when (val s = flow.value) { + is State.Available -> s.release + is State.Downloading -> s.release + is State.Ready -> s.release + else -> savedRelease(app) + } + + /** Posts "Gearslip X is out"; tapping it opens the app, where the Home card has the button. */ + private fun notify(context: Context, release: Release): Boolean { + if (context.checkSelfPermission(Manifest.permission.POST_NOTIFICATIONS) != PackageManager.PERMISSION_GRANTED) return false + context.getSystemService(NotificationManager::class.java).createNotificationChannel( + NotificationChannel(CHANNEL, "Updates", NotificationManager.IMPORTANCE_DEFAULT), + ) + val open = PendingIntent.getActivity( + context, NOTIFICATION_ID, + Intent(context, GearslipActivity::class.java).addFlags(Intent.FLAG_ACTIVITY_NEW_TASK or Intent.FLAG_ACTIVITY_CLEAR_TOP), + PendingIntent.FLAG_IMMUTABLE, + ) + val notification = NotificationCompat.Builder(context, CHANNEL) + .setSmallIcon(android.R.drawable.stat_sys_download_done) + .setContentTitle("Gearslip ${release.version} is out") + .setContentText("Tap to update.") + .setContentIntent(open) + .setAutoCancel(true) + .build() + return runCatching { NotificationManagerCompat.from(context).notify(NOTIFICATION_ID, notification) } + .onSuccess { GearslipLog.i("update: told the driver about ${release.version}") } + .onFailure { GearslipLog.w("update: could not post the notification (${it.message})") } + .isSuccess + } + + /** Asks GitHub now. Ignored while a check or download is under way. */ + fun check(context: Context) { + val current = flow.value + if (current is State.Checking || current is State.Downloading) return + val app = context.applicationContext + flow.value = State.Checking + Thread({ flow.value = checkNow(app) }, "gearslip-update").start() + } + + private fun checkNow(app: Context): State = runCatching { fetchNewest(channel(app)) }.fold( + onSuccess = { release -> + prefs(app).edit().putLong(KEY_CHECKED_AT, System.currentTimeMillis()).apply() + if (release != null && isNewer(release.version, BuildConfig.VERSION_NAME)) { + GearslipLog.i("update: ${release.version} is out (this is ${BuildConfig.VERSION_NAME})") + save(app, release) + State.Available(release) + } else { + prefs(app).edit().remove(KEY_FOUND).apply() + updatesDir(app).deleteRecursively() + State.UpToDate + } + }, + onFailure = { + GearslipLog.w("update: could not check: ${it.message}") + State.Failed("Couldn't reach GitHub. Check the connection and try again.") + }, + ) + + /** Downloads [release], then opens the installer once the file checks out. */ + fun download(context: Context, release: Release) { + if (flow.value is State.Downloading) return + val app = context.applicationContext + flow.value = State.Downloading(release, 0f) + Thread({ + val dir = updatesDir(app).apply { deleteRecursively(); mkdirs() } + val apk = File(dir, "Gearslip-${release.version}.apk") + flow.value = runCatching { + fetchApk(release, apk) + GearslipLog.i("update: downloaded ${release.version}") + State.Ready(release, apk) + }.getOrElse { + apk.delete() + GearslipLog.w("update: download failed: ${it.message}") + State.Failed(it.message ?: "The download failed. Try again.", release) + } + if (flow.value is State.Ready) install(app) + }, "gearslip-update").start() + } + + /** + * Hands the downloaded APK to Android's installer. The first time, Android wants the driver to + * allow installs from Gearslip; that opens its settings page, and the install carries on when + * they come back. + */ + fun install(context: Context) { + val ready = flow.value as? State.Ready ?: return + // Installing closes Gearslip, which would drop the car screen mid-drive. + if (carConnected) return + val pm = context.packageManager + if (!pm.canRequestPackageInstalls()) { + installWhenAllowed = true + context.startActivity( + Intent(Settings.ACTION_MANAGE_UNKNOWN_APP_SOURCES, Uri.parse("package:${context.packageName}")) + .addFlags(Intent.FLAG_ACTIVITY_NEW_TASK), + ) + return + } + val uri = FileProvider.getUriForFile(context, "${context.packageName}.logshare", ready.apk) + context.startActivity( + Intent(Intent.ACTION_VIEW) + .setDataAndType(uri, "application/vnd.android.package-archive") + .addFlags(Intent.FLAG_GRANT_READ_URI_PERMISSION or Intent.FLAG_ACTIVITY_NEW_TASK), + ) + } + + /** The newest release with an APK that [channel] takes. Drafts never count. */ + private fun fetchNewest(channel: Channel): Release? { + val stable = channel == Channel.STABLE + val conn = (URL(if (stable) LATEST else ALL).openConnection() as HttpURLConnection).apply { + connectTimeout = 15_000 + readTimeout = 15_000 + setRequestProperty("Accept", "application/vnd.github+json") + setRequestProperty("User-Agent", "Gearslip/${BuildConfig.VERSION_NAME}") + } + try { + if (conn.responseCode == 404) return null // no stable release yet + if (conn.responseCode !in 200..299) error("GitHub answered ${conn.responseCode}") + val body = conn.inputStream.bufferedReader().use { it.readText() } + if (stable) return release(JSONObject(body)) + val list = org.json.JSONArray(body) + return (0 until list.length()).map { list.getJSONObject(it) } + .filterNot { it.optBoolean("draft") } + .filter { + channel == Channel.ALPHA || !it.optBoolean("prerelease") || + it.optString("tag_name").contains("beta", ignoreCase = true) + } + .mapNotNull(::release) + .reduceOrNull { best, next -> if (isNewer(next.version, best.version)) next else best } + } finally { + conn.disconnect() + } + } + + /** A release as GitHub lists it, or null when it has no APK attached. */ + private fun release(json: JSONObject): Release? { + val assets = json.optJSONArray("assets") ?: return null + val apk = (0 until assets.length()).map { assets.getJSONObject(it) } + .firstOrNull { it.optString("name").endsWith(".apk") && !it.optString("name").contains("debug") } + ?: return null + return Release( + version = json.getString("tag_name").removePrefix("v"), + apkUrl = apk.getString("browser_download_url"), + size = apk.optLong("size", -1), + sha256 = apk.optString("digest").takeIf { it.startsWith("sha256:") }?.removePrefix("sha256:"), + pageUrl = json.optString("html_url"), + ) + } + + private fun fetchApk(release: Release, into: File) { + val conn = (URL(release.apkUrl).openConnection() as HttpURLConnection).apply { + connectTimeout = 15_000 + readTimeout = 30_000 + setRequestProperty("User-Agent", "Gearslip/${BuildConfig.VERSION_NAME}") + } + try { + if (conn.responseCode !in 200..299) error("GitHub answered ${conn.responseCode}") + val total = release.size.takeIf { it > 0 } ?: conn.contentLengthLong + val digest = MessageDigest.getInstance("SHA-256") + var done = 0L + var shown = -1 + conn.inputStream.use { input -> + into.outputStream().use { output -> + val buffer = ByteArray(64 * 1024) + while (true) { + val n = input.read(buffer) + if (n < 0) break + output.write(buffer, 0, n) + digest.update(buffer, 0, n) + done += n + // Whole percents only, so the screen isn't redrawn for every buffer. + val percent = if (total > 0) (done * 100 / total).toInt() else 0 + if (percent != shown) { + shown = percent + flow.value = State.Downloading(release, percent / 100f) + } + } + } + } + if (release.size > 0 && done != release.size) error("The download stopped early. Try again.") + val hex = digest.digest().joinToString("") { "%02x".format(it) } + if (release.sha256 != null && !hex.equals(release.sha256, ignoreCase = true)) { + error("The download came out damaged. Try again.") + } + } finally { + conn.disconnect() + } + } + + /** Installing waits until the car is unplugged. */ + val carConnected: Boolean + get() = SessionStatus.state.value.phase.let { it == SessionStatus.Phase.CONNECTING || it == SessionStatus.Phase.PROJECTING } + + private fun updatesDir(context: Context) = File(context.cacheDir, "updates") + + private fun save(context: Context, release: Release) { + val json = JSONObject() + .put("version", release.version) + .put("apkUrl", release.apkUrl) + .put("size", release.size) + .put("sha256", release.sha256 ?: "") + .put("pageUrl", release.pageUrl) + prefs(context).edit().putString(KEY_FOUND, json.toString()).apply() + } + + /** The release found last time, if it is still newer than this build (it isn't after updating). */ + private fun savedRelease(context: Context): Release? { + val json = prefs(context).getString(KEY_FOUND, null)?.let { runCatching { JSONObject(it) }.getOrNull() } ?: return null + val release = Release( + version = json.optString("version"), + apkUrl = json.optString("apkUrl"), + size = json.optLong("size", -1), + sha256 = json.optString("sha256").ifBlank { null }, + pageUrl = json.optString("pageUrl"), + ) + return release.takeIf { isNewer(it.version, BuildConfig.VERSION_NAME) } + } + + /** + * "0.2.00-alpha" beats "0.1.10-alpha": the numbers before the dash compare one by one. With the + * same numbers, a plain release beats a beta, and a beta beats an alpha, so "0.3.0" reaches + * someone on "0.3.0-beta". + */ + internal fun isNewer(candidate: String, installed: String): Boolean { + fun parts(v: String) = v.removePrefix("v").substringBefore('-').split('.').map { it.toIntOrNull() ?: 0 } + fun stage(v: String): Int { + val tag = v.substringAfter('-', "").lowercase() + return when { + tag.isEmpty() -> 3 + "beta" in tag -> 2 + "alpha" in tag -> 1 + else -> 0 + } + } + val a = parts(candidate) + val b = parts(installed) + for (i in 0 until maxOf(a.size, b.size)) { + val x = a.getOrElse(i) { 0 } + val y = b.getOrElse(i) { 0 } + if (x != y) return x > y + } + return stage(candidate) > stage(installed) + } +} diff --git a/gearslip/src/main/java/app/seb3thehacker/gearslip/wireless/WirelessLink.kt b/gearslip/src/main/java/app/seb3thehacker/gearslip/wireless/WirelessLink.kt new file mode 100644 index 0000000..b734dca --- /dev/null +++ b/gearslip/src/main/java/app/seb3thehacker/gearslip/wireless/WirelessLink.kt @@ -0,0 +1,454 @@ +package app.seb3thehacker.gearslip.wireless + +import android.annotation.SuppressLint +import android.bluetooth.BluetoothDevice +import android.bluetooth.BluetoothManager +import android.bluetooth.BluetoothProfile +import android.bluetooth.BluetoothServerSocket +import android.bluetooth.BluetoothSocket +import android.content.Context +import android.net.ConnectivityManager +import android.net.MacAddress +import android.net.Network +import android.net.NetworkCapabilities +import android.net.NetworkRequest +import android.net.wifi.WifiNetworkSpecifier +import android.provider.Settings +import app.seb3thehacker.gearslip.GearslipLog +import app.seb3thehacker.gearslip.Protobuf +import app.seb3thehacker.gearslip.SessionStatus +import app.seb3thehacker.gearslip.Wire +import java.io.DataInputStream +import java.io.IOException +import java.io.OutputStream +import java.net.ConnectException +import java.net.InetSocketAddress +import java.net.SocketTimeoutException +import java.net.Socket +import java.util.UUID + +/** + * Wireless Android Auto, phone side: the start-up that hands Gearslip a socket instead of a USB + * accessory. Everything after it - versions, TLS, the certificate, channels, video - is the same + * protocol [app.seb3thehacker.gearslip.GearslipRunner] already speaks over USB. + * + * 1. Bluetooth: open RFCOMM to the car's Android Auto wireless service. The car must already be + * paired with the phone, as it usually is for calls. + * 2. Over that channel, framed as [u16 length][u16 message id][protobuf]: + * <- VersionRequest -> VersionResponse + * <- StartRequest {ip, port} -> InfoRequest + * <- InfoResponse {ssid, password, bssid, security} + * 3. Join the car's Wi-Fi as a local-only network, so the phone's internet stays on mobile data, + * and say so over Bluetooth before connecting: + * -> StartResponse {phone ip, status} -> ConnectionStatus {status} + * 4. TCP to {ip, port} over that network, and hand the socket to [onConnected]. + * + * The Bluetooth channel stays open for the whole drive: the car pings over it, and some cars end + * the session when it closes. Message layout from LIVI's head-unit side (livi-aa/src/wpp.rs) and + * aasdk's aaw protos. + */ +object WirelessLink { + + private val log = GearslipLog.tagged("AAW") + + /** The Android Auto wireless service every head unit publishes over Bluetooth. */ + val AA_UUID: UUID = UUID.fromString("4de17a00-52cb-11e6-bdf4-0800200c9a66") + + private const val MSG_START_REQUEST = 1 + private const val MSG_INFO_REQUEST = 2 + private const val MSG_INFO_RESPONSE = 3 + /** + * The wireless setup's status for success, shared by every message that carries one. + * aa-proxy-rs sends 0 to real cars and they carry on; a Renault Media Nav given 1 went quiet + * and closed the channel. (open-android-auto's table, which says 1, is marked unverified.) + */ + private const val STATUS_SUCCESS = 0L + private const val MSG_VERSION_REQUEST = 4 + private const val MSG_VERSION_RESPONSE = 5 + private const val MSG_CONNECTION_STATUS = 6 + private const val MSG_START_RESPONSE = 7 + private const val MSG_PING = 8 + private const val MSG_PONG = 9 + + private const val LISTEN_MS = 120_000 // time to reconnect the phone from the car's Bluetooth menu + private const val TCP_TRIES = 12 // about a minute: a car's projection listener can start well after its Wi-Fi + private const val WIFI_WAIT_MS = 60_000 + + /** Set by the activity: starts a session on the car's socket. */ + @Volatile var onConnected: ((Socket) -> Unit)? = null + + @Volatile private var rfcomm: BluetoothSocket? = null + @Volatile private var listener: BluetoothServerSocket? = null + @Volatile private var tcp: Socket? = null + @Volatile private var network: ConnectivityManager.NetworkCallback? = null + @Volatile private var worker: Thread? = null + /** Set when Android drops the car's network mid-try, so the rest don't bind to a dead one. */ + @Volatile private var lostNetwork: Network? = null + /** Bumped on each Wi-Fi join and on stop, so an older run of TCP tries ends instead of racing the new one. */ + @Volatile private var tcpRun = 0 + private val writeLock = Any() + + /** Paired Bluetooth devices, cars or not: the driver picks theirs. Needs BLUETOOTH_CONNECT. */ + @SuppressLint("MissingPermission") + fun pairedDevices(context: Context): List = + context.getSystemService(BluetoothManager::class.java)?.adapter?.bondedDevices + ?.sortedBy { it.name ?: it.address } + .orEmpty() + + @SuppressLint("MissingPermission") + fun name(device: BluetoothDevice): String = device.name ?: device.address + + fun start(context: Context, device: BluetoothDevice) { + stop(context) + val app = context.applicationContext + worker = Thread({ + // A test feature: anything unexpected ends the attempt, not the app. + try { + run(app, device) + } catch (e: Exception) { + log.e("wireless start-up failed", e) + SessionStatus.failed("Wireless start-up failed", "${e.javaClass.simpleName}: ${e.message}") + } + }, "gearslip-wireless").apply { isDaemon = true; start() } + } + + /** Closes the Bluetooth channel, the socket and the Wi-Fi request. Safe to call any time. */ + fun stop(context: Context) { + tcpRun++ + val starting = worker != null && SessionStatus.state.value.phase == SessionStatus.Phase.CONNECTING + worker?.interrupt() + worker = null + runCatching { listener?.close() } + listener = null + runCatching { rfcomm?.close() } + rfcomm = null + runCatching { tcp?.close() } + tcp = null + network?.let { cb -> + runCatching { context.getSystemService(ConnectivityManager::class.java).unregisterNetworkCallback(cb) } + } + network = null + // Stopped before the car's session began, so nothing else resets Home's card. + if (starting) { + log.i("wireless start-up stopped") + SessionStatus.disconnected() + } + } + + /** A stable id for this phone as this app sees it, standing in for the serial a real phone sends. */ + @SuppressLint("HardwareIds") + private fun deviceId(context: Context): String = + Settings.Secure.getString(context.contentResolver, Settings.Secure.ANDROID_ID) ?: "gearslip" + + @SuppressLint("MissingPermission") + private fun run(context: Context, device: BluetoothDevice) { + val name = name(device) + log.i("--- wireless start-up with $name (${device.address}) ---") + SessionStatus.connecting("Bluetooth: reaching $name") + val socket = try { + // A running scan slows the connect, but stopping one needs BLUETOOTH_SCAN on Android + // 12+, which Gearslip doesn't ask for. Gearslip never scans, so skipping it is fine. + runCatching { context.getSystemService(BluetoothManager::class.java)?.adapter?.cancelDiscovery() } + device.createRfcommSocketToServiceRecord(AA_UUID).also { it.connect() } + } catch (e: SecurityException) { + log.e("Bluetooth: permission refused while opening the channel on $name", e) + SessionStatus.failed( + "Bluetooth permission missing", + "Gearslip needs the Nearby devices permission to reach $name. Allow it in the app's " + + "settings, then try again.", + ) + return + } catch (e: IOException) { + // Most factory head units publish no service of their own: they look for the phone's + // and connect to it, so wait for the car instead. + log.i("Bluetooth: $name has no Android Auto service to call (${e.message}); waiting for it to call us") + waitForCar(context, name) ?: return + } + rfcomm = socket + log.i("Bluetooth: channel open") + SessionStatus.connecting("Bluetooth: talking to $name") + + val input = DataInputStream(socket.inputStream) + val output = socket.outputStream + var target: Pair? = null + try { + while (!Thread.currentThread().isInterrupted) { + val length = input.readUnsignedShort() + val id = input.readUnsignedShort() + val body = ByteArray(length).also { input.readFully(it) } + when (id) { + MSG_VERSION_REQUEST -> { + log.i("<- VersionRequest") + log.hex(" version", body) + // Laid out the way a real phone's reply is (aa-proxy-rs and LIVI both + // parse it from real phones): 1 and 2 the version, echoing the car's; + // 3 a serial; 4 the status; 6 the device's ids. + val f = Wire.fields(body) + val serial = deviceId(context) + val ids = Protobuf.stringField(1, serial) + Protobuf.stringField(2, UUID.randomUUID().toString()) + val reply = Protobuf.varintField(1, Wire.varint(f, 1) ?: 1) + + Protobuf.varintField(2, Wire.varint(f, 2) ?: 0) + + Protobuf.stringField(3, serial) + + Protobuf.varintField(4, STATUS_SUCCESS) + + Protobuf.varint(((6 shl 3) or 2).toLong()) + Protobuf.varint(ids.size.toLong()) + ids + send(output, MSG_VERSION_RESPONSE, reply) + log.i("-> VersionResponse") + log.hex(" reply", reply) + } + MSG_START_REQUEST -> { + val f = Wire.fields(body) + val ip = f.firstOrNull { it.number == 1 && it.wireType == 2 }?.bytes?.decodeToString().orEmpty() + val port = Wire.varint(f, 2)?.toInt() ?: 0 + log.i("<- StartRequest: projection at $ip:$port") + target = ip to port + send(output, MSG_INFO_REQUEST, ByteArray(0)) + log.i("-> InfoRequest") + } + MSG_INFO_RESPONSE -> { + val f = Wire.fields(body) + fun text(n: Int) = f.firstOrNull { it.number == n && it.wireType == 2 }?.bytes?.decodeToString().orEmpty() + val ssid = text(1) + val password = text(2) + val bssid = text(3) + val security = Wire.varint(f, 4) ?: -1 + log.i("<- InfoResponse: ssid=\"$ssid\" bssid=$bssid security=$security password=${if (password.isEmpty()) "none" else "(${password.length} chars)"}") + val (ip, port) = target ?: run { + log.w("InfoResponse before StartRequest: no address to connect to yet") + "" to 0 + } + joinWifi(context, output, name, ssid, password, bssid, ip, port) + } + MSG_PING -> send(output, MSG_PONG, body) + MSG_CONNECTION_STATUS, MSG_START_RESPONSE -> { + log.i("<- message $id: status=${Wire.varint(Wire.fields(body), if (id == MSG_START_RESPONSE) 3 else 1)}") + } + else -> { + log.i("<- unknown message $id (${body.size} bytes)") + log.hex(" msg", body) + } + } + } + } catch (e: IOException) { + if (rfcomm != null) log.w("Bluetooth: channel closed (${e.message})") + } + } + + /** + * Publishes the Android Auto service on the phone, the way Android Auto itself does, and waits + * for the car to connect to it. A car looks for the service when the phone connects over + * Bluetooth, so the driver may have to reconnect the phone from the car's menu. + */ + @SuppressLint("MissingPermission") + private fun waitForCar(context: Context, name: String): BluetoothSocket? { + val adapter = context.getSystemService(BluetoothManager::class.java)?.adapter ?: return null + val server = try { + adapter.listenUsingRfcommWithServiceRecord("Android Auto", AA_UUID) + } catch (e: Exception) { + log.e("Bluetooth: couldn't publish the Android Auto service", e) + SessionStatus.failed("Couldn't publish the Android Auto service", "${e.javaClass.simpleName}: ${e.message}") + return null + } + listener = server + SessionStatus.connecting("Waiting for $name. If nothing happens, disconnect the phone in the car's Bluetooth menu and connect it again") + log.i("Bluetooth: Android Auto service published, waiting ${LISTEN_MS / 1000}s for $name") + return try { + server.accept(LISTEN_MS).also { + val from = runCatching { it.remoteDevice.name }.getOrNull() ?: "a device" + log.i("Bluetooth: $from connected to the Android Auto service") + } + } catch (e: IOException) { + if (listener == null) return null // stopped + log.w("Bluetooth: no car connected within ${LISTEN_MS / 1000}s (${e.message})") + SessionStatus.failed( + "Car didn't connect over Bluetooth", + "$name didn't open Android Auto with the phone. Check that wireless Android Auto is " + + "on in the car's settings, then reconnect the phone from the car's Bluetooth menu.", + ) + null + } finally { + runCatching { server.close() } + listener = null + } + } + + private fun send(output: OutputStream, id: Int, body: ByteArray) { + val frame = ByteArray(4 + body.size) + frame[0] = (body.size shr 8).toByte(); frame[1] = body.size.toByte() + frame[2] = (id shr 8).toByte(); frame[3] = id.toByte() + body.copyInto(frame, 4) + synchronized(writeLock) { output.write(frame); output.flush() } + } + + /** + * Asks Android for the car's network. The first time, Android shows a "Connect to device" + * prompt on the phone; the driver has to accept it. + */ + private fun joinWifi( + context: Context, output: OutputStream, name: String, + ssid: String, password: String, bssid: String, ip: String, port: Int, + ) { + SessionStatus.connecting("Wi-Fi: joining $name's network") + val spec = WifiNetworkSpecifier.Builder().setSsid(ssid).apply { + runCatching { MacAddress.fromString(bssid) }.getOrNull() + ?.takeIf { it != MacAddress.fromString("00:00:00:00:00:00") } + ?.let { setBssid(it) } + if (password.isNotEmpty()) setWpa2Passphrase(password) + }.build() + val request = NetworkRequest.Builder() + .addTransportType(NetworkCapabilities.TRANSPORT_WIFI) + .removeCapability(NetworkCapabilities.NET_CAPABILITY_INTERNET) + .setNetworkSpecifier(spec) + .build() + val manager = context.getSystemService(ConnectivityManager::class.java) + val callback = object : ConnectivityManager.NetworkCallback() { + override fun onAvailable(net: Network) { + log.i("Wi-Fi: joined \"$ssid\"") + // Real phones report the join over Bluetooth before opening TCP, and some cars + // only start listening once they hear it. + val phoneIp = manager.getLinkProperties(net)?.linkAddresses + ?.firstOrNull { it.address is java.net.Inet4Address }?.address?.hostAddress.orEmpty() + runCatching { + send(output, MSG_START_RESPONSE, Protobuf.stringField(1, phoneIp) + Protobuf.varintField(3, STATUS_SUCCESS)) + send(output, MSG_CONNECTION_STATUS, Protobuf.varintField(1, STATUS_SUCCESS)) + log.i("-> StartResponse (phone at $phoneIp), ConnectionStatus: success") + }.onFailure { log.w("Bluetooth: couldn't report the Wi-Fi join (${it.message})") } + val run = ++tcpRun + Thread({ connectTcp(context, output, name, net, ip, port, run) }, "gearslip-wireless-tcp").start() + } + + override fun onUnavailable() { + log.w("Wi-Fi: couldn't join \"$ssid\" (declined, out of range, or wrong password)") + runCatching { send(output, MSG_CONNECTION_STATUS, Protobuf.varintField(1, -1)) } + SessionStatus.failed( + "Couldn't join the car's Wi-Fi", + "Android didn't connect to $name's network. If a \"Connect to device\" prompt " + + "appeared, it has to be accepted.", + ) + } + + // Android can hold an app's traffic back on a network it has joined (battery or data + // limits, local-network rules); the packets are then dropped without an error. + override fun onBlockedStatusChanged(net: Network, blocked: Boolean) { + log.i("Wi-Fi: Android ${if (blocked) "BLOCKS" else "allows"} Gearslip's traffic on \"$ssid\"") + } + + private var capsLogged = false + override fun onCapabilitiesChanged(net: Network, caps: NetworkCapabilities) { + if (capsLogged) return + capsLogged = true + log.i("Wi-Fi: \"$ssid\" capabilities $caps") + } + + override fun onLost(net: Network) { + lostNetwork = net + log.w( + "Wi-Fi: lost \"$ssid\" - Android dropped the car's network. Picking it by hand " + + "in the Wi-Fi list does this; the remaining tries go over the phone's own Wi-Fi", + ) + } + } + network?.let { runCatching { manager.unregisterNetworkCallback(it) } } + network = callback + manager.requestNetwork(request, callback, WIFI_WAIT_MS) + } + + /** + * Run once when the first TCP try fails, to tell "the car ignores us" from "wrong port". A + * refused port answers at once, so it proves the car is reachable; a timeout on every port + * means the car drops the phone's traffic, usually because it is still waiting for a step. + */ + @SuppressLint("MissingPermission") + private fun probe(context: Context, net: Network, ip: String, port: Int) { + runCatching { + val manager = context.getSystemService(ConnectivityManager::class.java) + val link = manager?.getLinkProperties(net) + log.i("probe: phone ${link?.linkAddresses?.joinToString()} on ${link?.interfaceName}, routes ${link?.routes?.joinToString()}") + val adapter = context.getSystemService(BluetoothManager::class.java)?.adapter + val calls = adapter?.getProfileConnectionState(BluetoothProfile.HEADSET) + val audio = adapter?.getProfileConnectionState(BluetoothProfile.A2DP) + log.i("probe: Bluetooth (any device) calls ${profileState(calls)}, media audio ${profileState(audio)}") + log.i( + "probe: background data ${manager?.restrictBackgroundStatus} (1 off, 2 allowed, 3 restricted), " + + "active network ${manager?.activeNetwork}, car network $net", + ) + }.onFailure { log.w("probe: couldn't read the link (${it.message})") } + for (p in listOf(port, 5288, 5277, 80, 443).distinct()) { + val result = try { + net.socketFactory.createSocket().use { it.connect(InetSocketAddress(ip, p), 1_500) } + "open" + } catch (e: SocketTimeoutException) { + "no answer" + } catch (e: ConnectException) { + if (e.message?.contains("ECONNREFUSED") == true) "refused (car is reachable)" else "failed: ${e.message}" + } catch (e: Exception) { + "failed: ${e.javaClass.simpleName} ${e.message}" + } + log.i("probe: $ip:$p $result") + } + } + + private fun profileState(state: Int?) = when (state) { + BluetoothProfile.STATE_CONNECTED -> "connected" + BluetoothProfile.STATE_CONNECTING -> "connecting" + BluetoothProfile.STATE_DISCONNECTED -> "not connected" + else -> "unknown" + } + + /** Another network the phone has with an address in the car's /24, or null. */ + private fun mainOnCarNetwork(context: Context, ip: String, except: Network): Network? { + val manager = context.getSystemService(ConnectivityManager::class.java) ?: return null + val prefix = ip.substringBeforeLast('.') + "." + return manager.allNetworks.firstOrNull { candidate -> + candidate != except && manager.getLinkProperties(candidate)?.linkAddresses + ?.any { it.address is java.net.Inet4Address && it.address.hostAddress?.startsWith(prefix) == true } == true + } + } + + private fun connectTcp(context: Context, output: OutputStream, name: String, net: Network, ip: String, port: Int, run: Int) { + SessionStatus.connecting("Wi-Fi: reaching $name") + lostNetwork = null + var last: Exception? = null + repeat(TCP_TRIES) { attempt -> + if (run != tcpRun) { + log.i("TCP: stopping the tries on the old network - the car started over") + return + } + try { + // A network Android has dropped refuses every bind (EPERM). If the driver joined the + // car's Wi-Fi by hand, the phone's ordinary route reaches it, so use that instead. + // The phone's main Wi-Fi on the car's network wins: that is how Android Auto itself + // connects, and a Pixel 6 passed pings but no TCP over the side connection Android + // opens for Gearslip's request. + val main = mainOnCarNetwork(context, ip, net) + if (main != null && attempt == 0) log.i("TCP: the phone's main Wi-Fi is on the car's network, using it") + val socket = when { + main != null -> main.socketFactory.createSocket() + lostNetwork == net -> Socket() + else -> net.socketFactory.createSocket() + } + socket.tcpNoDelay = true + socket.connect(InetSocketAddress(ip, port), 5_000) + tcp = socket + log.i("TCP: connected to $ip:$port (try ${attempt + 1})") + val start = onConnected + if (start == null) { + log.w("TCP: connected, but nothing is waiting for the socket") + socket.close() + } else { + start(socket) + } + return + } catch (e: Exception) { + last = e + log.w("TCP: try ${attempt + 1} to $ip:$port failed: ${e.message}") + if (attempt == 0) probe(context, net, ip, port) + Thread.sleep(1_000) + } + } + log.e("TCP: gave up on $ip:$port", last) + SessionStatus.failed( + "Car's Wi-Fi joined, but no connection", + "Gearslip joined $name's network but couldn't reach Android Auto at $ip:$port.", + ) + } +} diff --git a/gearslip/src/main/java/com/k2fsa/sherpa/onnx/FeatureConfig.kt b/gearslip/src/main/java/com/k2fsa/sherpa/onnx/FeatureConfig.kt new file mode 100644 index 0000000..44491c7 --- /dev/null +++ b/gearslip/src/main/java/com/k2fsa/sherpa/onnx/FeatureConfig.kt @@ -0,0 +1,12 @@ +// From sherpa-onnx v1.13.8 (Apache-2.0, https://github.com/k2-fsa/sherpa-onnx), kotlin-api. +package com.k2fsa.sherpa.onnx + +data class FeatureConfig( + var sampleRate: Int = 16000, + var featureDim: Int = 80, + var dither: Float = 0.0f +) + +fun getFeatureConfig(sampleRate: Int, featureDim: Int): FeatureConfig { + return FeatureConfig(sampleRate = sampleRate, featureDim = featureDim) +} diff --git a/gearslip/src/main/java/com/k2fsa/sherpa/onnx/HomophoneReplacerConfig.kt b/gearslip/src/main/java/com/k2fsa/sherpa/onnx/HomophoneReplacerConfig.kt new file mode 100644 index 0000000..d282f0d --- /dev/null +++ b/gearslip/src/main/java/com/k2fsa/sherpa/onnx/HomophoneReplacerConfig.kt @@ -0,0 +1,8 @@ +// From sherpa-onnx v1.13.8 (Apache-2.0, https://github.com/k2-fsa/sherpa-onnx), kotlin-api. +package com.k2fsa.sherpa.onnx + +data class HomophoneReplacerConfig( + var dictDir: String = "", // unused + var lexicon: String = "", + var ruleFsts: String = "", +) diff --git a/gearslip/src/main/java/com/k2fsa/sherpa/onnx/OfflineRecognizer.kt b/gearslip/src/main/java/com/k2fsa/sherpa/onnx/OfflineRecognizer.kt new file mode 100644 index 0000000..ac229ac --- /dev/null +++ b/gearslip/src/main/java/com/k2fsa/sherpa/onnx/OfflineRecognizer.kt @@ -0,0 +1,255 @@ +// From sherpa-onnx v1.13.8 (Apache-2.0, https://github.com/k2-fsa/sherpa-onnx), kotlin-api. +// The JNI names are fixed to this package; the upstream catalog of pretrained models is left out. +package com.k2fsa.sherpa.onnx + +import android.content.res.AssetManager + +data class OfflineRecognizerResult( + val text: String, + val tokens: Array, + val timestamps: FloatArray, + val lang: String, + val emotion: String, + val event: String, + + // valid only for TDT models + val durations: FloatArray, +) + +data class OfflineTransducerModelConfig( + var encoder: String = "", + var decoder: String = "", + var joiner: String = "", + var qnnConfig: QnnConfig = QnnConfig(), +) + +data class OfflineParaformerModelConfig( + var model: String = "", + var qnnConfig: QnnConfig = QnnConfig(), +) + +data class OfflineNemoEncDecCtcModelConfig( + var model: String = "", +) + +data class OfflineDolphinModelConfig( + var model: String = "", +) + +data class OfflineZipformerCtcModelConfig( + var model: String = "", + var qnnConfig: QnnConfig = QnnConfig(), +) + +data class OfflineWenetCtcModelConfig( + var model: String = "", +) + +data class OfflineOmnilingualAsrCtcModelConfig( + var model: String = "", +) + +data class OfflineMedAsrCtcModelConfig( + var model: String = "", +) + +data class OfflineFireRedAsrCtcModelConfig( + var model: String = "", +) + +data class OfflineFunAsrNanoModelConfig( + var encoderAdaptor: String = "", + var llm: String = "", + var embedding: String = "", + var tokenizer: String = "", + var systemPrompt: String = "You are a helpful assistant.", + var userPrompt: String = "语音转写：", + var maxNewTokens: Int = 512, + var temperature: Float = 1e-6f, + var topP: Float = 0.8f, + var seed: Int = 42, + var language: String = "", + var itn: Boolean = true, + var hotwords: String = "", +) + +data class OfflineQwen3AsrModelConfig( + var convFrontend: String = "", + var encoder: String = "", + var decoder: String = "", + var tokenizer: String = "", + var maxTotalLen: Int = 512, + var maxNewTokens: Int = 128, + var temperature: Float = 1e-6f, + var topP: Float = 0.8f, + var seed: Int = 42, + var hotwords: String = "", +) + +data class OfflineWhisperModelConfig( + var encoder: String = "", + var decoder: String = "", + var language: String = "en", // Used with multilingual model + var task: String = "transcribe", // transcribe or translate + var tailPaddings: Int = 1000, // Padding added at the end of the samples + var enableTokenTimestamps: Boolean = false, + var enableSegmentTimestamps: Boolean = false, +) + +data class OfflineCanaryModelConfig( + var encoder: String = "", + var decoder: String = "", + var srcLang: String = "en", + var tgtLang: String = "en", + var usePnc: Boolean = true, +) + +data class OfflineCohereTranscribeModelConfig( + var encoder: String = "", + var decoder: String = "", + var language: String = "", + var usePunct: Boolean = true, + var useItn: Boolean = true, +) + +data class OfflineFireRedAsrModelConfig( + var encoder: String = "", + var decoder: String = "", +) + +// For moonshine v1, you need four models. +// For moonshine v2, you need two models. +// - v1: preprocessor, encoder, uncachedDecoder, cachedDecoder +// - v2: encoder, mergedDecoder +data class OfflineMoonshineModelConfig( + var preprocessor: String = "", + var encoder: String = "", + var uncachedDecoder: String = "", + var cachedDecoder: String = "", + var mergedDecoder: String = "", +) + +data class OfflineSenseVoiceModelConfig( + var model: String = "", + var language: String = "", + var useInverseTextNormalization: Boolean = true, + var qnnConfig: QnnConfig = QnnConfig(), +) + +data class OfflineModelConfig( + var transducer: OfflineTransducerModelConfig = OfflineTransducerModelConfig(), + var paraformer: OfflineParaformerModelConfig = OfflineParaformerModelConfig(), + var whisper: OfflineWhisperModelConfig = OfflineWhisperModelConfig(), + var fireRedAsr: OfflineFireRedAsrModelConfig = OfflineFireRedAsrModelConfig(), + var moonshine: OfflineMoonshineModelConfig = OfflineMoonshineModelConfig(), + var nemo: OfflineNemoEncDecCtcModelConfig = OfflineNemoEncDecCtcModelConfig(), + var senseVoice: OfflineSenseVoiceModelConfig = OfflineSenseVoiceModelConfig(), + var dolphin: OfflineDolphinModelConfig = OfflineDolphinModelConfig(), + var zipformerCtc: OfflineZipformerCtcModelConfig = OfflineZipformerCtcModelConfig(), + var wenetCtc: OfflineWenetCtcModelConfig = OfflineWenetCtcModelConfig(), + var omnilingual: OfflineOmnilingualAsrCtcModelConfig = OfflineOmnilingualAsrCtcModelConfig(), + var medasr: OfflineMedAsrCtcModelConfig = OfflineMedAsrCtcModelConfig(), + var funasrNano: OfflineFunAsrNanoModelConfig = OfflineFunAsrNanoModelConfig(), + var qwen3Asr: OfflineQwen3AsrModelConfig = OfflineQwen3AsrModelConfig(), + var fireRedAsrCtc: OfflineFireRedAsrCtcModelConfig = OfflineFireRedAsrCtcModelConfig(), + var canary: OfflineCanaryModelConfig = OfflineCanaryModelConfig(), + var cohereTranscribe: OfflineCohereTranscribeModelConfig = + OfflineCohereTranscribeModelConfig(), + var teleSpeech: String = "", + var numThreads: Int = 1, + var debug: Boolean = false, + var provider: String = "cpu", + var modelType: String = "", + var tokens: String = "", + var modelingUnit: String = "", + var bpeVocab: String = "", +) + +data class OfflineRecognizerConfig( + var featConfig: FeatureConfig = FeatureConfig(), + var modelConfig: OfflineModelConfig = OfflineModelConfig(), + // var lmConfig: OfflineLMConfig(), // TODO(fangjun): enable it + var hr: HomophoneReplacerConfig = HomophoneReplacerConfig(), + var decodingMethod: String = "greedy_search", + var maxActivePaths: Int = 4, + var hotwordsFile: String = "", + var hotwordsScore: Float = 1.5f, + var ruleFsts: String = "", + var ruleFars: String = "", + var blankPenalty: Float = 0.0f, +) + +class OfflineRecognizer( + assetManager: AssetManager? = null, + val config: OfflineRecognizerConfig, +) { + private var ptr: Long + + init { + ptr = if (assetManager != null) { + newFromAsset(assetManager, config) + } else { + newFromFile(config) + } + require(ptr != 0L) { + "Invalid OfflineRecognizerConfig: failed to create native OfflineRecognizer" + } + } + + protected fun finalize() { + if (ptr != 0L) { + delete(ptr) + ptr = 0 + } + } + + fun release() = finalize() + + fun createStream(): OfflineStream { + val p = createStream(ptr) + return OfflineStream(p) + } + + fun createStream(hotwords: String): OfflineStream { + val p = createStreamWithHotwords(ptr, hotwords) + return OfflineStream(p) + } + + fun getResult(stream: OfflineStream): OfflineRecognizerResult { + return getResult(stream.ptr) + } + + fun decode(stream: OfflineStream) = decode(ptr, stream.ptr) + + fun setConfig(config: OfflineRecognizerConfig) = setConfig(ptr, config) + + private external fun delete(ptr: Long) + + private external fun createStream(ptr: Long): Long + + private external fun createStreamWithHotwords(ptr: Long, hotwords: String): Long + + private external fun setConfig(ptr: Long, config: OfflineRecognizerConfig) + + private external fun newFromAsset( + assetManager: AssetManager, + config: OfflineRecognizerConfig, + ): Long + + private external fun newFromFile( + config: OfflineRecognizerConfig, + ): Long + + private external fun decode(ptr: Long, streamPtr: Long) + + private external fun getResult(streamPtr: Long): OfflineRecognizerResult + + companion object { + init { + System.loadLibrary("sherpa-onnx-jni") + } + + @JvmStatic + external fun prependAdspLibraryPath(newPath: String) // for qnn + } +} diff --git a/gearslip/src/main/java/com/k2fsa/sherpa/onnx/OfflineStream.kt b/gearslip/src/main/java/com/k2fsa/sherpa/onnx/OfflineStream.kt new file mode 100644 index 0000000..12ac836 --- /dev/null +++ b/gearslip/src/main/java/com/k2fsa/sherpa/onnx/OfflineStream.kt @@ -0,0 +1,43 @@ +// From sherpa-onnx v1.13.8 (Apache-2.0, https://github.com/k2-fsa/sherpa-onnx), kotlin-api. +package com.k2fsa.sherpa.onnx + +class OfflineStream(var ptr: Long) { + init { + require(ptr != 0L) { "Failed to create native OfflineStream" } + } + + fun acceptWaveform(samples: FloatArray, sampleRate: Int) = + acceptWaveform(ptr, samples, sampleRate) + + fun setOption(key: String, value: String) = setOption(ptr, key, value) + + fun getOption(key: String): String = getOption(ptr, key) + + protected fun finalize() { + if (ptr != 0L) { + delete(ptr) + ptr = 0 + } + } + + fun release() = finalize() + + fun use(block: (OfflineStream) -> Unit) { + try { + block(this) + } finally { + release() + } + } + + private external fun acceptWaveform(ptr: Long, samples: FloatArray, sampleRate: Int) + private external fun setOption(ptr: Long, key: String, value: String) + private external fun getOption(ptr: Long, key: String): String + private external fun delete(ptr: Long) + + companion object { + init { + System.loadLibrary("sherpa-onnx-jni") + } + } +} diff --git a/gearslip/src/main/java/com/k2fsa/sherpa/onnx/OnlineRecognizer.kt b/gearslip/src/main/java/com/k2fsa/sherpa/onnx/OnlineRecognizer.kt new file mode 100644 index 0000000..3cd4a16 --- /dev/null +++ b/gearslip/src/main/java/com/k2fsa/sherpa/onnx/OnlineRecognizer.kt @@ -0,0 +1,158 @@ +// From sherpa-onnx v1.13.8 (Apache-2.0, https://github.com/k2-fsa/sherpa-onnx), kotlin-api. +// The JNI names are fixed to this package; the upstream catalog of pretrained models is left out. +package com.k2fsa.sherpa.onnx + +import android.content.res.AssetManager + +data class EndpointRule( + var mustContainNonSilence: Boolean, + var minTrailingSilence: Float, + var minUtteranceLength: Float, +) + +data class EndpointConfig( + var rule1: EndpointRule = EndpointRule(false, 2.4f, 0.0f), + var rule2: EndpointRule = EndpointRule(true, 1.4f, 0.0f), + var rule3: EndpointRule = EndpointRule(false, 0.0f, 20.0f) +) + +data class OnlineTransducerModelConfig( + var encoder: String = "", + var decoder: String = "", + var joiner: String = "", + var qnnConfig: QnnConfig = QnnConfig(), +) + +data class OnlineParaformerModelConfig( + var encoder: String = "", + var decoder: String = "", +) + +data class OnlineZipformer2CtcModelConfig( + var model: String = "", +) + +data class OnlineNeMoCtcModelConfig( + var model: String = "", +) + +data class OnlineToneCtcModelConfig( + var model: String = "", +) + +data class OnlineModelConfig( + var transducer: OnlineTransducerModelConfig = OnlineTransducerModelConfig(), + var paraformer: OnlineParaformerModelConfig = OnlineParaformerModelConfig(), + var zipformer2Ctc: OnlineZipformer2CtcModelConfig = OnlineZipformer2CtcModelConfig(), + var neMoCtc: OnlineNeMoCtcModelConfig = OnlineNeMoCtcModelConfig(), + var toneCtc: OnlineToneCtcModelConfig = OnlineToneCtcModelConfig(), + var tokens: String = "", + var numThreads: Int = 1, + var debug: Boolean = false, + var provider: String = "cpu", + var modelType: String = "", + var modelingUnit: String = "", + var bpeVocab: String = "", +) + +data class OnlineLMConfig( + var model: String = "", + var scale: Float = 0.5f, +) + +data class OnlineCtcFstDecoderConfig( + var graph: String = "", + var maxActive: Int = 3000, +) + +data class OnlineRecognizerConfig( + var featConfig: FeatureConfig = FeatureConfig(), + var modelConfig: OnlineModelConfig = OnlineModelConfig(), + var lmConfig: OnlineLMConfig = OnlineLMConfig(), + var ctcFstDecoderConfig: OnlineCtcFstDecoderConfig = OnlineCtcFstDecoderConfig(), + var hr: HomophoneReplacerConfig = HomophoneReplacerConfig(), + var endpointConfig: EndpointConfig = EndpointConfig(), + var enableEndpoint: Boolean = true, + var decodingMethod: String = "greedy_search", + var maxActivePaths: Int = 4, + var hotwordsFile: String = "", + var hotwordsScore: Float = 1.5f, + var ruleFsts: String = "", + var ruleFars: String = "", + var blankPenalty: Float = 0.0f, +) + +data class OnlineRecognizerResult( + val text: String, + val tokens: Array, + val timestamps: FloatArray, + val ysProbs: FloatArray, + // TODO(fangjun): Add more fields +) + +class OnlineRecognizer( + assetManager: AssetManager? = null, + val config: OnlineRecognizerConfig, +) { + private var ptr: Long + + init { + ptr = if (assetManager != null) { + newFromAsset(assetManager, config) + } else { + newFromFile(config) + } + require(ptr != 0L) { + "Invalid OnlineRecognizerConfig: failed to create native OnlineRecognizer" + } + } + + protected fun finalize() { + if (ptr != 0L) { + delete(ptr) + ptr = 0 + } + } + + fun release() = finalize() + + fun createStream(hotwords: String = ""): OnlineStream { + val p = createStream(ptr, hotwords) + return OnlineStream(p) + } + + fun reset(stream: OnlineStream) = reset(ptr, stream.ptr) + fun decode(stream: OnlineStream) = decode(ptr, stream.ptr) + fun isEndpoint(stream: OnlineStream) = isEndpoint(ptr, stream.ptr) + fun isReady(stream: OnlineStream) = isReady(ptr, stream.ptr) + fun getResult(stream: OnlineStream): OnlineRecognizerResult { + return getResult(ptr, stream.ptr) + } + + private external fun delete(ptr: Long) + + private external fun newFromAsset( + assetManager: AssetManager, + config: OnlineRecognizerConfig, + ): Long + + private external fun newFromFile( + config: OnlineRecognizerConfig, + ): Long + + private external fun createStream(ptr: Long, hotwords: String): Long + private external fun reset(ptr: Long, streamPtr: Long) + private external fun decode(ptr: Long, streamPtr: Long) + private external fun isEndpoint(ptr: Long, streamPtr: Long): Boolean + private external fun isReady(ptr: Long, streamPtr: Long): Boolean + private external fun getResult(ptr: Long, streamPtr: Long): OnlineRecognizerResult + + companion object { + init { + System.loadLibrary("sherpa-onnx-jni") + } + + @JvmStatic + external fun prependAdspLibraryPath(newPath: String) // for qnn + } +} diff --git a/gearslip/src/main/java/com/k2fsa/sherpa/onnx/OnlineStream.kt b/gearslip/src/main/java/com/k2fsa/sherpa/onnx/OnlineStream.kt new file mode 100644 index 0000000..6eebf5e --- /dev/null +++ b/gearslip/src/main/java/com/k2fsa/sherpa/onnx/OnlineStream.kt @@ -0,0 +1,47 @@ +// From sherpa-onnx v1.13.8 (Apache-2.0, https://github.com/k2-fsa/sherpa-onnx), kotlin-api. +package com.k2fsa.sherpa.onnx + +class OnlineStream(var ptr: Long = 0) { + init { + require(ptr != 0L) { "Failed to create native OnlineStream" } + } + + fun acceptWaveform(samples: FloatArray, sampleRate: Int) = + acceptWaveform(ptr, samples, sampleRate) + + fun inputFinished() = inputFinished(ptr) + + fun setOption(key: String, value: String) = setOption(ptr, key, value) + + fun getOption(key: String): String = getOption(ptr, key) + + protected fun finalize() { + if (ptr != 0L) { + delete(ptr) + ptr = 0 + } + } + + fun release() = finalize() + + fun use(block: (OnlineStream) -> Unit) { + try { + block(this) + } finally { + release() + } + } + + private external fun acceptWaveform(ptr: Long, samples: FloatArray, sampleRate: Int) + private external fun inputFinished(ptr: Long) + private external fun setOption(ptr: Long, key: String, value: String) + private external fun getOption(ptr: Long, key: String): String + private external fun delete(ptr: Long) + + + companion object { + init { + System.loadLibrary("sherpa-onnx-jni") + } + } +} diff --git a/gearslip/src/main/java/com/k2fsa/sherpa/onnx/QnnConfig.kt b/gearslip/src/main/java/com/k2fsa/sherpa/onnx/QnnConfig.kt new file mode 100644 index 0000000..95be41c --- /dev/null +++ b/gearslip/src/main/java/com/k2fsa/sherpa/onnx/QnnConfig.kt @@ -0,0 +1,8 @@ +// From sherpa-onnx v1.13.8 (Apache-2.0, https://github.com/k2-fsa/sherpa-onnx), kotlin-api. +package com.k2fsa.sherpa.onnx + +data class QnnConfig( + var backendLib: String = "", + var contextBinary: String = "", + var systemLib: String = "", +) diff --git a/gearslip/src/main/assets/speech/tiny_acft_q5_1.bin b/gearslip/src/main/jniLibs/arm64-v8a/libsherpa-onnx-jni.so similarity index 59% rename from gearslip/src/main/assets/speech/tiny_acft_q5_1.bin rename to gearslip/src/main/jniLibs/arm64-v8a/libsherpa-onnx-jni.so index 541f1df..b2f18d1 100644 Binary files a/gearslip/src/main/assets/speech/tiny_acft_q5_1.bin and b/gearslip/src/main/jniLibs/arm64-v8a/libsherpa-onnx-jni.so differ diff --git a/gearslip/src/main/res/drawable/ic_launcher_monochrome.xml b/gearslip/src/main/res/drawable/ic_launcher_monochrome.xml index e173db8..d6da12a 100644 --- a/gearslip/src/main/res/drawable/ic_launcher_monochrome.xml +++ b/gearslip/src/main/res/drawable/ic_launcher_monochrome.xml @@ -9,12 +9,10 @@ android:fillColor="#FF000000" android:fillAlpha="0.35"/> - diff --git a/gearslip/src/main/res/xml/share_paths.xml b/gearslip/src/main/res/xml/share_paths.xml index 9d97a8c..4307a1f 100644 --- a/gearslip/src/main/res/xml/share_paths.xml +++ b/gearslip/src/main/res/xml/share_paths.xml @@ -1,5 +1,6 @@ - + + diff --git a/gearslip/src/test/java/app/seb3thehacker/gearslip/ConnectionLifecycleTest.kt b/gearslip/src/test/java/app/seb3thehacker/gearslip/ConnectionLifecycleTest.kt new file mode 100644 index 0000000..2966b2e --- /dev/null +++ b/gearslip/src/test/java/app/seb3thehacker/gearslip/ConnectionLifecycleTest.kt @@ -0,0 +1,136 @@ +package app.seb3thehacker.gearslip + +import java.io.ByteArrayInputStream +import java.io.ByteArrayOutputStream +import java.io.IOException +import java.io.InputStream +import java.nio.ByteBuffer +import java.util.concurrent.CountDownLatch +import java.util.concurrent.ExecutorService +import java.util.concurrent.TimeUnit +import javax.net.ssl.SSLContext +import org.junit.Assert.assertEquals +import org.junit.Assert.assertTrue +import org.junit.Test + +class ConnectionLifecycleTest { + private val version = message(1, byteArrayOf(0, 1, 0, 3)) + + private fun message(id: Int, body: ByteArray = byteArrayOf()) = Frames.build( + Frames.CHANNEL_CONTROL, false, byteArrayOf(0, id.toByte()) + body, + ) + + private fun hello(): ByteArray { + val client = SSLContext.getDefault().createSSLEngine().apply { + useClientMode = true + enabledProtocols = arrayOf("TLSv1.2") + beginHandshake() + } + val packet = ByteBuffer.allocate(client.session.packetBufferSize) + client.wrap(ByteBuffer.allocate(0), packet) + return message(3, packet.array().copyOf(packet.position())) + } + + private fun identity(): CertProvider.Identity { + val generated = SelfSignedCert.generate() + return CertProvider.Identity( + generated.keyStore, generated.certificate, SelfSignedCert.PASSWORD, "generated test identity", + ) + } + + private fun runner(input: InputStream) = GearslipRunner(input, ByteArrayOutputStream(), ::identity) + + @Test fun `rejected authentication ignores subsequent success and service discovery in the same read`() { + for (status in listOf(-2, -3)) { + runner(ByteArrayInputStream( + version + hello() + message(4, Protobuf.varintField(1, status.toLong())) + + message(4, Protobuf.varintField(1, 0)) + message(6) + version, + )).run() + assertEquals( + if (status == -2) SessionReport.Category.CERTIFICATE else SessionReport.Category.AUTH, + SessionReport.category, + ) + assertEquals(SessionStatus.Phase.FAILED, SessionStatus.state.value.phase) + assertEquals("Head unit rejected the certificate", SessionStatus.state.value.headline) + assertTrue(SessionReport.stage < SessionReport.Stage.ACCEPTED) + } + } + + @Test fun `TLS failure ignores later protocol messages`() { + val fatalAlert = message(3, byteArrayOf(21, 3, 3, 0, 2, 2, 42)) + runner(ByteArrayInputStream(version + hello() + fatalAlert + version + message(6))).run() + assertEquals(SessionReport.Category.TLS, SessionReport.category) + assertEquals("Head unit rejected the certificate", SessionStatus.state.value.headline) + assertTrue(SessionReport.stage < SessionReport.Stage.ACCEPTED) + } + + @Test fun `authentication success before TLS completes is ignored`() { + runner(ByteArrayInputStream(version + message(4, Protobuf.varintField(1, 0)))).run() + assertEquals(SessionReport.Category.TLS, SessionReport.category) + assertEquals(SessionReport.Stage.VERSIONS, SessionReport.stage) + } + + @Test fun `stop followed by EOF does not report a handshake failure`() { + stopDuringRead { -1 } + } + + @Test fun `stop followed by a read exception does not report a USB failure`() { + stopDuringRead { throw IOException("transport closed by owner") } + } + + @Test fun `stop ignores bytes returned by an in-flight read`() { + stopDuringRead { bytes -> version.copyInto(bytes); version.size } + } + + private fun stopDuringRead(result: (ByteArray) -> Int) { + lateinit var runner: GearslipRunner + val input = object : InputStream() { + override fun read(): Int = error("bulk reads only") + override fun read(bytes: ByteArray, offset: Int, length: Int): Int { + runner.stop() + return result(bytes) + } + } + runner = runner(input) + runner.run() + assertEquals(SessionReport.Category.NONE, SessionReport.category) + assertEquals(SessionStatus.Phase.DISCONNECTED, SessionStatus.state.value.phase) + } + + @Test fun `stop while preparing TLS leaves the session disconnected`() { + val releaseIdentity = CountDownLatch(1) + val runner = GearslipRunner(ByteArrayInputStream(version + hello()), ByteArrayOutputStream(), { + check(releaseIdentity.await(5, TimeUnit.SECONDS)) + identity() + }) + val reader = Thread { runner.run() } + reader.start() + try { + // The reader waits on FutureTask.get while the identity provider is held above. + val deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(5) + while (reader.state != Thread.State.WAITING && reader.isAlive && System.nanoTime() < deadline) { + reader.join(1) + } + assertEquals(Thread.State.WAITING, reader.state) + runner.stop() + } finally { + releaseIdentity.countDown() + reader.join(5_000) + } + assertTrue("reader must finish", !reader.isAlive) + assertEquals(SessionReport.Category.NONE, SessionReport.category) + assertEquals(SessionStatus.Phase.DISCONNECTED, SessionStatus.state.value.phase) + } + + @Test fun `EOF and transport failure shut down the writer`() { + val failing = object : InputStream() { + override fun read(): Int = throw IOException("test disconnect") + } + for (input in listOf(ByteArrayInputStream(version), failing)) { + val runner = runner(input) + runner.run() + val field = GearslipRunner::class.java.getDeclaredField("writer").apply { isAccessible = true } + assertTrue((field.get(runner) as ExecutorService).isShutdown) + } + } +} diff --git a/gearslip/src/test/java/app/seb3thehacker/gearslip/car/CarToastsTest.kt b/gearslip/src/test/java/app/seb3thehacker/gearslip/car/CarToastsTest.kt new file mode 100644 index 0000000..05c8c90 --- /dev/null +++ b/gearslip/src/test/java/app/seb3thehacker/gearslip/car/CarToastsTest.kt @@ -0,0 +1,109 @@ +package app.seb3thehacker.gearslip.car + +import androidx.car.app.CarToast +import org.junit.After +import org.junit.Assert.assertEquals +import org.junit.Assert.assertNotSame +import org.junit.Assert.assertNull +import org.junit.Assert.assertSame +import org.junit.Before +import org.junit.Test + +class CarToastsTest { + @Before + @After + fun clearToast() { + while (true) CarToasts.message.value?.let(CarToasts::dismiss) ?: break + } + + @Test + fun aNewToastWaitsForTheOneOnScreen() { + val owner = Any() + CarToasts.show(owner, "First", CarToast.LENGTH_SHORT) + val first = requireNotNull(CarToasts.message.value) + CarToasts.show(owner, "Second", CarToast.LENGTH_LONG) + + assertSame(first, CarToasts.message.value) + assertEquals(1, CarToasts.waiting.value) + CarToasts.dismiss(first) + assertEquals("Second", CarToasts.message.value?.text) + assertEquals(0, CarToasts.waiting.value) + } + + @Test + fun expiredTimerCannotDismissItsReplacement() { + val owner = Any() + CarToasts.show(owner, "First", CarToast.LENGTH_SHORT) + val first = requireNotNull(CarToasts.message.value) + CarToasts.show(owner, "Second", CarToast.LENGTH_LONG) + CarToasts.dismiss(first) + val second = requireNotNull(CarToasts.message.value) + + CarToasts.dismiss(first) + assertSame(second, CarToasts.message.value) + CarToasts.dismiss(second) + assertNull(CarToasts.message.value) + } + + @Test + fun identicalMessagesStillRestartTheDisplayTimer() { + val owner = Any() + CarToasts.show(owner, "Saved", CarToast.LENGTH_SHORT) + val first = requireNotNull(CarToasts.message.value) + CarToasts.show(owner, "Saved", CarToast.LENGTH_SHORT) + val repeated = requireNotNull(CarToasts.message.value) + + assertNotSame(first, repeated) + assertEquals(0, CarToasts.waiting.value) + CarToasts.dismiss(first) + assertSame(repeated, CarToasts.message.value) + } + + @Test + fun disconnectOnlyClearsItsOwnToast() { + val nav = Any() + val browse = Any() + CarToasts.show(nav, "Route saved", CarToast.LENGTH_SHORT) + CarToasts.show(browse, "Added to playlist", CarToast.LENGTH_SHORT) + + CarToasts.clear(nav) + assertEquals("Added to playlist", CarToasts.message.value?.text) + CarToasts.clear(browse) + assertNull(CarToasts.message.value) + } + + @Test + fun aBurstKeepsOnlyTheNewestFewWaiting() { + val owner = Any() + listOf("1", "2", "3", "4", "5", "6").forEach { CarToasts.show(owner, it, CarToast.LENGTH_SHORT) } + + assertEquals("1", CarToasts.message.value?.text) + assertEquals(3, CarToasts.waiting.value) + CarToasts.message.value?.let(CarToasts::dismiss) + assertEquals("4", CarToasts.message.value?.text) + } + + @Test + fun blankRequestsDoNotReplaceAVisibleMessage() { + val owner = Any() + CarToasts.show(owner, "Saved", CarToast.LENGTH_SHORT) + val visible = CarToasts.message.value + + CarToasts.show(owner, " \n\t", CarToast.LENGTH_LONG) + assertSame(visible, CarToasts.message.value) + assertEquals(0, CarToasts.waiting.value) + } + + @Test + fun unknownDurationsFallBackToShortRatherThanStayingVisible() { + val owner = Any() + CarToasts.show(owner, "Short", CarToast.LENGTH_SHORT) + assertEquals(4_000L, CarToasts.message.value?.durationMs) + CarToasts.message.value?.let(CarToasts::dismiss) + CarToasts.show(owner, "Long", CarToast.LENGTH_LONG) + assertEquals(7_000L, CarToasts.message.value?.durationMs) + CarToasts.message.value?.let(CarToasts::dismiss) + CarToasts.show(owner, "Unknown", Int.MAX_VALUE) + assertEquals(4_000L, CarToasts.message.value?.durationMs) + } +} diff --git a/gradle/libs.versions.toml b/gradle/libs.versions.toml index f967a0d..0075398 100644 --- a/gradle/libs.versions.toml +++ b/gradle/libs.versions.toml @@ -14,6 +14,7 @@ material = "1.14.0" constraintlayout = "2.2.1" zxing = "3.5.4" okhttp = "5.4.0" +commonsCompress = "1.28.0" kotlinxSerializationJson = "1.11.0" ossLicenses = "17.5.1" webkit = "1.16.0" @@ -49,6 +50,7 @@ androidx-constraintlayout = { group = "androidx.constraintlayout", name = "const google-material = { group = "com.google.android.material", name = "material", version.ref = "material" } zxing-core = { group = "com.google.zxing", name = "core", version.ref = "zxing" } okhttp = { group = "com.squareup.okhttp3", name = "okhttp", version.ref = "okhttp" } +commons-compress = { group = "org.apache.commons", name = "commons-compress", version.ref = "commonsCompress" } kotlinx-serialization-json = { group = "org.jetbrains.kotlinx", name = "kotlinx-serialization-json", version.ref = "kotlinxSerializationJson" } google-oss-licenses = { group = "com.google.android.gms", name = "play-services-oss-licenses", version.ref = "ossLicenses" } androidx-webkit = { group = "androidx.webkit", name = "webkit", version.ref = "webkit" }