Marketplace Finish: repair sets whose marker says complete but files are missing
"2 chosen maps are only partly downloaded (Population density, Yandex satellite imagery). Finish" - and Finish did nothing, silently. Root cause: the banner and the button asked two different questions. market::unfinished counts a set as unfinished when check_installed's file count found it short (launcher.log: "popdark says it is installed but holds 293056 of 400268 files", "yandex ... 2371 of 28213"). Finish went through apply_market, whose market::to_add only reads the markers - .blocks-done.json listing every block - so it found nothing to add, nothing to remove, and returned Ok. And even a forced install could not have repaired it: the installer skips every block the marker lists (install.rs `pending`). - odian-packs: InstallOptions::recheck_tile_sets. For those sets install_tiles counts the tiles on disk per block (tiles_on_disk_per_block: z/x/y.ext names only, so .blocks-done.json and the tile server's .missing negatives never count), takes off `done` every block holding fewer than the index names (forget_short_blocks - more is still whole: a PC bakes its own tiles in), rewrites the marker at once so a stopped run reads as partial, and fetches just those blocks; finish_block's rename replaces what is there. The drive-copy path skips its "recorded complete" shortcut for them too. - launcher: Finish passes market::short_tile_sets(market_checked) through Inner::market_recheck (taken by install_market_items), apply_market runs when a re-check is all there is to do, and the log says which sets are repaired. Counted, not hashed: the index names a block's tiles only by digest, so a block whose lost tiles were replaced by as many baked ones reads whole. Spot-check: odian-packs install::tests (a short block is dropped, a whole one and one with extra baked tiles kept, markers not counted); launcher market::tests::finish_repairs_the_sets_a_count_found_short with the counts from the real log.
This commit is contained in:
@@ -190,6 +190,10 @@ struct Inner {
|
||||
dev_refusal: Option<String>,
|
||||
/// What the last count found on disk, kept for as long as nothing can have changed it.
|
||||
market_checked: Option<Vec<market::Check>>,
|
||||
/// Tile sets (`style/dataset`) the next marketplace download re-counts block by block
|
||||
/// instead of trusting their markers: the ones `market_checked` found short, handed over by
|
||||
/// Finish and taken by `install_market_items`.
|
||||
market_recheck: BTreeSet<String>,
|
||||
/// Read once at start: this launcher ships on an ODIAN drive (`settings::portable_root`).
|
||||
portable: bool,
|
||||
/// A newer launcher is being checked for or downloaded (see `update_self`): nothing may
|
||||
@@ -444,7 +448,7 @@ impl Launcher {
|
||||
.map_err(|e| format!("cannot start the HTTP client: {e}"))?;
|
||||
Ok(Launcher {
|
||||
http,
|
||||
inner: Mutex::new(Inner { settings, state, online: false, available: None, token, phase: Phase::Checking, error: None, sync: None, engine_url, live: None, map_from_site: None, map_problem: None, tiles_waiting: None, unfinished: None, dev: None, dev_refusal: None, market_checked: None, portable: crate::settings::portable_drive().is_some(), updating_self: false, offline_reason: None, drive_extras: None }),
|
||||
inner: Mutex::new(Inner { settings, state, online: false, available: None, token, phase: Phase::Checking, error: None, sync: None, engine_url, live: None, map_from_site: None, map_problem: None, tiles_waiting: None, unfinished: None, dev: None, dev_refusal: None, market_checked: None, market_recheck: BTreeSet::new(), portable: crate::settings::portable_drive().is_some(), updating_self: false, offline_reason: None, drive_extras: None }),
|
||||
stack: Mutex::new(None),
|
||||
stack_states: std::sync::Mutex::new(Vec::new()),
|
||||
sign_in_cancel: Notify::new(),
|
||||
@@ -818,11 +822,19 @@ impl Launcher {
|
||||
/// Download what was chosen and never fully arrived, without changing the choice and
|
||||
/// without removing anything: the selection it applies is everything this PC holds or
|
||||
/// asked for, so `apply_market` finds nothing to delete and only fills the gaps.
|
||||
///
|
||||
/// "Never fully arrived" is also a set whose marker says every block landed while the count
|
||||
/// found files missing: the hub counts those as unfinished (`market::unfinished`), so Finish
|
||||
/// must fetch them too. They go down as sets to re-count block by block - the installer
|
||||
/// skips on that very marker, and without this Finish found nothing to add and did nothing.
|
||||
pub async fn finish_market(&self, app: &AppHandle) -> Result<(), String> {
|
||||
let (manifest, _, install_dir) = self.market_manifest(app).await?;
|
||||
let extras = self.inner.lock().await.state.extras.clone().unwrap_or_default();
|
||||
let (extras, short) = {
|
||||
let inner = self.inner.lock().await;
|
||||
(inner.state.extras.clone().unwrap_or_default(), market::short_tile_sets(inner.market_checked.as_deref().unwrap_or_default()))
|
||||
};
|
||||
let keep = market::held_keys(&manifest, &install_dir.join("data"), &extras);
|
||||
self.apply_market(app, keep.into_iter().collect()).await
|
||||
self.apply_market_rechecking(app, keep.into_iter().collect(), short).await
|
||||
}
|
||||
|
||||
/// Ask the site what it has baked since this PC last looked. Quiet on failure: a site
|
||||
@@ -1583,6 +1595,11 @@ impl Launcher {
|
||||
}
|
||||
|
||||
pub async fn apply_market(&self, app: &AppHandle, selected: Vec<String>) -> Result<(), String> {
|
||||
self.apply_market_rechecking(app, selected, BTreeSet::new()).await
|
||||
}
|
||||
|
||||
/// `apply_market`, also repairing `recheck` (`style/dataset`): sets counted short on disk.
|
||||
async fn apply_market_rechecking(&self, app: &AppHandle, selected: Vec<String>, recheck: BTreeSet<String>) -> Result<(), String> {
|
||||
let (manifest, installed, install_dir) = self.market_manifest(app).await?;
|
||||
let extras = market::optional_keys(&manifest, selected);
|
||||
{
|
||||
@@ -1598,13 +1615,17 @@ impl Launcher {
|
||||
}
|
||||
let data = install_dir.join("data");
|
||||
let adds = market::to_add(&manifest, &data, &extras);
|
||||
if !installed || (adds.is_empty() && market::to_remove(&manifest, &data, &extras).is_empty()) {
|
||||
// Only a set still chosen and still in this version is worth re-counting.
|
||||
let (wanted, _) = market::install_filter(&manifest, &extras, true);
|
||||
let recheck: BTreeSet<String> = recheck.into_iter().filter(|set| wanted.contains(set)).collect();
|
||||
if !installed || (adds.is_empty() && recheck.is_empty() && market::to_remove(&manifest, &data, &extras).is_empty()) {
|
||||
self.emit(app).await;
|
||||
return Ok(());
|
||||
}
|
||||
if !adds.is_empty() && !self.inner.lock().await.online {
|
||||
if (!adds.is_empty() || !recheck.is_empty()) && !self.inner.lock().await.online {
|
||||
return Err("Your DHP is unreachable: connect to download marketplace items.".to_string());
|
||||
}
|
||||
self.inner.lock().await.market_recheck = recheck;
|
||||
self.start_run(app, None, true).await
|
||||
}
|
||||
|
||||
@@ -1675,21 +1696,35 @@ impl Launcher {
|
||||
install_dir: &Path,
|
||||
cancel: &CancellationToken,
|
||||
) -> Result<String, InstallFailure> {
|
||||
let (installed, extras) = {
|
||||
let inner = self.inner.lock().await;
|
||||
(inner.state.installed_version.clone(), inner.state.extras.clone().unwrap_or_default())
|
||||
let (installed, extras, recheck) = {
|
||||
let mut inner = self.inner.lock().await;
|
||||
// Taken, not read: a later marketplace download re-counts only if Finish asks again.
|
||||
(inner.state.installed_version.clone(), inner.state.extras.clone().unwrap_or_default(), std::mem::take(&mut inner.market_recheck))
|
||||
};
|
||||
let installed = installed.ok_or_else(|| "ODIAN is not installed.".to_string())?;
|
||||
let verified = odian_packs::version_manifest(source, &installed, odian_packs::bundled_public_key()).await?;
|
||||
let manifest = verified.manifest();
|
||||
let titles: Vec<String> =
|
||||
market::to_add(manifest, &install_dir.join("data"), &extras).into_iter().map(|item| item.title).collect();
|
||||
self.log(app, "info", format!("Downloading {}", titles.join(", ")));
|
||||
if !titles.is_empty() {
|
||||
self.log(app, "info", format!("Downloading {}", titles.join(", ")));
|
||||
}
|
||||
if !recheck.is_empty() {
|
||||
self.log(
|
||||
app,
|
||||
"info",
|
||||
format!(
|
||||
"Fetching what is missing on disk from {} (their markers say complete; the count says otherwise).",
|
||||
recheck.iter().cloned().collect::<Vec<_>>().join(", ")
|
||||
),
|
||||
);
|
||||
}
|
||||
let (tile_sets, src_paths) = market::install_filter(manifest, &extras, true);
|
||||
let options = InstallOptions {
|
||||
only: vec![Stage::Src, Stage::Tiles],
|
||||
tile_sets: Some(tile_sets),
|
||||
src_paths: Some(src_paths),
|
||||
recheck_tile_sets: recheck,
|
||||
..InstallOptions::default()
|
||||
};
|
||||
let progress = {
|
||||
|
||||
@@ -337,6 +337,17 @@ pub fn unfinished(
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// The tile sets a count found short, as the installer names them (`style/dataset`): what
|
||||
/// Finish hands it to re-count block by block, because their markers say they are whole.
|
||||
pub fn short_tile_sets(checked: &[Check]) -> BTreeSet<String> {
|
||||
checked
|
||||
.iter()
|
||||
.filter(|check| !check.whole)
|
||||
.filter_map(|check| check.key.strip_prefix("tiles:"))
|
||||
.map(str::to_string)
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// Whether the chosen optional items still need downloading.
|
||||
pub fn to_add(manifest: &Manifest, data_dir: &Path, extras: &BTreeSet<String>) -> Vec<Item> {
|
||||
items(manifest, data_dir, extras)
|
||||
@@ -389,6 +400,21 @@ pub fn remove_counting(
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn finish_repairs_the_sets_a_count_found_short() {
|
||||
// What the log said on a real PC: two markers claiming "installed" over missing files.
|
||||
let check = |key: &str, found, expected| Check { key: key.to_string(), found, expected, whole: found >= expected };
|
||||
let checked = [
|
||||
check("tiles:popdark/protomaps_20260423", 293_056, 400_268),
|
||||
check("tiles:yandex/protomaps_20260423", 2_371, 28_213),
|
||||
check("tiles:dark_v2/protomaps_20260423", 95_000, 90_501),
|
||||
];
|
||||
let short: Vec<String> = short_tile_sets(&checked).into_iter().collect();
|
||||
// In the installer's own spelling, and a set holding MORE than it names is not short.
|
||||
assert_eq!(short, ["popdark/protomaps_20260423", "yandex/protomaps_20260423"]);
|
||||
assert!(short_tile_sets(&[]).is_empty());
|
||||
}
|
||||
|
||||
fn manifest() -> Manifest {
|
||||
serde_json::from_value(serde_json::json!({
|
||||
"schema": 1, "product": "odian", "version": "0.61.0", "released_at": "2026-09-22T12:00:00Z",
|
||||
|
||||
Reference in New Issue
Block a user