From 2c9bd0a7f267840355b6311b2dd940f199a496cc Mon Sep 17 00:00:00 2001 From: johnnyq Date: Wed, 11 Nov 2026 21:13:02 -0500 Subject: [PATCH] Let the database update get past 3.2.8 on installs whose tables have different collations, and on ones already stuck there: 3.2.8 matched license names against template names across software and software_templates, the second being one of the tables 2.1.6 to 2.4.1 created without a COLLATE, so where the server default is not utf8mb4_general_ci, MariaDB 11.5 and later, the comparison failed with Illegal mix of collations after the two ALTERs above it had gone in, and every run after that stopped at the first of them with Duplicate column name software_template_id, short of 3.6.5 which puts those tables right; admin/database_updates.php now runs databaseCollationRepair before each update it applies as well as after the last, so no update meets two collations whatever version it starts from, loads functions/database_collation.php itself when the process started on code from before it, and does not try a table again in the same run once it has failed, databaseCollationRepair taking a list of tables to skip; 3.2.8.php is changed in what it can get through and not in what it makes, each ALTER checked first so it can run again from wherever it stopped, the comparison naming its collation and the UPDATE leaving licenses that already have a template; every update from 2.0.7 to 3.6.5 was run on MariaDB with those 22 tables in utf8mb4_uca1400_ai_ci and 3.2.8 was the only one that failed, then the whole history run through the new runner with the tables put wrong before every update, on an install left as the forum report had it and from a process started on older code; CONTRIBUTING.md records the exception to never editing a shipped migration and the 26.11.7 changelog section has the fix --- CHANGELOG.md | 4 +- CONTRIBUTING.md | 2 +- admin/database_updates.php | 63 ++++++++++++++++++++++++-------- admin/database_updates/3.2.8.php | 38 +++++++++++++++---- functions/database_collation.php | 16 +++++--- 5 files changed, 92 insertions(+), 31 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ef7dd2703..c55b01110 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,7 +6,7 @@ This file documents all notable changes made to ITFlow. ### Upgrading -Update from Maintenance > Update — Queue Update hands the job to cron, which updates the files and applies this release's two database changes (3.6.3 to 3.6.5) in the same pass. The first adds indexes to the tickets, ticket replies, invoices and expenses tables: a few seconds on most installs, and up to a few minutes on one with many years of tickets. The second does nothing on most installs; on one first set up before March 2026 that runs on MySQL 8, or on MariaDB 11.5 or later, it rebuilds about twenty small tables, which takes a few seconds. Nothing else is required. +Update from Maintenance > Update — Queue Update hands the job to cron, which updates the files and applies this release's two database changes (3.6.3 to 3.6.5) in the same pass. The first adds indexes to the tickets, ticket replies, invoices and expenses tables: a few seconds on most installs, and up to a few minutes on one with many years of tickets. The second does nothing on most installs; on one first set up before March 2026 that runs on MySQL 8, or on MariaDB 11.5 or later, it rebuilds about twenty small tables, which takes a few seconds. Nothing else is required. If a database update stopped earlier with `Duplicate column name 'software_template_id'`, this release is the fix: update, and it carries on from where it stopped. ### New Features & Updates @@ -18,6 +18,7 @@ Update from Maintenance > Update — Queue Update hands the job to cron, which u - Dashboard: on some upgraded installs the dashboard came up blank after 26.11.4 and the menu under your name did nothing; a product's page, the products list and a draft invoice, quote or recurring invoice could stop the same way. All of them open again. Installs set up new were not affected. - Database: the blank dashboard above came from tables that earlier updates had created with text rules different from the rest of the database - on MySQL 8 and on recent MariaDB, and only on an install that was upgraded, not one set up new. This update puts every table on the same rules, so the same error can't turn up on another page later. Nothing looks different and no data changes. +- Updating: on some installs the database update to 26.11 stopped part-way, and every try after that failed with `Duplicate column name 'software_template_id'`, leaving the database behind the files. These were older installs, first set up before mid-2025, whose database runs on MariaDB 11.5 or later - what Debian 13 comes with. The update now gets through, and on an install already stuck it picks up where it stopped. - Lists: a list long enough to run past 10,000 pages - closed tickets at 10 a page on an install with over 100,000 of them, say - stopped drawing at its page numbers, so the rest of the page was missing and its menus and buttons did nothing. Lists of any length now finish, with their page numbers. ### Developer Updates @@ -30,6 +31,7 @@ Update from Maintenance > Update — Queue Update hands the job to cron, which u - `functions/database_collation.php` is new: `databaseCollationExpected()` reads from `db.sql` each table's default collation and each text column's (two are `utf8mb4_bin`, the rest `utf8mb4_general_ci`), `databaseCollationDifferences()` compares that with `information_schema`, and `databaseCollationRepair()` gives each table that differs one `ALTER` - 3.6.5's, a `MODIFY` a column from its `SHOW CREATE TABLE` line with only the collation replaced, with `db.sql` deciding what each should be, so a binary column that was flattened goes back to `utf8mb4_bin` too. It returns the tables changed and, for each it couldn't, why; a table is changed whole or not at all. Tables and columns `db.sql` doesn't have are not looked at. `healthItflowChecks()` gains a Database collation check (a warning when any table differs), and `healthStructureDifferences()` gains Different collation rows - a row a table, and one more for a column with a collation of its own - which also go into Copy for support. The button posts `repair_database_collation` to the new `admin/post/debug.php`: CSRF-checked, `ignore_user_abort()` and no time limit so a large table isn't cut off with the browser, an audit log entry naming the tables, and a warning in App Logs for each one left. Run on MySQL 8.0 and MariaDB 10.11 through the page and against the whole schema, as 3.6.5 was. - The connection's collation is pinned: `databaseConnectionCollation()` (`functions/database_collation.php`), called at the end of `functions.php`, runs `SET collation_connection = 'utf8mb4_general_ci'` on the connection `config.php` opened. `config.php` never said, so the connection took the server's - `utf8mb4_0900_ai_ci` on MySQL 8 - and a string the server makes itself, `CAST(x AS CHAR)` for one, came out in that and failed beside a column with `Illegal mix of collations`, on a new install as much as an upgraded one. No query did that yet (the only `CAST`/`CONVERT` is the reprice one, which names its collation). Only the collation is set, not the character set: no `SET NAMES`, no `mysqli_set_charset()`. On a server that defaults to `latin1` the connection has always been `latin1`, everything stored went in through it and reads back right through it, and changing the character set there shows every accented letter already stored as two wrong ones - tried on such a server, where `SET NAMES utf8mb4` did exactly that and `collation_connection` alone left what was stored, and what is stored after, the same. It costs one short statement a request, about 0.04 ms on a local socket. - The updater repairs collations after the database updates it applies: `admin/database_updates.php` calls `databaseCollationRepair()` when a run applied at least one and finished, so a table a migration creates without a `COLLATE` is given `db.sql`'s in the same run, before any page reads it. What it changed and what it couldn't go to App Logs and to `scripts/update_cli.php`'s output; a table it can't change is left as it was and never fails the update. A run that stops at a failed migration leaves this for the run that finishes. Run through `update_cli.php --update_db` on MySQL 8.0 with a migration written as 2.4.1 was. +- The update that stopped was database 3.2.8. Its last statement matched license names against template names across `software` and `software_templates`; the second is one of the tables 2.1.6 to 2.4.1 created without a `COLLATE`, so where the server's default isn't `utf8mb4_general_ci` the comparison failed with `Illegal mix of collations` - after the two `ALTER`s above it had gone in, which is why every later run stopped at the first of them. 3.6.5, which puts those tables right, comes after it. Two changes. `admin/database_updates.php` now runs `databaseCollationRepair()` before each update it applies as well as after the last, so no update meets two collations whatever version it starts from; it loads `functions/database_collation.php` itself when the process started on code from before it, and a table that fails is not tried again in the same run (`databaseCollationRepair()` takes a list to skip). And `3.2.8.php` is changed in what it can get through, not in what it makes: each `ALTER` is checked first, the comparison names its collation, and the `UPDATE` leaves licenses that already have a template. Every update from 2.0.7 to 3.6.5 was run on MariaDB with those 22 tables in `utf8mb4_uca1400_ai_ci` - 3.2.8 was the only one that failed - and the whole history then run through the new runner with the tables put wrong before every update, on an install left as the forum report had it, and from a process started on older code. - `includes/filter_footer.php` works `$pages_split` out for any number of pages (10, 100, 1,000, 10,000 and so on up). It was set only up to 10,000 pages, and past that `is_int($i / $pages_split)` threw a `DivisionByZeroError` in the middle of the pager. ## [26.11.6] diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index dc769ddb4..3d208bf07 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -267,7 +267,7 @@ Two standing exceptions: junction tables (`client_tags`, `service_assets`, …) **Schema changes require two edits in one PR:** 1. `db.sql` — so fresh installs get the new schema. -2. `admin/database_updates/.php` — a new file named for the version it upgrades **to**, containing only the queries that apply the change. Migrations are sequential and rolling-release; never edit a historical file. An install that has run a migration never runs it again, so an edit only reaches installs that hadn't updated yet and leaves the rest different: 2.3.7 first created `contract_templates.company_id`, was then changed to leave it out, and every install that ran it in v25.11 kept the column until 3.6.3 dropped it. To change what a shipped migration did, add a new one. A `CREATE TABLE` in a migration ends `ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci`, as every table in `db.sql` does: leave the `COLLATE` out and the table takes the server's default, which on MySQL 8 and recent MariaDB is a different one, and the first query to put its text beside another table's fails with "Illegal mix of collations" (3.6.5 had to bring 22 such tables back). The updater now checks every table against `db.sql` after the migrations it applies and repairs one that differs, and the connection is given the same collation on every request (`functions/database_collation.php`) - a net under this rule, not a reason to skip it, and the reason a query never has to name a collation of its own. +2. `admin/database_updates/.php` — a new file named for the version it upgrades **to**, containing only the queries that apply the change. Migrations are sequential and rolling-release; never edit a historical file. An install that has run a migration never runs it again, so an edit only reaches installs that hadn't updated yet and leaves the rest different: 2.3.7 first created `contract_templates.company_id`, was then changed to leave it out, and every install that ran it in v25.11 kept the column until 3.6.3 dropped it. To change what a shipped migration did, add a new one. The one exception is a migration that can't finish on some installs, which a new one can never reach: change it so that it can, without changing what it makes (3.2.8 was). Write each step so the migration can run again from wherever it stopped - check `information_schema` before an `ADD`, as 3.3.4 does - because MySQL and MariaDB don't undo the statements that went in before the one that failed. A `CREATE TABLE` in a migration ends `ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci`, as every table in `db.sql` does: leave the `COLLATE` out and the table takes the server's default, which on MySQL 8 and recent MariaDB is a different one, and the first query to put its text beside another table's fails with "Illegal mix of collations" (3.6.5 had to bring 22 such tables back). The updater now checks every table against `db.sql` after the migrations it applies and repairs one that differs, and the connection is given the same collation on every request (`functions/database_collation.php`) - a net under this rule, not a reason to skip it, and the reason a query never has to name a collation of its own. That is the whole job. `LATEST_DATABASE_VERSION` is derived from the highest-numbered filename in `admin/database_updates/`, and the runner (`admin/database_updates.php`) steps `config_current_database_version` after each file succeeds — so there is no constant to bump and no version-bump query to write. Each migration file needs the standard `defined('FROM_DB_UPDATER') || die(...)` guard at the top; copy an existing file's header. diff --git a/admin/database_updates.php b/admin/database_updates.php index 03d5d77c4..49d377927 100644 --- a/admin/database_updates.php +++ b/admin/database_updates.php @@ -32,6 +32,44 @@ $database_updates_applied = []; // Versions successfully applied this run $database_updates_error = null; // "version: error message" if a migration failed $database_updates_collations = null; // ['changed' => [table, ...], 'failed' => [table => why]] when collations were checked +// Every table's collation against db.sql's (functions/database_collation.php), before each +// update and after the last - $database_updates_collations is what that changed and couldn't. +// +// Before each, because an update can compare text in one table with text in another, and two +// tables in different collations stop it with "Illegal mix of collations". The tables that +// 2.1.6 to 2.4.1 created took the server's collation, which on MariaDB 11.5 and later isn't +// the other tables'; 3.2.8 then matched license names against template names across the two, +// failed with its columns already added, and from there every run stopped at "Duplicate +// column name" - and 3.6.5, which puts those tables right, is on the far side of it. With +// the tables brought into line first, no update meets that, whichever version it starts from. +// +// After the last, because one of these updates may be the one that made a table without a +// COLLATE - right on the server it was written on, wrong on the others. Put right here, in +// the run that made it, the slip never reaches a page. +// +// A process that started on code from before these functions (see update_cli.php, on handing +// the database phase to a new process - one that couldn't, reads this file fresh off the new +// tree against the functions it started with) loads them here. Nothing in this can fail the +// update: a table that can't be changed is left as it was, said so, and not tried again. +if (!function_exists('databaseCollationRepair') && is_file(__DIR__ . '/../functions/database_collation.php')) { + require_once __DIR__ . '/../functions/database_collation.php'; +} + +$database_updates_collation_pass = function () use (&$database_updates_collations) { + if (!function_exists('databaseCollationRepair')) { + return; + } + try { + $pass = databaseCollationRepair(array_keys($database_updates_collations['failed'] ?? [])); + $database_updates_collations = [ + 'changed' => array_values(array_unique(array_merge($database_updates_collations['changed'] ?? [], $pass['changed']))), + 'failed' => ($database_updates_collations['failed'] ?? []) + $pass['failed'], + ]; + } catch (Throwable $e) { + error_log("ITFlow database update: checking collations failed: " . $e->getMessage()); + } +}; + // Collect and order the migration files by version (glob sorts alphabetically, // which would put 2.4.10 before 2.4.9 - version_compare gets it right) $database_update_files = []; @@ -53,6 +91,8 @@ foreach ($database_update_files as $version => $file) { continue; } + $database_updates_collation_pass(); + try { require $file; } catch (Throwable $e) { @@ -69,26 +109,19 @@ foreach ($database_update_files as $version => $file) { $database_updates_applied[] = $version; } -// After the updates, every table against db.sql's collation (functions/database_collation.php). -// A migration that creates a table without a COLLATE gives it the server's default - right on -// the server it was written on, and utf8mb4_0900_ai_ci on MySQL 8 or utf8mb4_uca1400_ai_ci on -// MariaDB 11.5 and later, where the first query to read it beside another table then fails. -// 2.1.6 to 2.4.1 did exactly that, and it took 26.11.4's dashboard to show it. Putting it -// right here, in the run that made the table, means such a slip never reaches a page. -// -// Only when this run applied something and finished: a run that stopped has the database -// part-way, and the re-run that completes it comes through here. Nothing here can fail the -// update - a table that can't be changed is left as it was and said so. -if ($database_updates_applied && !$database_updates_error && function_exists('databaseCollationRepair')) { +if ($database_updates_applied && !$database_updates_error) { + $database_updates_collation_pass(); +} + +if ($database_updates_collations && function_exists('logApp')) { try { - $database_updates_collations = databaseCollationRepair(); if ($database_updates_collations['changed']) { - logApp('Database Update', 'info', "Gave " . (count($database_updates_collations['changed']) === 1 ? '1 table' : count($database_updates_collations['changed']) . ' tables') . " the collation db.sql has after the database update: " . implode(', ', $database_updates_collations['changed'])); + logApp('Database Update', 'info', "Gave " . (count($database_updates_collations['changed']) === 1 ? '1 table' : count($database_updates_collations['changed']) . ' tables') . " the collation db.sql has during the database update: " . implode(', ', $database_updates_collations['changed'])); } foreach ($database_updates_collations['failed'] as $table => $why) { - logApp('Database Update', 'warning', "After the database update the table $table doesn't have the collation db.sql has, and could not be given it: $why"); + logApp('Database Update', 'warning', "The table $table doesn't have the collation db.sql has, and the database update could not give it: $why"); } } catch (Throwable $e) { - error_log("ITFlow database update: checking collations failed: " . $e->getMessage()); + error_log("ITFlow database update: logging collations failed: " . $e->getMessage()); } } diff --git a/admin/database_updates/3.2.8.php b/admin/database_updates/3.2.8.php index a143b8d25..c4cbdb6e5 100644 --- a/admin/database_updates/3.2.8.php +++ b/admin/database_updates/3.2.8.php @@ -7,20 +7,42 @@ defined('FROM_DB_UPDATER') || die("Direct file access is not allowed"); + // Changed after it shipped, in what it can get through and not in what it does. As + // first written, the last statement compared a license's name with a template's, and on + // an install where software_templates had another collation than software (MariaDB 11.5 + // and later gave it one - see 3.6.5) that stopped with "Illegal mix of collations". By + // then the columns above it were in, so every run after stopped at the first line with + // "Duplicate column name 'software_template_id'" and the install could go no further. + // Each step is now checked first, so the update can run again from wherever it was + // left, and the comparison names its collation. An install that ran it through has + // exactly what this makes. + // The license template a license was made from - 0 for none - mysqli_query($mysqli, "ALTER TABLE `software` - ADD `software_template_id` int(11) NOT NULL DEFAULT 0 AFTER `software_client_id`, - ADD KEY `software_template_id` (`software_template_id`)"); + $itflow_column = mysqli_query($mysqli, "SELECT 1 FROM information_schema.COLUMNS + WHERE TABLE_SCHEMA = DATABASE() AND TABLE_NAME = 'software' AND COLUMN_NAME = 'software_template_id' LIMIT 1"); + if (!$itflow_column || mysqli_num_rows($itflow_column) === 0) { + mysqli_query($mysqli, "ALTER TABLE `software` + ADD `software_template_id` int(11) NOT NULL DEFAULT 0 AFTER `software_client_id`, + ADD KEY `software_template_id` (`software_template_id`)"); + } // Who made a license template and who last changed it - mysqli_query($mysqli, "ALTER TABLE `software_templates` - ADD `software_template_created_by` int(11) NOT NULL DEFAULT 0 AFTER `software_template_archived_at`, - ADD `software_template_updated_by` int(11) NOT NULL DEFAULT 0 AFTER `software_template_created_by`"); + $itflow_column = mysqli_query($mysqli, "SELECT 1 FROM information_schema.COLUMNS + WHERE TABLE_SCHEMA = DATABASE() AND TABLE_NAME = 'software_templates' AND COLUMN_NAME = 'software_template_created_by' LIMIT 1"); + if (!$itflow_column || mysqli_num_rows($itflow_column) === 0) { + mysqli_query($mysqli, "ALTER TABLE `software_templates` + ADD `software_template_created_by` int(11) NOT NULL DEFAULT 0 AFTER `software_template_archived_at`, + ADD `software_template_updated_by` int(11) NOT NULL DEFAULT 0 AFTER `software_template_created_by`"); + } // A license called exactly what one template - and only one - is called was made from it, // as the New License form filled the name in from the template. It isn't counted as a change. + // Only licenses not yet given one, so a second run leaves alone what the first one set. mysqli_query($mysqli, "UPDATE `software` JOIN ( SELECT MIN(`software_template_id`) AS `template_id`, LOWER(TRIM(`software_template_name`)) AS `template_name` FROM `software_templates` GROUP BY LOWER(TRIM(`software_template_name`)) HAVING COUNT(*) = 1 - ) AS `templates` ON LOWER(TRIM(`software`.`software_name`)) = `templates`.`template_name` - SET `software`.`software_template_id` = `templates`.`template_id`, `software`.`software_updated_at` = `software`.`software_updated_at`"); + ) AS `templates` ON CONVERT(LOWER(TRIM(`software`.`software_name`)) USING utf8mb4) COLLATE utf8mb4_general_ci = `templates`.`template_name` + SET `software`.`software_template_id` = `templates`.`template_id`, `software`.`software_updated_at` = `software`.`software_updated_at` + WHERE `software`.`software_template_id` = 0"); + + unset($itflow_column); diff --git a/functions/database_collation.php b/functions/database_collation.php index 1b0f4593f..baa665f2a 100644 --- a/functions/database_collation.php +++ b/functions/database_collation.php @@ -24,9 +24,12 @@ * Two things keep it that way. The connection is given the tables' collation * on every request (databaseConnectionCollation(), from functions.php), so a * string the server makes itself matches them. And the database updater runs - * the repair after the updates it applies (admin/database_updates.php), so a - * table a future update creates without a COLLATE is put right in the same - * run, on the servers where that matters. + * the repair before each update it applies and after the last + * (admin/database_updates.php), so an update never meets two collations and + * a table one creates without a COLLATE is put right in the same run. + * + * This file is loaded by that updater in a process that started on older + * code, so it leans on nothing but the connection and db.sql. */ /* @@ -159,7 +162,8 @@ function databaseCollationDifferences() { /* * Give every table and text column in databaseCollationDifferences() the * collation db.sql has for it. Returns ['changed' => [table, ...], 'failed' - * => [table => why]]. + * => [table => why]]. $skip is tables to leave - the ones that failed a + * moment ago, for a caller that comes back more than once in a run. * * A table's ALTER changes all of it or none of it, so one that fails is * exactly as it was and the rest go on. What stops one is a unique text @@ -172,11 +176,11 @@ function databaseCollationDifferences() { * mode is put back after; strict mode itself stays on, so text that would * not fit is an error and never cut short. */ -function databaseCollationRepair() { +function databaseCollationRepair(array $skip = []) { global $mysqli; $changed = []; $failed = []; - $differences = databaseCollationDifferences(); + $differences = array_diff_key((array) databaseCollationDifferences(), array_flip($skip)); if (!$differences) { return ['changed' => $changed, 'failed' => $failed]; }