diff --git a/CHANGELOG.md b/CHANGELOG.md index 0c662356b..ef7dd2703 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,6 +12,7 @@ Update from Maintenance > Update — Queue Update hands the job to cron, which u - Speed: the dashboard, and the menu on every page, are much quicker on an ITFlow with years of history in it. Most of what they show is about what is still open - open tickets, unpaid invoices, unbilled time - and finding those few meant reading through every ticket and invoice ever made, several times over. They are now found directly. On a test install with 280,000 tickets, 380,000 replies and 70,000 invoices the dashboard went from about 5 seconds to under 1, a technician's dashboard from under 3 seconds to a tenth of one, and every other page opened about half a second sooner. A smaller install won't see much difference. Every figure shown is the same as before. - Debug: Admin > Debug now checks that every table uses the same text rules (the collation) a new install has. Any that don't are listed under Database, with a Repair collations button that puts them right - nothing in the tables changes. Different rules in different tables are what blanked the dashboard for some installs after 26.11.4; this update already puts those right, and the check is there for anything it had to leave and for a database brought over from another server. +- Database: ITFlow now keeps the database's text rules the same everywhere by itself. Each time it updates, it checks every table afterwards and corrects any that came out different, and it tells the database to use the tables' rules for its own work too. There is nothing to set, and nothing looks different - it is there so that the kind of error that blanked the dashboard can't come back with a later update. ### Bug Fixes @@ -27,6 +28,8 @@ Update from Maintenance > Update — Queue Update hands the job to cron, which u - `productRepriceUnionSql()` gives every text column of its three branches one collation (`CONVERT(... USING utf8mb4) COLLATE utf8mb4_general_ci`). Database 2.4.1 made `quote_items` and `recurring_invoice_items` with `DEFAULT CHARSET=utf8mb4` and no `COLLATE`, so on a server whose default isn't `utf8mb4_general_ci` (MySQL 8: `utf8mb4_0900_ai_ci`) their `item_name` differs from `invoice_items` and the `UNION ALL` threw `Illegal mix of collations for operation 'UNION'` - from `productRepriceCount()`, which the dashboard's attention list calls. The tables themselves are left as they are. - Database 3.6.5 (`admin/database_updates/3.6.5.php`) brings every ITFlow table whose default collation, or any text column, isn't `utf8mb4_general_ci` to it - in practice the 22 tables that 2.1.6 to 2.4.1 created without a `COLLATE` (`quote_items`, `recurring_invoice_items`, `contracts`, `credits`, `payment_methods`, `software_keys` and the rest), which took the server's default: `utf8mb4_0900_ai_ci` on MySQL 8, `utf8mb4_uca1400_ai_ci` on MariaDB 11.5 and later. It reads `information_schema`, so a table already right is not touched and a second run changes nothing; `db.sql` already had them all and is unchanged. Each table is one `ALTER` of `MODIFY`s built from its own `SHOW CREATE TABLE` lines with only the collation replaced, not `CONVERT TO CHARACTER SET` as 2.0.6 used: `CONVERT TO` would also take `calendars.calendar_feed_key` and `user_calendar_feeds.feed_key`, which are `utf8mb4_bin` on purpose (a `MODIFY` beside it in the same `ALTER` does not hold them back, on MySQL or MariaDB), and would turn a `utf8mb3` `TEXT` into a `MEDIUMTEXT`. Columns with a binary collation and tables that aren't ITFlow's are left alone. `NO_ZERO_DATE` and `NO_ZERO_IN_DATE` are lifted for the connection while it runs, because `records.record_updated_at` defaults to a zero date and MySQL 8 otherwise refuses any `ALTER` of that table. A table that can't be changed - two rows of a unique text column that only the old collation tells apart - is left exactly as it was, with a warning in App Logs and the PHP error log, and the update carries on. Run on MySQL 8.0 and MariaDB 10.11 against the whole schema with a row in every table, from `utf8mb4_0900_ai_ci`, `utf8mb4_uca1400_ai_ci`, `utf8mb3` and `latin1`: every definition and every row came out the same as a new install's. - `functions/database_collation.php` is new: `databaseCollationExpected()` reads from `db.sql` each table's default collation and each text column's (two are `utf8mb4_bin`, the rest `utf8mb4_general_ci`), `databaseCollationDifferences()` compares that with `information_schema`, and `databaseCollationRepair()` gives each table that differs one `ALTER` - 3.6.5's, a `MODIFY` a column from its `SHOW CREATE TABLE` line with only the collation replaced, with `db.sql` deciding what each should be, so a binary column that was flattened goes back to `utf8mb4_bin` too. It returns the tables changed and, for each it couldn't, why; a table is changed whole or not at all. Tables and columns `db.sql` doesn't have are not looked at. `healthItflowChecks()` gains a Database collation check (a warning when any table differs), and `healthStructureDifferences()` gains Different collation rows - a row a table, and one more for a column with a collation of its own - which also go into Copy for support. The button posts `repair_database_collation` to the new `admin/post/debug.php`: CSRF-checked, `ignore_user_abort()` and no time limit so a large table isn't cut off with the browser, an audit log entry naming the tables, and a warning in App Logs for each one left. Run on MySQL 8.0 and MariaDB 10.11 through the page and against the whole schema, as 3.6.5 was. +- The connection's collation is pinned: `databaseConnectionCollation()` (`functions/database_collation.php`), called at the end of `functions.php`, runs `SET collation_connection = 'utf8mb4_general_ci'` on the connection `config.php` opened. `config.php` never said, so the connection took the server's - `utf8mb4_0900_ai_ci` on MySQL 8 - and a string the server makes itself, `CAST(x AS CHAR)` for one, came out in that and failed beside a column with `Illegal mix of collations`, on a new install as much as an upgraded one. No query did that yet (the only `CAST`/`CONVERT` is the reprice one, which names its collation). Only the collation is set, not the character set: no `SET NAMES`, no `mysqli_set_charset()`. On a server that defaults to `latin1` the connection has always been `latin1`, everything stored went in through it and reads back right through it, and changing the character set there shows every accented letter already stored as two wrong ones - tried on such a server, where `SET NAMES utf8mb4` did exactly that and `collation_connection` alone left what was stored, and what is stored after, the same. It costs one short statement a request, about 0.04 ms on a local socket. +- The updater repairs collations after the database updates it applies: `admin/database_updates.php` calls `databaseCollationRepair()` when a run applied at least one and finished, so a table a migration creates without a `COLLATE` is given `db.sql`'s in the same run, before any page reads it. What it changed and what it couldn't go to App Logs and to `scripts/update_cli.php`'s output; a table it can't change is left as it was and never fails the update. A run that stops at a failed migration leaves this for the run that finishes. Run through `update_cli.php --update_db` on MySQL 8.0 with a migration written as 2.4.1 was. - `includes/filter_footer.php` works `$pages_split` out for any number of pages (10, 100, 1,000, 10,000 and so on up). It was set only up to 10,000 pages, and past that `is_int($i / $pages_split)` threw a `DivisionByZeroError` in the middle of the pager. ## [26.11.6] diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index a61ab7cb1..dc769ddb4 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -267,7 +267,7 @@ Two standing exceptions: junction tables (`client_tags`, `service_assets`, …) **Schema changes require two edits in one PR:** 1. `db.sql` — so fresh installs get the new schema. -2. `admin/database_updates/.php` — a new file named for the version it upgrades **to**, containing only the queries that apply the change. Migrations are sequential and rolling-release; never edit a historical file. An install that has run a migration never runs it again, so an edit only reaches installs that hadn't updated yet and leaves the rest different: 2.3.7 first created `contract_templates.company_id`, was then changed to leave it out, and every install that ran it in v25.11 kept the column until 3.6.3 dropped it. To change what a shipped migration did, add a new one. A `CREATE TABLE` in a migration ends `ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci`, as every table in `db.sql` does: leave the `COLLATE` out and the table takes the server's default, which on MySQL 8 and recent MariaDB is a different one, and the first query to put its text beside another table's fails with "Illegal mix of collations" (3.6.5 had to bring 22 such tables back). +2. `admin/database_updates/.php` — a new file named for the version it upgrades **to**, containing only the queries that apply the change. Migrations are sequential and rolling-release; never edit a historical file. An install that has run a migration never runs it again, so an edit only reaches installs that hadn't updated yet and leaves the rest different: 2.3.7 first created `contract_templates.company_id`, was then changed to leave it out, and every install that ran it in v25.11 kept the column until 3.6.3 dropped it. To change what a shipped migration did, add a new one. A `CREATE TABLE` in a migration ends `ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci`, as every table in `db.sql` does: leave the `COLLATE` out and the table takes the server's default, which on MySQL 8 and recent MariaDB is a different one, and the first query to put its text beside another table's fails with "Illegal mix of collations" (3.6.5 had to bring 22 such tables back). The updater now checks every table against `db.sql` after the migrations it applies and repairs one that differs, and the connection is given the same collation on every request (`functions/database_collation.php`) - a net under this rule, not a reason to skip it, and the reason a query never has to name a collation of its own. That is the whole job. `LATEST_DATABASE_VERSION` is derived from the highest-numbered filename in `admin/database_updates/`, and the runner (`admin/database_updates.php`) steps `config_current_database_version` after each file succeeds — so there is no constant to bump and no version-bump query to write. Each migration file needs the standard `defined('FROM_DB_UPDATER') || die(...)` guard at the top; copy an existing file's header. diff --git a/admin/database_updates.php b/admin/database_updates.php index 8a6fe054c..03d5d77c4 100644 --- a/admin/database_updates.php +++ b/admin/database_updates.php @@ -30,6 +30,7 @@ define("FROM_DB_UPDATER", true); // Outputs for the caller (update_cli.php) $database_updates_applied = []; // Versions successfully applied this run $database_updates_error = null; // "version: error message" if a migration failed +$database_updates_collations = null; // ['changed' => [table, ...], 'failed' => [table => why]] when collations were checked // Collect and order the migration files by version (glob sorts alphabetically, // which would put 2.4.10 before 2.4.9 - version_compare gets it right) @@ -67,3 +68,27 @@ foreach ($database_update_files as $version => $file) { $database_current_version = $version; $database_updates_applied[] = $version; } + +// After the updates, every table against db.sql's collation (functions/database_collation.php). +// A migration that creates a table without a COLLATE gives it the server's default - right on +// the server it was written on, and utf8mb4_0900_ai_ci on MySQL 8 or utf8mb4_uca1400_ai_ci on +// MariaDB 11.5 and later, where the first query to read it beside another table then fails. +// 2.1.6 to 2.4.1 did exactly that, and it took 26.11.4's dashboard to show it. Putting it +// right here, in the run that made the table, means such a slip never reaches a page. +// +// Only when this run applied something and finished: a run that stopped has the database +// part-way, and the re-run that completes it comes through here. Nothing here can fail the +// update - a table that can't be changed is left as it was and said so. +if ($database_updates_applied && !$database_updates_error && function_exists('databaseCollationRepair')) { + try { + $database_updates_collations = databaseCollationRepair(); + if ($database_updates_collations['changed']) { + logApp('Database Update', 'info', "Gave " . (count($database_updates_collations['changed']) === 1 ? '1 table' : count($database_updates_collations['changed']) . ' tables') . " the collation db.sql has after the database update: " . implode(', ', $database_updates_collations['changed'])); + } + foreach ($database_updates_collations['failed'] as $table => $why) { + logApp('Database Update', 'warning', "After the database update the table $table doesn't have the collation db.sql has, and could not be given it: $why"); + } + } catch (Throwable $e) { + error_log("ITFlow database update: checking collations failed: " . $e->getMessage()); + } +} diff --git a/functions.php b/functions.php index 39c96734c..aaac666ee 100644 --- a/functions.php +++ b/functions.php @@ -107,3 +107,7 @@ require_once __DIR__ . '/functions/settings.php'; require_once __DIR__ . '/functions/mail_queue.php'; require_once __DIR__ . '/functions/managed.php'; require_once __DIR__ . '/functions/demo.php'; + +// The connection config.php opened takes the collation the tables have, so that a string the +// server makes (CAST(x AS CHAR), say) matches them - functions/database_collation.php +databaseConnectionCollation(); diff --git a/functions/database_collation.php b/functions/database_collation.php index 3dcbd5ed6..1b0f4593f 100644 --- a/functions/database_collation.php +++ b/functions/database_collation.php @@ -20,8 +20,51 @@ * CHARACTER SET, which takes every text column in the table - the binary ones * with the rest - and turns a utf8mb3 TEXT into a MEDIUMTEXT. Only tables and * columns db.sql has are looked at; anything else in the database is left. + * + * Two things keep it that way. The connection is given the tables' collation + * on every request (databaseConnectionCollation(), from functions.php), so a + * string the server makes itself matches them. And the database updater runs + * the repair after the updates it applies (admin/database_updates.php), so a + * table a future update creates without a COLLATE is put right in the same + * run, on the servers where that matters. */ +/* + * Give the connection the collation the tables have. + * + * config.php connects without saying, so the connection takes the server's: + * utf8mb4_0900_ai_ci on MySQL 8. A quoted string in a query gives way to the + * column it meets, so nothing shows - until a query has the server make a + * string of its own. CAST(x AS CHAR) comes out in the connection's collation + * and does not give way: beside a column, in a comparison or a UNION, it is + * "Illegal mix of collations", on a new install as much as an upgraded one. + * + * Only the collation is set, never the character set (SET NAMES, or + * mysqli_set_charset()). On a server that defaults to latin1 - MySQL 5.7, + * MariaDB before 11.6 where the packager didn't change it - the connection + * has been latin1 since the install was made, and everything in it was + * written through that and reads back right through it. Changing the + * character set there turns every accented letter already stored into two + * wrong ones. Setting collation_connection changes what the server makes + * strings as, and leaves how text travels in and out exactly as it was - + * tried on a latin1 server: what was stored reads back the same, and what + * is written after is stored the same. + * + * Here and not in config.php, which each install has its own copy of. It is + * one short statement on a connection that is already open. + */ +function databaseConnectionCollation() { + global $mysqli; + if (!isset($mysqli) || !($mysqli instanceof mysqli)) { + return; + } + try { + mysqli_query($mysqli, "SET collation_connection = 'utf8mb4_general_ci'"); + } catch (Throwable $e) { + // The page goes on with the server's own - as every page did before this + } +} + /** A text column's type as SHOW CREATE TABLE and db.sql write it, for a regex. An enum's values are quoted, with a quote inside one doubled. */ function databaseCollationTextType() { return "(?:var)?char\\(\\d+\\)|(?:tiny|medium|long)?text|(?:enum|set)\\('(?:[^']|'')*'(?:,'(?:[^']|'')*')*\\)"; diff --git a/scripts/update_cli.php b/scripts/update_cli.php index 7da2abf2e..21af9f02f 100644 --- a/scripts/update_cli.php +++ b/scripts/update_cli.php @@ -194,6 +194,14 @@ foreach ($database_updates_applied as $applied_version) { echo "Applied database update $applied_version\n"; } +// The collations the runner checked after them (admin/database_updates.php) +if (!empty($database_updates_collations['changed'])) { + echo "Gave " . (count($database_updates_collations['changed']) === 1 ? '1 table' : count($database_updates_collations['changed']) . ' tables') . " the collation db.sql has: " . implode(', ', $database_updates_collations['changed']) . "\n"; +} +foreach ($database_updates_collations['failed'] ?? [] as $collation_table => $collation_why) { + echo "Warning: the table $collation_table doesn't have the collation db.sql has and could not be given it ($collation_why) - Admin > Debug lists it.\n"; +} + if ($database_updates_error) { $stopped_at_version = count($database_updates_applied) > 0 ? end($database_updates_applied) : $old_db_version; fwrite(STDERR, "Error: database update failed at $database_updates_error\n");