Speed up Transactions, Reconcile and the transactions export again by reading the ledger lines first and joining each to its account with STRAIGHT_JOIN instead of letting the optimizer look lines up through a temporary index, and stop billable expenses waiting to be invoiced from slowing every page: database update 3.5.5 indexes expenses.expense_refund_of_id so the side menu count, the invoice page, Add Expenses and the expense quick look no longer check each waiting expense against every expense; 26.11.2 changelog updated

This commit is contained in:
johnnyq
2026-11-02 16:29:18 -05:00
parent 469f3df2e7
commit 278df4e7b3
4 changed files with 39 additions and 4 deletions

View File

@@ -6,11 +6,12 @@ This file documents all notable changes made to ITFlow.
### Upgrading
Update from Maintenance > Update — Queue Update hands the job to cron, which updates the files and applies this release's one database change (3.5.3 to 3.5.4) in the same pass. Nothing else is required.
Update from Maintenance > Update — Queue Update hands the job to cron, which updates the files and applies this release's two database changes (3.5.3 to 3.5.5) in the same pass. Nothing else is required.
### Bug Fixes
- Transactions: the list, its Transfers pill, Reconcile and the transactions export got slower as the books grew, taking several seconds with a few years of history, and saving a new transfer waited on the same list. They load quickly again. On a test with eight years of books, the Transfers pill went from over six seconds to about a third of a second.
- Transactions: the list, its Transfers pill, Reconcile and the transactions export got slower as the books grew, taking several seconds with a few years of history, and saving a new transfer waited on the same list. They load quickly again. On a test with eight years of books, the Transfers pill went from over six seconds to under a fifth of a second.
- Expenses: billable expenses waiting to go on an invoice slowed down every page, because the side menu counts them and each one was checked against every expense. With a few hundred waiting, that was about a third of a second on every page. The invoice page, Add Expenses and an expense's quick look had the same delay. All of them are quick again.
### Developer Updates
@@ -19,6 +20,8 @@ Update from Maintenance > Update — Queue Update hands the job to cron, which u
- `agent/transactions.php` passes it for one account sorted by date, the only view that shows the balance.
- The transactions export passes it for one account. Reconcile (`reconcileLines()` and the uncleared count in `agent/reconcile.php`) never does.
- `agent/transactions.php` works out the Transfers summary cards in the same query as the pill counts and totals, so the Transfers pill reads the ledger twice instead of three times.
- `accountLedgerSql()` reads its lines first and joins each to its account with `STRAIGHT_JOIN`. Left to choose, the optimizer started from the accounts and found their lines through an index it built on the lines' temporary table (`derived_with_keys`), which was two to three times slower.
- Database 3.5.5 indexes `expenses.expense_refund_of_id`. `expenseUnbilledCount()` (the side menu, on every agent page), `expenseUnbilled()`, `expenseOnInvoice()` and `expenseRow()` sum each expense's refunds in a correlated subquery, which scanned every expense for each one. Like 3.5.4, it is safe to run again, and `db.sql` has the key.
## [26.11.1]

View File

@@ -0,0 +1,26 @@
<?php
/*
* ITFlow - Database update to version 3.5.5 (from 3.5.4)
* Included by admin/database_updates.php - do not access directly
*
* Index expenses.expense_refund_of_id - the expense a refund gives back.
* What is left to bill of an expense is its amount less its refunds, and
* each expense waiting to be billed looked its refunds up with nothing to
* find them by, so every one was checked against every expense. The side
* menu counts them on every agent page (expenseUnbilledCount()), and the
* invoice page, Add Expenses and an expense's quick look do the same.
*
* Re-runnable: skipped where the index is already there.
*/
defined('FROM_DB_UPDATER') || die("Direct file access is not allowed");
$itflow_index_exists = mysqli_query($mysqli, "SELECT 1 FROM information_schema.STATISTICS
WHERE TABLE_SCHEMA = DATABASE() AND TABLE_NAME = 'expenses' AND INDEX_NAME = 'expense_refund_of_id' LIMIT 1");
if (!$itflow_index_exists || mysqli_num_rows($itflow_index_exists) === 0) {
mysqli_query($mysqli, "ALTER TABLE `expenses` ADD KEY `expense_refund_of_id` (`expense_refund_of_id`)");
}
unset($itflow_index_exists);

3
db.sql
View File

@@ -1746,7 +1746,8 @@ CREATE TABLE `expenses` (
KEY `expense_invoice_id` (`expense_invoice_id`),
KEY `expense_import_key` (`expense_import_key`),
KEY `expense_reimbursement_id` (`expense_reimbursement_id`),
KEY `expense_reconciliation_id` (`expense_reconciliation_id`)
KEY `expense_reconciliation_id` (`expense_reconciliation_id`),
KEY `expense_refund_of_id` (`expense_refund_of_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci;
/*!40101 SET character_set_client = @saved_cs_client */;

View File

@@ -363,6 +363,11 @@ function accountLedgerKinds() {
* it is only worked out with $with_balance - where it is shown. Without it,
* txn_balance is NULL: counts, totals and lists across every account don't
* need it.
*
* The lines are read first and each looks its account up (STRAIGHT_JOIN).
* Left to choose, the optimizer starts from the few accounts and finds their
* lines through an index it builds on the lines' temporary table, which is
* two to three times slower.
*/
function accountLedgerSql($account_id = 0, $with_balance = true) {
$account_id = intval($account_id);
@@ -416,7 +421,7 @@ function accountLedgerSql($account_id = 0, $with_balance = true) {
" . ($with_balance ? "accounts.opening_balance + SUM(ledger_lines.txn_amount) OVER (PARTITION BY ledger_lines.txn_account_id ORDER BY ledger_lines.txn_date ASC, ledger_lines.txn_created_at ASC,
ledger_lines.txn_order ASC, ledger_lines.txn_id ASC)" : "NULL") . " AS txn_balance
FROM ($rows) AS ledger_lines
JOIN accounts ON accounts.account_id = ledger_lines.txn_account_id
STRAIGHT_JOIN accounts ON accounts.account_id = ledger_lines.txn_account_id
LEFT JOIN accounts AS other_account ON ledger_lines.txn_kind IN ('transfer_in', 'transfer_out') AND other_account.account_id = ledger_lines.txn_other_account_id
LEFT JOIN account_reconciliations AS cleared ON cleared.reconciliation_id = ledger_lines.txn_reconciliation_id AND cleared.reconciliation_finished_at IS NOT NULL
LEFT JOIN categories ON category_id = ledger_lines.txn_category_id