diff --git a/CHANGELOG.md b/CHANGELOG.md index 7d6733ca1..888a218a8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,7 +6,7 @@ This file documents all notable changes made to ITFlow. ### Upgrading -Update from Maintenance > Update — Queue Update hands the job to cron and it applies on its own. There is no database change in this release, so nothing else is required. If clients have signed contracts with you, read the first Contracts entry under Bug Fixes: a signed copy may give shorter response times than the ones your SLAs time tickets by. From 26.11.3 or earlier, the same update also brings 26.11.4 and its database changes - see its notes below. +Update from Maintenance > Update — Queue Update hands the job to cron, which updates the files and applies this release's one database change (3.6.5 to 3.6.6) in the same pass. It adds indexes to the audit log and to tickets: a few seconds on most installs, about 2 seconds for every million audit log entries on one with years of them. Nothing else is required. If clients have signed contracts with you, read the first Contracts entry under Bug Fixes: a signed copy may give shorter response times than the ones your SLAs time tickets by. From 26.11.3 or earlier, the same update also brings 26.11.4 and its database changes - see its notes below. ### New Features & Updates @@ -29,6 +29,7 @@ Update from Maintenance > Update — Queue Update hands the job to cron and it a - SLAs: archiving an SLA says first what stops getting it - the priorities it is the default for and the contracts that give it - and that tickets already on it keep their targets. A priority whose default has been archived says so, where it used to read None. - SLAs: holidays and closure days are views of the same list, coming up and past, searched by name, date, month or day of the week. Each shows its date and how far off it is, and one that falls on a day you're shut anyway says so. Add US holidays opens a window that says how many of each year's are already in and offers the first year with some to add. Removing a day asks first, and says what changes. - SLAs: beside the list, Default SLA by priority uses the select picker, with each SLA's targets under its name. Business hours and alerts has the business days as buttons, the hours, when to warn and who else to email, and says when cron isn't running, as no warning goes out without it. On tickets now counts the open tickets with an SLA, those close to a target and those that missed one, each a link to the tickets. +- Speed: six pages read every entry of their table each time they opened, and do so no more - on an ITFlow with years behind it they now open in a fraction of the time. On a test install with eight years of data (a million audit log entries, 280,000 tickets, 70,000 invoices and 100,000 payments, income and expenses): Admin > Audit Logs from 2.5 seconds to 0.16, and to a few hundredths filtered to one person or client; Admin > Sign-ins from 0.4 to 0.17, and By IP from 1.8 to 0.23; Admin > Users from 1.1 seconds to a hundredth; Transactions from 0.95 to 0.39; Income from 0.70 to 0.39; Invoices from 0.43 to 0.15. Every count, total and row is the same as before. ### Bug Fixes @@ -39,6 +40,9 @@ Update from Maintenance > Update — Queue Update hands the job to cron and it a - Contracts: a contract's response times - on its page, in its window and in the document the client signs - were worded in plain time, where SLA clocks count only your business hours. With a 9 to 5 day, a 48-hour resolution target read "2 days" and a 24-hour one "24 hrs", where the clock gives 6 and 3 business days. They now read as the clock counts them - "4 business hours", "6 business days" - the same as Admin > SLAs and the emails to clients. A contract already signed, or out for signature, keeps the wording it was sent with and is not marked as changed. Open the signed copy of any contract whose targets run to a business day or more: if it promises more than you meant, set the SLA's targets to what you meant in Admin > SLAs, or send the contract to be signed again. - Contracts: because the wording now follows your business hours, changing them changes what a target of a business day or more comes to. A contract signed from this release on is then marked as changed since it was signed, as it is when its SLA's targets are changed. - Dashboard: it could take seconds to open - 2 seconds on one install, over 20 on a test one - on an ITFlow with many assets that weren't made from an invoice, which is most of them. The Hardware not on assets count was to blame, and the invoices list it opens, with Hardware not on assets as its filter, was as slow. Both are back to a few hundredths of a second, and every number is the same as before. +- Projects: the projects list, and its Stalled and Ready to close views, could take seconds to open - over 5 on a test install - where no ticket was on a project yet, or very few were. They open at once. +- Invoices: opening an invoice read through every ticket to find the ones billed on it - three quarters of a second with 280,000 tickets. It finds them directly. +- Sign-ins: people or addresses that tied on what the list was sorted by could come back in a different order each time, and so turn up on two pages or on none. They keep one order. ### Developer Updates @@ -73,6 +77,12 @@ Update from Maintenance > Update — Queue Update hands the job to cron and it a - The old wording stays behind `$old_wording` on `slaMinutesText()`, passed along by `contractSlaTargets()`, `contractSlaTableHtml()`, `contractPlaceholderValues()`, `contractFillTerms()` and `contractDocumentHtml()`, and only `contractDocumentIs($row, $as_of, $hash)` asks for it. `contractSignatureCheckChanged()` and `contractSignatureOutdated()` compare a round's `contract_signature_document_hash` through it - the document in today's wording, then in the old - so a round sent up to 26.11.7 is not withdrawn or marked outdated by the rewording alone. A document's fingerprint now depends on the business days and hours wherever a target is a business day or more. - Hooks for tests and scripts: `data-sla-view` on each view's link, `data-sla-list` with `data-sla-id` on each row, `data-sla-default-for`, `data-sla-archive` and `data-sla-restore`; `data-sla-days` with `data-sla-day` (the date) on each row; `data-sla-new`, `data-sla-new-day` and `data-sla-us-holidays`; `data-sla-default` (the priority) on each default's select, `data-sla-closures`, `data-sla-alerts-stopped` and `data-sla-now`. - `productHardwarePending()` and the invoices list's `hardware` filter count each line's assets once and join them, through the new `productHardwareMadeJoinSql()` (a `LEFT JOIN` onto `invoice_items` of assets with `asset_invoice_id > 0`, grouped by invoice and line), with `productHardwareShortSql()` now comparing against `hardware_made.made`. Both had a `COUNT(*)` subquery a line, `WHERE asset_invoice_id = AND asset_invoice_item_id = `. With a value that changes from row to row the database goes by the index's statistics, and where nearly every asset has `asset_invoice_id` 0 they say a lookup finds every asset - so it read the whole assets table once a hardware line. On a test install with 25,000 assets, none from an invoice, and 21,000 hardware lines, the dashboard's count took 22.9 seconds and read 185 million rows; it now takes 21 ms and reads 80,000. `asset_invoice_id > 0` is a range, which the database measures on the index itself, so the join reads only the assets made from an invoice whatever the statistics say. Run with none, 3,000 and 21,000 of the 25,000 assets made from an invoice: the same counts as before each time, and 21 to 32 ms. The queries that look assets up by one invoice (`productInvoiceHardware()`, the invoice PDF's serials) give the database a constant and were never affected. +- Database 3.6.6 (`admin/database_updates/3.6.6.php`) adds three indexes, each only where it isn't there already, and `db.sql` has them for new installs: `logs` (`log_user_id`, `log_type`, `log_action`) as `log_user_type`; `logs` (`log_type`, `log_action`, `log_user_id`, `log_ip`, `log_created_at`) as `log_type`; `tickets` (`ticket_invoice_id`). An index that can't be added is left out, with a warning in App Logs and the PHP error log, and the update goes on. The reasoning for each is in the migration's comment. +- Admin > Audit Logs: the list drops `SQL_CALC_FOUND_ROWS` and takes its count from its view's. `auditLogViewCounts()` (`functions/audit_log.php`) counts the views with one `GROUP BY log_type, log_action` - each view's where is on those two columns alone, so it reads `KEY log_type` and no entry - in place of a `SUM()` of every view's condition over every entry, and joins users and clients only for a search. It also hands back the types and actions in the view on show, and `auditLogKindsSql()` turns them into a condition on `KEY log_type`, so a view or an action with few entries in it (Deletions) is found on the index, not by reading the log back from the newest until 50 turn up; a view with none skips the list query. The Who and Client lists, and the type and action lists, come from `auditLogDistinctSql()` - a `GROUP BY` on the index that leads with the column - in place of `EXISTS (SELECT 1 FROM logs ...)` a user or client and `SELECT DISTINCT` over the log; `admin/modals/audit_log/audit_log_export.php` uses the same. +- Admin > Users' last sign-in and last portal sign-in, and the New IP check on a sign-in, are each one step into `KEY log_user_type`; the queries are as they were. Admin > Sign-ins reads only `KEY log_type` for every query that lists or counts: the emails with no account an address tried (`unknown`, which needs `log_description`) are counted in their own query for the page's addresses. Every sort ends on `log_ip` or `log_user_id`. +- `listNewestFirst()` (`functions/app.php`) pages a list sorted by date newest first from the newest dates back: the page's query with its date from a month ago, then a year, then four, and only then all of it, kept as soon as the rows found fill the page - every row older than the cutoff sorts after every row newer than it, so the page is the same. On a union the date is handed down to each table and its date index (`payment_date`, `revenue_date`, `expense_date`). Transactions and Income use it in their default order - Transactions only without the running balance, which is a total over an account's whole history - and take their count from their pill's, with `SQL_CALC_FOUND_ROWS` gone. `incomeUnionSql(false)` leaves out the clients, accounts and categories joins, with their names NULL, for Income's pill counts and total when there is no search. +- Invoices: `$paid_join`, a `LEFT JOIN` to every payment grouped by invoice, is replaced by `$paid_sql`, a lookup for each invoice that needs it - the page's 25, and for the money along the top only the unpaid ones, as `CASE WHEN invoice_status IN (...) AND balance > 0` stops before the lookup on the rest. The list's count is its pill's. +- Lookups a row by a column that most rows leave at 0 now also say the column is `> 0`: the project lookups (`PROJECT_TICKETS_SQL` in `functions/project.php`, used by `projectLastActivitySql()`, `projectSecondsSql()` and the Stalled and Ready to close views, and the ticket counts in client onboarding) and the expense refunds in `functions/expenses.php`. With a value that changes row to row the database goes by the index's statistics; where nearly every row has 0 they say a lookup finds every row and it read the whole table for each - with no ticket on a project, Projects read 33.6 million rows. A range on a constant it measures on the index itself, and that holds it to the index whatever the statistics say. Found, with the rest, by loading all 233 agent, report and admin pages on the eight-year test install with the slow query log on; afterwards none takes over half a second. Old and new code were then run side by side against the same database over 103 filter, sort and page combinations of the pages changed, and at 500 rows a page: the same rows, counts and totals in every one, Sign-ins' tied rows aside. ## [26.11.7] diff --git a/admin/audit_logs.php b/admin/audit_logs.php index 393f5a479..e6c7d25b6 100644 --- a/admin/audit_logs.php +++ b/admin/audit_logs.php @@ -39,30 +39,31 @@ $view = isset($_GET['view']) && is_string($_GET['view']) && isset($views[$_GET[' $filter_src = array_merge($_GET, ['dtf' => $dtf, 'dtt' => $dtt, 'view' => $view]); $joins = auditLogJoinsSql(); -$sql = mysqli_query($mysqli, "SELECT SQL_CALC_FOUND_ROWS log_id, log_type, log_action, log_description, log_ip, log_user_agent, log_created_at, log_client_id, log_user_id, log_entity_id, - user_name, user_avatar, user_type, client_name - FROM logs $joins - WHERE " . auditLogFiltersSql($filter_src) . " - ORDER BY {$sortable_columns[$sort]} - LIMIT $record_from, $record_to"); -$num_rows = mysqli_fetch_row(mysqli_query($mysqli, "SELECT FOUND_ROWS()")); +// Each view counted for the rest of the filters (functions/audit_log.php). The list's own count +// is its view's, where SQL_CALC_FOUND_ROWS counted every entry the filters let through before +// the first page came back. +$counts = auditLogViewCounts($filter_src, $view, $view_kinds); +$num_rows = [$counts[$view]]; +// The page of entries. A view, or an action, as the types and actions in it - the same entries, +// found on KEY log_type instead of by reading the log back from the newest until enough turn up. +$kinds_sql = ($view !== '' || (is_string($_GET['action'] ?? null) && trim($_GET['action']) !== '')) ? ' AND ' . auditLogKindsSql($view_kinds) : ''; $rows = []; -while ($row = mysqli_fetch_assoc($sql)) { - $rows[] = $row; +if ($num_rows[0] > 0) { + $sql = mysqli_query($mysqli, "SELECT log_id, log_type, log_action, log_description, log_ip, log_user_agent, log_created_at, log_client_id, log_user_id, log_entity_id, + user_name, user_avatar, user_type, client_name + FROM logs $joins + WHERE " . auditLogFiltersSql($filter_src) . $kinds_sql . " + ORDER BY {$sortable_columns[$sort]} + LIMIT $record_from, $record_to"); + while ($row = mysqli_fetch_assoc($sql)) { + $rows[] = $row; + } } $records = auditLogRecords($rows); $new_ips = signInNewIpIds($rows); $kinds = auditLogKinds(); -// Each view counted for the rest of the filters -$count_sql = []; -foreach ($views as $definition) { - $count_sql[] = 'COALESCE(SUM(' . ($definition['where'] === '' ? '1' : $definition['where']) . '), 0)'; -} -$counts = array_combine(array_keys($views), array_map('intval', mysqli_fetch_row(mysqli_query($mysqli, "SELECT " . implode(', ', $count_sql) . " - FROM logs $joins WHERE " . auditLogFiltersSql(array_merge($filter_src, ['view' => ''])))))); - $client_filter = is_scalar($_GET['client'] ?? null) ? intval($_GET['client']) : 0; $user_filter = is_string($_GET['user'] ?? null) ? $_GET['user'] : ''; $type_filter = is_string($_GET['type'] ?? null) ? $_GET['type'] : ''; @@ -141,7 +142,7 @@ $view_empty = [ @@ -152,7 +153,7 @@ $view_empty = [ @@ -178,7 +179,7 @@ $view_empty = [ @@ -101,7 +101,7 @@ ob_start(); diff --git a/admin/sign_ins.php b/admin/sign_ins.php index 1d34de089..cb6f3aa22 100644 --- a/admin/sign_ins.php +++ b/admin/sign_ins.php @@ -30,19 +30,20 @@ $like = $q !== '' ? "AND (user_name LIKE '%$q%' OR user_email LIKE '%$q%' OR log $joins = "LEFT JOIN users ON users.user_id = log_user_id AND log_user_id > 0"; // Only these sort, for the view on show +// Each ends on the address or the person, so rows that tie keep one order from page to page $sortable_columns = $view === 'ips' ? [ 'log_ip' => "log_ip $order", - 'sign_ins' => "sign_ins $order, last_seen DESC", - 'failures' => "failures $order, last_seen DESC", - 'accounts' => "accounts $order, last_seen DESC", - 'first_seen' => "first_seen $order", - 'last_seen' => "last_seen $order", + 'sign_ins' => "sign_ins $order, last_seen DESC, log_ip", + 'failures' => "failures $order, last_seen DESC, log_ip", + 'accounts' => "accounts $order, last_seen DESC, log_ip", + 'first_seen' => "first_seen $order, log_ip", + 'last_seen' => "last_seen $order, log_ip", ] : [ - 'user_name' => "user_name $order", - 'sign_ins' => "sign_ins $order, last_seen DESC", - 'failures' => "failures $order, last_seen DESC", - 'ips' => "ips $order, last_seen DESC", - 'last_seen' => "last_seen $order", + 'user_name' => "user_name $order, log_user_id", + 'sign_ins' => "sign_ins $order, last_seen DESC, log_user_id", + 'failures' => "failures $order, last_seen DESC, log_user_id", + 'ips' => "ips $order, last_seen DESC, log_user_id", + 'last_seen' => "last_seen $order, log_user_id", ]; if (!isset($sortable_columns[$sort])) { $sort = 'last_seen'; @@ -53,7 +54,6 @@ if ($view === 'ips') { $sql = mysqli_query($mysqli, "SELECT SQL_CALC_FOUND_ROWS log_ip, SUM(log_action = 'Success') AS sign_ins, SUM($failed) AS failures, SUM(log_action = 'Blocked') AS blocked, COUNT(DISTINCT NULLIF(log_user_id, 0)) AS accounts, - COUNT(DISTINCT IF(log_user_id = 0 AND log_action = 'Failed', log_description, NULL)) AS unknown, MIN(log_created_at) AS first_seen, MAX(log_created_at) AS last_seen, MAX(log_id) AS last_id FROM logs $joins WHERE $types AND log_ip <> '' $range $like @@ -97,6 +97,15 @@ if ($view === 'ips') { $last_devices[$row['log_ip']] = auditLogDevice($row['log_user_agent']); } } + // The emails with no account each address tried, for the addresses on the page - here and not + // in the list's own query, which then reads nothing but KEY log_type (the description, + // where the email is, isn't in it) + $unknown_emails = []; + $sql = mysqli_query($mysqli, "SELECT log_ip, COUNT(DISTINCT log_description) FROM logs + WHERE log_ip IN ($ip_list) AND $types AND log_action = 'Failed' AND log_user_id = 0 $range GROUP BY log_ip"); + while ($row = mysqli_fetch_row($sql)) { + $unknown_emails[$row[0]] = intval($row[1]); + } $turned_away = []; $sql = mysqli_query($mysqli, "SELECT log_ip, COUNT(*) FROM logs WHERE log_ip IN ($ip_list) AND $types AND $failed AND log_created_at > NOW() - INTERVAL " . SIGN_IN_WINDOW_MINUTES . " MINUTE GROUP BY log_ip HAVING COUNT(*) >= " . SIGN_IN_LOCKOUT); @@ -221,7 +230,7 @@ $audit_url = function (array $query) use ($dates_query) { foreach ($rows as $row) { $ip = (string) $row['log_ip']; $people = $tried[$ip] ?? []; - $unknown = intval($row['unknown']); ?> + $unknown = $unknown_emails[$row['log_ip']] ?? 0; ?>
diff --git a/agent/client_onboarding/kickoff.php b/agent/client_onboarding/kickoff.php index 2e4fc8752..aac4fc97c 100644 --- a/agent/client_onboarding/kickoff.php +++ b/agent/client_onboarding/kickoff.php @@ -12,7 +12,7 @@ defined('FROM_CLIENT_ONBOARDING') || die("Direct file access is not allowed"); $co_projects = []; $co_sql = mysqli_query($mysqli, "SELECT project_id, project_prefix, project_number, project_name, project_due, project_completed_at, - (SELECT COUNT(*) FROM tickets WHERE ticket_project_id = project_id) AS tickets + (SELECT COUNT(*) FROM tickets WHERE ticket_project_id = project_id AND ticket_project_id > 0) AS tickets FROM projects WHERE project_client_id = $client_id AND project_archived_at IS NULL ORDER BY project_id DESC"); while ($co_project = mysqli_fetch_assoc($co_sql)) { $co_projects[] = $co_project; diff --git a/agent/income.php b/agent/income.php index 92fa8556a..5f2edf1fd 100644 --- a/agent/income.php +++ b/agent/income.php @@ -63,28 +63,32 @@ $where = incomeFiltersSql($filter_src, $summary, false); $show_query = $show !== '' ? 'AND ' . $shows[$show]['where'] : ''; $union = incomeUnionSql(); -$sql = mysqli_query($mysqli, "SELECT SQL_CALC_FOUND_ROWS income.* FROM ($union) AS income - WHERE $where $show_query - ORDER BY {$sortable_columns[$sort]} $order, income.income_created_at DESC, income.income_id DESC LIMIT $record_from, $record_to"); -$num_rows = mysqli_fetch_row(mysqli_query($mysqli, "SELECT FOUND_ROWS()")); -$rows = []; -while ($row = mysqli_fetch_assoc($sql)) { - $rows[] = $row; -} - -// The pills, and the total of what is listed +// The pills, and the total of what is listed - without the names unless a search looks at them $count_sql = []; foreach ($shows as $key => $s) { $count_sql[] = $key === '' ? 'COUNT(*)' : "COALESCE(SUM({$s['where']}), 0)"; } $row = mysqli_fetch_row(mysqli_query($mysqli, "SELECT " . implode(', ', $count_sql) . ", COALESCE(SUM(CASE WHEN 1 $show_query THEN income.income_amount END), 0) - FROM ($union) AS income WHERE $where")); + FROM (" . incomeUnionSql(trim((string) ($_GET['q'] ?? '')) !== '') . ") AS income WHERE $where")); $counts = []; foreach (array_keys($shows) as $i => $key) { $counts[$key] = intval($row[$i]); } $listed_total = floatval($row[count($shows)]); +// The page. Its count is its pill's - SQL_CALC_FOUND_ROWS counted every line again - and in date +// order, newest first, it is read from the newest dates back (listNewestFirst()) +$num_rows = [$counts[$show]]; +$income_page = function ($and) use ($mysqli, $union, $where, $show_query, $sortable_columns, $sort, $order, $record_from, $record_to) { + $sql = mysqli_query($mysqli, "SELECT income.* FROM ($union) AS income + WHERE $where $show_query $and + ORDER BY {$sortable_columns[$sort]} $order, income.income_created_at DESC, income.income_id DESC LIMIT $record_from, $record_to"); + return mysqli_fetch_all($sql, MYSQLI_ASSOC); +}; +$rows = $sort === 'income_date' && $order === 'DESC' + ? listNewestFirst($income_page, 'income.income_date', min($record_to, $num_rows[0] - $record_from)) + : $income_page(''); + $stats = incomeStats($client_page ? $client_id : 0); $money = function ($amount, $currency = '') { return escapeHtml(billingMoney($amount, $currency)); diff --git a/agent/invoices.php b/agent/invoices.php index 0a8915e19..50b7df73c 100644 --- a/agent/invoices.php +++ b/agent/invoices.php @@ -77,30 +77,15 @@ if (!isset($sortable_columns[$sort])) { $sort = 'invoice_number'; } -$paid_join = "LEFT JOIN (SELECT payment_invoice_id, SUM(payment_amount) AS paid FROM payments GROUP BY payment_invoice_id) AS invoice_payments ON payment_invoice_id = invoice_id"; +// What an invoice has been paid - looked up for the invoices that need it (the page's, and the +// unpaid ones for the money along the top) where a LEFT JOIN to every payment grouped by invoice +// added up all of them on every load +$paid_sql = "(SELECT COALESCE(SUM(payment_amount), 0) FROM payments WHERE payment_invoice_id = invoices.invoice_id AND payment_invoice_id > 0)"; $scope_query = clientScopeSql('invoice_client_id'); $search_query = "AND (CONCAT(invoice_prefix, invoice_number) LIKE '%$q%' OR invoice_scope LIKE '%$q%' OR client_name LIKE '%$q%' OR invoice_status LIKE '%$q%' OR invoice_amount LIKE '%$q%' OR category_name LIKE '%$q%')"; $date_query = "AND DATE(invoice_date) BETWEEN '$dtf' AND '$dtt'"; $filters = "$scope_query $client_query $recurring_query $hardware_query $category_query $date_query $search_query"; -$sql = mysqli_query($mysqli, "SELECT SQL_CALC_FOUND_ROWS invoices.*, client_name, category_name, category_color, - recurring_invoice_prefix, recurring_invoice_number, - COALESCE(invoice_payments.paid, 0) AS paid, invoice_amount - COALESCE(invoice_payments.paid, 0) AS balance - FROM invoices - LEFT JOIN clients ON invoice_client_id = client_id - LEFT JOIN categories ON invoice_category_id = category_id - LEFT JOIN recurring_invoices ON invoice_recurring_invoice_id = recurring_invoice_id - $paid_join - WHERE 1 $filters $status_query - ORDER BY {$sortable_columns[$sort]} $order, invoice_id DESC LIMIT $record_from, $record_to"); - -$num_rows = mysqli_fetch_row(mysqli_query($mysqli, "SELECT FOUND_ROWS()")); - -$invoices = []; -while ($row = mysqli_fetch_assoc($sql)) { - $invoices[] = $row; -} - // How many of each, for the pills - what the filters above leave, before the pill $count_sql = []; foreach ($statuses as $key => $s) { @@ -113,6 +98,23 @@ foreach (array_keys($statuses) as $i => $key) { $counts[$key] = intval($row[$i]); } +// The page - its count is its pill's (SQL_CALC_FOUND_ROWS counted the same rows again) +$num_rows = [$counts[$status]]; +$sql = mysqli_query($mysqli, "SELECT invoices.*, client_name, category_name, category_color, + recurring_invoice_prefix, recurring_invoice_number, + $paid_sql AS paid, invoice_amount - $paid_sql AS balance + FROM invoices + LEFT JOIN clients ON invoice_client_id = client_id + LEFT JOIN categories ON invoice_category_id = category_id + LEFT JOIN recurring_invoices ON invoice_recurring_invoice_id = recurring_invoice_id + WHERE 1 $filters $status_query + ORDER BY {$sortable_columns[$sort]} $order, invoice_id DESC LIMIT $record_from, $record_to"); + +$invoices = []; +while ($row = mysqli_fetch_assoc($sql)) { + $invoices[] = $row; +} + // The money along the top - everything in this scope, not narrowed by search, date or pill $money_scope = "$scope_query $client_query $recurring_query"; $money = mysqli_fetch_assoc(mysqli_query($mysqli, "SELECT @@ -122,8 +124,8 @@ $money = mysqli_fetch_assoc(mysqli_query($mysqli, "SELECT COALESCE(SUM(CASE WHEN $open_sql AND balance > 0 AND invoice_due < CURDATE() THEN 1 END), 0) AS overdue_count, COALESCE(SUM(CASE WHEN invoice_status = 'Draft' THEN invoice_amount END), 0) AS draft, COALESCE(SUM(CASE WHEN invoice_status = 'Draft' THEN 1 END), 0) AS draft_count - FROM (SELECT invoice_status, invoice_due, invoice_amount, invoice_amount - COALESCE(invoice_payments.paid, 0) AS balance - FROM invoices $paid_join WHERE 1 $money_scope) AS scoped")); + FROM (SELECT invoice_status, invoice_due, invoice_amount, invoice_amount - $paid_sql AS balance + FROM invoices WHERE 1 $money_scope) AS scoped")); $collected = mysqli_fetch_assoc(mysqli_query($mysqli, "SELECT COALESCE(SUM(payment_amount), 0) AS amount, COUNT(*) AS payments FROM payments JOIN invoices ON payment_invoice_id = invoice_id WHERE payment_date >= CURDATE() - INTERVAL 30 DAY $money_scope")); diff --git a/agent/transactions.php b/agent/transactions.php index 3e340832a..f4ee5c2c6 100644 --- a/agent/transactions.php +++ b/agent/transactions.php @@ -75,12 +75,6 @@ $show_query = $show !== '' ? 'AND ' . $shows[$show]['where'] : ''; // The balance after each line only where it shows - one account, in date order $ledger = accountLedgerSql($account_filter, $balance_visible); -$sql = mysqli_query($mysqli, "SELECT SQL_CALC_FOUND_ROWS ledger.* FROM ($ledger) AS ledger WHERE $where $show_query ORDER BY $order_sql LIMIT $record_from, $record_to"); -$num_rows = mysqli_fetch_row(mysqli_query($mysqli, "SELECT FOUND_ROWS()")); -$rows = []; -while ($row = mysqli_fetch_assoc($sql)) { - $rows[] = accountLedgerVisible($row); -} // The pills, and the money in and out of what is listed - and, on every account's // transfers, what was moved, what paid off cards and what payment processors paid out. @@ -107,6 +101,19 @@ $counts = []; foreach (array_keys($shows) as $i => $key) { $counts[$key] = intval($row[$i]); } + +// The page. Its count is its pill's - SQL_CALC_FOUND_ROWS counted every line again. In date +// order, newest first, without the balance after each line, it is read from the newest dates back +// (listNewestFirst()); the balance is a running total over the account's whole history, so with +// it the whole ledger is still read. +$num_rows = [$counts[$show]]; +$ledger_page = function ($and) use ($mysqli, $ledger, $where, $show_query, $order_sql, $record_from, $record_to) { + $sql = mysqli_query($mysqli, "SELECT ledger.* FROM ($ledger) AS ledger WHERE $where $show_query $and ORDER BY $order_sql LIMIT $record_from, $record_to"); + return mysqli_fetch_all($sql, MYSQLI_ASSOC); +}; +$rows = array_map('accountLedgerVisible', $sort === 'txn_date' && $order === 'DESC' && !$balance_visible + ? listNewestFirst($ledger_page, 'ledger.txn_date', min($record_to, $num_rows[0] - $record_from)) + : $ledger_page('')); $n = count($shows); $total_in = round(floatval($row[$n]), 2); $total_out = round(floatval($row[$n + 1]), 2); diff --git a/db.sql b/db.sql index f75f13368..2bb8c1932 100644 --- a/db.sql +++ b/db.sql @@ -2017,7 +2017,9 @@ CREATE TABLE `logs` ( KEY `log_created_at` (`log_created_at`), KEY `log_user_id` (`log_user_id`,`log_client_id`), KEY `log_client_id` (`log_client_id`,`log_created_at`), - KEY `log_entity_id` (`log_entity_id`,`log_type`) + KEY `log_entity_id` (`log_entity_id`,`log_type`), + KEY `log_user_type` (`log_user_id`,`log_type`,`log_action`), + KEY `log_type` (`log_type`,`log_action`,`log_user_id`,`log_ip`,`log_created_at`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci; /*!40101 SET character_set_client = @saved_cs_client */; @@ -4163,7 +4165,8 @@ CREATE TABLE `tickets` ( KEY `ticket_resolved_at` (`ticket_resolved_at`), KEY `ticket_closed_at` (`ticket_closed_at`), KEY `ticket_created_at` (`ticket_created_at`), - KEY `ticket_billable` (`ticket_billable`,`ticket_invoice_id`,`ticket_status`,`ticket_client_id`,`ticket_archived_at`) + KEY `ticket_billable` (`ticket_billable`,`ticket_invoice_id`,`ticket_status`,`ticket_client_id`,`ticket_archived_at`), + KEY `ticket_invoice_id` (`ticket_invoice_id`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci; /*!40101 SET character_set_client = @saved_cs_client */; diff --git a/functions/app.php b/functions/app.php index e713205ad..8de0b24b4 100644 --- a/functions/app.php +++ b/functions/app.php @@ -243,6 +243,36 @@ function getFieldById($table, $id, $field) { // Recursive function to display folder options - Used in folders files and documents +/* + * A page of a list in newest-first date order, read from the newest dates + * back. $page(string $and) runs the list's query with $and added to its + * WHERE and returns the page's rows; $date_column is the list's date, which + * must be what the list sorts by first, descending; $needed is how many rows + * the page has - min(the page size, the list's count less the rows before + * the page). + * + * Tried over the last month, then the last year, then four, and only then the + * lot. Every row older than a cutoff sorts after every row newer than it, so + * when the rows from the cutoff on fill the page, the page is the same as the + * whole list's would be. A list over a union (Transactions, Income) has the + * database build and sort every line of it to give back 25; a date on the + * union is handed down to each table, where its date index finds the month's + * lines alone. + */ +function listNewestFirst(callable $page, $date_column, $needed) { + $needed = intval($needed); + if ($needed <= 0) { + return []; + } + foreach (['-1 month', '-1 year', '-4 years'] as $back) { + $rows = $page("AND $date_column >= '" . date('Y-m-d', strtotime($back)) . "'"); + if (count($rows) >= $needed) { + return $rows; + } + } + return $page(''); +} + function displayFolderOptions($parent_folder_id, $client_id, $indent = 0) { global $mysqli; diff --git a/functions/audit_log.php b/functions/audit_log.php index c7df2cfc2..102969e32 100644 --- a/functions/audit_log.php +++ b/functions/audit_log.php @@ -176,6 +176,9 @@ function auditLogActionPill($action) { * app log entries cleared, and changes to users, roles, API keys, the master * key, settings and backups; data out is passwords looked at, exports, * downloads and shared links. + * + * Each where is on log_type and log_action and nothing else - + * auditLogViewCounts() counts them from those two alone. */ function auditLogViews() { $signins = "log_type IN ('Login', 'Login 2FA', 'Logout', 'Client Login')"; @@ -261,6 +264,68 @@ function auditLogFiltersSql(array $src, &$summary = []) { } /** The joins the filters need - who did it, and the client. */ +/* + * How many entries each view (auditLogViews()) holds for the rest of the + * filters in $src - [view => count], '' the lot - and $kinds, the types and + * actions in $view_for as [log_type => [log_action, ...]]. + * + * Counted by type and action and then added up, where a SUM() of each view's + * condition over every entry was: a view's where is on log_type and + * log_action alone, so the counts come from KEY log_type, without reading an + * entry. The users and clients joins only come in for a search, which looks + * at their names - otherwise a LEFT JOIN on their keys counts the same. + */ +function auditLogViewCounts(array $src, $view_for = '', &$kinds = []) { + global $mysqli; + $views = auditLogViews(); + $flags = []; + foreach ($views as $key => $definition) { + $flags[] = ($definition['where'] === '' ? '1' : "({$definition['where']})") . " AS `view_$key`"; + } + $q = is_scalar($src['q'] ?? null) ? trim((string) $src['q']) : ''; + $joins = $q !== '' ? auditLogJoinsSql() : ''; + $sql = mysqli_query($mysqli, "SELECT log_type, log_action, COUNT(*) AS n, " . implode(', ', $flags) . " FROM logs $joins + WHERE " . auditLogFiltersSql(array_merge($src, ['view' => ''])) . " + GROUP BY log_type, log_action"); + $counts = array_fill_keys(array_keys($views), 0); + $kinds = []; + while ($row = mysqli_fetch_assoc($sql)) { + foreach (array_keys($views) as $key) { + if (intval($row["view_$key"])) { + $counts[$key] += intval($row['n']); + if ($key === $view_for) { + $kinds[$row['log_type']][] = $row['log_action']; + } + } + } + } + return $counts; +} + +/* + * $kinds from auditLogViewCounts() as SQL - the entries of those types and + * actions, by the index, where a view's condition or an action alone isn't + * something the index can find. '0' for none. + */ +function auditLogKindsSql(array $kinds) { + $or = []; + foreach ($kinds as $type => $actions) { + $or[] = "(log_type = '" . escapeSql($type) . "' AND log_action IN ('" . implode("', '", array_map('escapeSql', $actions)) . "'))"; + } + return $or ? '(' . implode(' OR ', $or) . ')' : '0'; +} + +/* + * The values of $columns (log_user_id, log_client_id, log_type, or log_type, + * log_action) found in the log, each once - as a GROUP BY on the index that + * leads with them, so the database steps from one value to the next on the + * index and reads nothing in between. EXISTS (SELECT 1 FROM logs ...) a user + * or a client, and DISTINCT, both read the whole log. + */ +function auditLogDistinctSql($columns) { + return "SELECT $columns FROM logs GROUP BY $columns"; +} + function auditLogJoinsSql() { return "LEFT JOIN users ON user_id = log_user_id AND log_user_id > 0 LEFT JOIN clients ON client_id = log_client_id AND log_client_id > 0"; } diff --git a/functions/client_onboarding.php b/functions/client_onboarding.php index 30d35b5be..78373051b 100644 --- a/functions/client_onboarding.php +++ b/functions/client_onboarding.php @@ -211,7 +211,7 @@ function clientOnboardingSteps($client_id, $row = null) { break; case 'kickoff': $project = mysqli_fetch_assoc(mysqli_query($mysqli, "SELECT project_id, project_prefix, project_number, project_name, - (SELECT COUNT(*) FROM tickets WHERE ticket_project_id = project_id) AS tickets + (SELECT COUNT(*) FROM tickets WHERE ticket_project_id = project_id AND ticket_project_id > 0) AS tickets FROM projects WHERE project_client_id = $client_id AND project_archived_at IS NULL ORDER BY project_id = " . intval($row['onboarding_project_id'] ?? 0) . " DESC, project_id DESC LIMIT 1")); $done = $done || $project !== null; diff --git a/functions/expenses.php b/functions/expenses.php index cd154d50e..3db8a36c5 100644 --- a/functions/expenses.php +++ b/functions/expenses.php @@ -42,8 +42,8 @@ function expenseRow($expense_id, $scoped = true) { refunded.expense_date AS refund_of_date, refunded.expense_amount AS refund_of_amount, refunded.expense_description AS refund_of_description, payer.user_name AS paid_by_name, paid_back.reimbursement_date, paid_back_account.account_name AS reimbursed_account_name, paying_out.reimbursement_id AS reimbursement_out_id, (SELECT u.user_name FROM users AS u WHERE u.user_id = paying_out.reimbursement_user_id) AS reimbursed_to_name, - (SELECT COALESCE(SUM(-r.expense_amount), 0) FROM expenses AS r WHERE r.expense_refund_of_id = expenses.expense_id) AS refunded_amount, - (SELECT COUNT(*) FROM expenses AS r WHERE r.expense_refund_of_id = expenses.expense_id) AS refund_count + (SELECT COALESCE(SUM(-r.expense_amount), 0) FROM expenses AS r WHERE r.expense_refund_of_id = expenses.expense_id AND r.expense_refund_of_id > 0) AS refunded_amount, + (SELECT COUNT(*) FROM expenses AS r WHERE r.expense_refund_of_id = expenses.expense_id AND r.expense_refund_of_id > 0) AS refund_count FROM expenses LEFT JOIN vendors ON vendor_id = expenses.expense_vendor_id LEFT JOIN categories ON category_id = expenses.expense_category_id @@ -1019,7 +1019,7 @@ function expenseUnbilled($client_id = 0, array $ids = []) { . ' ' . clientScopeSql('expense_client_id'); $rows = []; $sql = mysqli_query($mysqli, "SELECT expenses.*, vendor_name, category_name, client_name, client_abbreviation, - (SELECT COALESCE(SUM(-r.expense_amount), 0) FROM expenses AS r WHERE r.expense_refund_of_id = expenses.expense_id) AS refunded, + (SELECT COALESCE(SUM(-r.expense_amount), 0) FROM expenses AS r WHERE r.expense_refund_of_id = expenses.expense_id AND r.expense_refund_of_id > 0) AS refunded, DATEDIFF(CURDATE(), expense_date) AS waiting FROM expenses JOIN clients ON client_id = expense_client_id @@ -1062,7 +1062,7 @@ function expenseUnbilledCount($client_id = 0) { $client_id = intval($client_id); $row = mysqli_fetch_row(mysqli_query($mysqli, "SELECT COUNT(*), COALESCE(SUM(ROUND((expense_amount - refunded) * (1 + expense_markup / 100), 2)), 0) FROM (SELECT expense_amount, expense_markup, - (SELECT COALESCE(SUM(-r.expense_amount), 0) FROM expenses AS r WHERE r.expense_refund_of_id = expenses.expense_id) AS refunded + (SELECT COALESCE(SUM(-r.expense_amount), 0) FROM expenses AS r WHERE r.expense_refund_of_id = expenses.expense_id AND r.expense_refund_of_id > 0) AS refunded FROM expenses WHERE " . expenseUnbilledSql() . ($client_id ? " AND expense_client_id = $client_id" : '') . ' ' . clientScopeSql('expense_client_id') . ") AS waiting WHERE expense_amount - refunded > 0")); @@ -1238,7 +1238,7 @@ function expenseOnInvoice($invoice_id) { $invoice_id = intval($invoice_id); $rows = []; $sql = mysqli_query($mysqli, "SELECT expenses.*, vendor_name, category_name, item_subtotal, - (SELECT COALESCE(SUM(-r.expense_amount), 0) FROM expenses AS r WHERE r.expense_refund_of_id = expenses.expense_id) AS refunded + (SELECT COALESCE(SUM(-r.expense_amount), 0) FROM expenses AS r WHERE r.expense_refund_of_id = expenses.expense_id AND r.expense_refund_of_id > 0) AS refunded FROM expenses LEFT JOIN vendors ON vendor_id = expense_vendor_id LEFT JOIN categories ON category_id = expense_category_id diff --git a/functions/income.php b/functions/income.php index 69083cb25..fb22a86fa 100644 --- a/functions/income.php +++ b/functions/income.php @@ -58,30 +58,39 @@ function incomeOnlineSql($reference = 'payment_reference') { * income_reference, income_account_id, account_name, account_archived_at * and income_online. Select from it as a derived table. */ -function incomeUnionSql() { - return "SELECT 'payment' AS income_type, payment_id AS income_id, payment_date AS income_date, payment_created_at AS income_created_at, - payment_invoice_id AS income_invoice_id, invoice_prefix, invoice_number, invoice_status, invoice_scope AS income_description, - COALESCE(invoice_category_id, 0) AS income_category_id, category_name, category_color, category_icon, - COALESCE(invoice_client_id, 0) AS income_client_id, client_name, payment_amount AS income_amount, payment_currency_code AS income_currency_code, - payment_method AS income_method, payment_reference AS income_reference, payment_account_id AS income_account_id, account_name, account_archived_at, - IF(" . incomeOnlineSql() . ", 1, 0) AS income_online - FROM payments - LEFT JOIN invoices ON invoice_id = payment_invoice_id +function incomeUnionSql($names = true) { + // Without $names - for counts and totals with no search - the clients, accounts and + // categories joins are left out and their names come back NULL: the ids, which the filters + // and the pills go by, are on the payment, its invoice and the revenue. One join a line + // fewer is most of what counting the lines takes. + $payment_names = $names ? " LEFT JOIN clients ON client_id = invoice_client_id LEFT JOIN accounts ON account_id = payment_account_id - LEFT JOIN categories ON category_id = invoice_category_id + LEFT JOIN categories ON category_id = invoice_category_id" : ''; + $revenue_names = $names ? " + LEFT JOIN clients ON client_id = revenue_client_id + LEFT JOIN accounts ON account_id = revenue_account_id + LEFT JOIN categories ON category_id = revenue_category_id" : ''; + $category = $names ? 'category_name, category_color, category_icon' : 'NULL AS category_name, NULL AS category_color, NULL AS category_icon'; + $client = $names ? 'client_name' : 'NULL AS client_name'; + $account = $names ? 'account_name, account_archived_at' : 'NULL AS account_name, NULL AS account_archived_at'; + return "SELECT 'payment' AS income_type, payment_id AS income_id, payment_date AS income_date, payment_created_at AS income_created_at, + payment_invoice_id AS income_invoice_id, invoice_prefix, invoice_number, invoice_status, invoice_scope AS income_description, + COALESCE(invoice_category_id, 0) AS income_category_id, $category, + COALESCE(invoice_client_id, 0) AS income_client_id, $client, payment_amount AS income_amount, payment_currency_code AS income_currency_code, + payment_method AS income_method, payment_reference AS income_reference, payment_account_id AS income_account_id, $account, + IF(" . incomeOnlineSql() . ", 1, 0) AS income_online + FROM payments + LEFT JOIN invoices ON invoice_id = payment_invoice_id $payment_names WHERE payment_archived_at IS NULL " . clientScopeSql('COALESCE(invoice_client_id, 0)') . " UNION ALL SELECT 'revenue', revenue_id, revenue_date, revenue_created_at, 0, NULL, NULL, NULL, revenue_description, - revenue_category_id, category_name, category_color, category_icon, - revenue_client_id, client_name, revenue_amount, revenue_currency_code, - revenue_payment_method, revenue_reference, revenue_account_id, account_name, account_archived_at, + revenue_category_id, $category, + revenue_client_id, $client, revenue_amount, revenue_currency_code, + revenue_payment_method, revenue_reference, revenue_account_id, $account, 0 - FROM revenues - LEFT JOIN clients ON client_id = revenue_client_id - LEFT JOIN accounts ON account_id = revenue_account_id - LEFT JOIN categories ON category_id = revenue_category_id + FROM revenues $revenue_names WHERE revenue_archived_at IS NULL AND " . incomeRevenueSql() . ' ' . clientScopeSql('revenue_client_id'); } diff --git a/functions/project.php b/functions/project.php index 3c98762a7..4f49d4bfd 100644 --- a/functions/project.php +++ b/functions/project.php @@ -240,15 +240,27 @@ function projectDateSql($input) { * ---------- progress, time and activity ---------- */ +/* + * Correlated on projects, the lookups below find a project's tickets by + * KEY ticket_project_id - and each says ticket_project_id > 0 as well, which + * changes nothing they find. A lookup by a value that changes from project + * to project goes by the index's statistics; most tickets are on no project, + * and on an install where none is yet they say every ticket has the same + * value, so the database read every ticket once for each project (Projects + * took over 5 seconds on a test install with 280,000 tickets). A range on a + * constant it measures on the index itself, and that holds it to the index. + */ +define('PROJECT_TICKETS_SQL', 'pt.ticket_project_id = projects.project_id AND pt.ticket_project_id > 0 AND pt.ticket_archived_at IS NULL'); + /** SQL: when its tickets last moved - the latest ticket update, or NULL with none. Correlated on projects. */ function projectLastActivitySql() { - return "(SELECT MAX(COALESCE(pt.ticket_updated_at, pt.ticket_created_at)) FROM tickets pt WHERE pt.ticket_project_id = projects.project_id AND pt.ticket_archived_at IS NULL)"; + return "(SELECT MAX(COALESCE(pt.ticket_updated_at, pt.ticket_created_at)) FROM tickets pt WHERE " . PROJECT_TICKETS_SQL . ")"; } /** SQL: seconds logged on its tickets. Correlated on projects. */ function projectSecondsSql() { return "(SELECT COALESCE(SUM(TIME_TO_SEC(pr.ticket_reply_time_worked)), 0) FROM ticket_replies pr JOIN tickets pt ON pt.ticket_id = pr.ticket_reply_ticket_id - WHERE pt.ticket_project_id = projects.project_id AND pr.ticket_reply_archived_at IS NULL)"; + WHERE pt.ticket_project_id = projects.project_id AND pt.ticket_project_id > 0 AND pr.ticket_reply_archived_at IS NULL)"; } /** @@ -260,8 +272,8 @@ function projectViews($with_budget = true) { global $session_user_id; $open = PROJECT_OPEN_SQL; $started = "(projects.project_start IS NULL OR projects.project_start <= CURDATE())"; - $has_tickets = "EXISTS (SELECT 1 FROM tickets rt WHERE rt.ticket_project_id = projects.project_id AND rt.ticket_archived_at IS NULL)"; - $has_open = "EXISTS (SELECT 1 FROM tickets rt WHERE rt.ticket_project_id = projects.project_id AND rt.ticket_archived_at IS NULL AND rt.ticket_resolved_at IS NULL AND rt.ticket_closed_at IS NULL)"; + $has_tickets = "EXISTS (SELECT 1 FROM tickets pt WHERE " . PROJECT_TICKETS_SQL . ")"; + $has_open = "EXISTS (SELECT 1 FROM tickets pt WHERE " . PROJECT_TICKETS_SQL . " AND pt.ticket_resolved_at IS NULL AND pt.ticket_closed_at IS NULL)"; $views = [ '' => ['label' => 'Active', 'icon' => 'fa-project-diagram', 'where' => $open, 'title' => 'Not completed or archived'], 'mine' => ['label' => 'Mine', 'icon' => 'fa-user-tie', 'where' => "$open AND projects.project_manager = " . intval($session_user_id), 'title' => 'Active projects you manage'],