diff --git a/CHANGELOG.md b/CHANGELOG.md index 865b44240..46e3501ec 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -92,6 +92,7 @@ Update from Maintenance > Update — Queue Update hands the job to cron, which u - Maintenance > Cron: the new Cron Watch job, every 5 minutes, emails your admins when a job fails, gets stuck or falls behind - once when it starts, again once a day while it lasts, and once more when it's fixed. The Alerts card turns the emails off, sends them to other addresses instead, and sends a test. From Mountaineer IT's build of ITFlow. - Maintenance > Cron: an email can't say cron has stopped, since cron sends it. The Alerts card has a private health address for an uptime monitor such as Uptime Kuma or UptimeRobot: it answers OK while all is well and an error when cron isn't running or a job is in trouble. Check it every 5 minutes. Make a new address if the old one gets out. From Mountaineer IT's build of ITFlow. - Development mode, for a test copy of your ITFlow: turn it on in Admin > Backup before you restore production's backup onto a test server, and nothing reaches your clients, your team or the systems production talks to. Every email goes to a test inbox you give, with who it was for at the front of its subject. Texts wait in the queue, the ticket mailbox isn't read, AI Triage doesn't run, Stripe charges nothing (autopay included), nothing is done to RMM devices, log entries aren't sent to your syslog server, a managed ITFlow doesn't check in with its manager, and telemetry isn't sent. A red DEVELOPMENT badge shows in the top bar for everyone. It's kept in config.php rather than the database, so a restore can't turn it off. Never turn it on in production. From Mountaineer IT's build of ITFlow. +- Maintenance > Backup: restore a backup from the page - Restore in a backup's menu, or upload one under Restoring, from another ITFlow say, with its key. An upload of up to 5 GB goes up in pieces, joins the list as Uploaded and opens its Restore window. That window says what's in the backup and refuses one from a newer version, and you confirm with your password and by typing RESTORE. Cron does it within a minute: ITFlow backs itself up first, kept as Before restore so the restore can be undone, then restores the backup and brings its database up to this version. The top of the page shows it waiting (with Cancel), running, and how it went. The command line restore is still under Restoring, for a backup of any size. From Mountaineer IT's build of ITFlow. ### Bug Fixes @@ -211,6 +212,7 @@ Update from Maintenance > Update — Queue Update hands the job to cron, which u - Database 3.7.8: `ticket_replies.ticket_reply_work_start` and `ticket_reply_work_end`, in company time - `ticketWorkTimesReady()`. `add_ticket_reply` takes `work_date`, `work_start` and `work_end` (`ticketWorkRange()`; an end before the start is the next day, under 24 hours) for a note with time worked, and `ticketWorkTimeWorked()` makes the time worked their length unless it agrees to the minute; without them, or from the bulk reply, the API or email, they stay NULL - it ended when it was posted. `edit_ticket_reply` takes the same fields: changed times set the time worked, the same times with a new time worked move the start, times left as `ticketWorkTimesShown()` filled them on a note without any keep it without, and no time worked clears them. The reply box's time row is Time worked and When it was worked, above Status and Send; `agent/js/ticket_work_times.js` follows `#hours` / `#minutes` / `#seconds` from `ticket_time_tracking.js` (at least a minute), in company time from `#workTimes` `data-now`, and refuses the form in the capture phase, before the timer resets, when a note with time worked has no valid times. From Mountaineer IT's build of ITFlow. - Database 3.7.9: `config_cron_alert_email` (default 1), `config_cron_alert_to`, `config_cron_health_key` and `config_cron_alert_state` (JSON: `sent`, each problem's key and when it was last emailed; `waiting`, each job that is on and has never run and since when) - `cronHealthReady()`. `functions/cron_health.php`: `cronHealthProblems()` reads the heartbeat, `config_enable_cron` and each enabled registry job's row (not those a manager has paused) into problems with a key, kind (`not_running`, `off`, `failed`, `stuck` after `CRON_HEALTH_STUCK_MINUTES`, `overdue` - past two intervals and at least ten minutes over, or two hours past a daily time; a job never run is late once `waiting` is older than that), text and error. `includes/top_nav.php` shows them to admins as `#itflowCronHealth`. `cron/cron_watch.php` is the `cron_watch` job, every 5 minutes: `cronHealthWatch()` records `waiting`, then emails the problems that are new, a day old or gone with `cronHealthAlert()` through `cronHealthMail()` (`emailQueue()`, to `config_cron_alert_to`, else active admins - `cronHealthRecipients()`), leaving out `not_running` and its own; Admin > Mail previews it as `cron_alert`. `health.php` answers 200 `OK` or 503 with a line per problem, given `config_cron_health_key` as `?key=` or an `X-Health-Key` header, and 403 otherwise. `admin/post/cron.php` has `edit_cron_alerts`, `test_cron_alert` (to the addresses typed) and `renew_cron_health_key`, locked in the demo. From Mountaineer IT's build of ITFlow. - Development mode (`functions/dev_mode.php`): `$config_dev_mode` and `$config_dev_mail_to` in config.php - `devModeOn()`, `devModeMailTo()`. `edit_dev_mode` (`admin/post/backup.php`, locked in the demo) writes both with `devModeConfigSet()`, which replaces each line or adds them at the end and invalidates opcache; when config.php can't be written, Admin > Backup shows the lines to add. `sendQueueEmail()` (`cron/mail_queue.php`) sends to `devModeMailTo()` with `[DEV to ]` in front of the subject, and with no test inbox the queue is cancelled as in the demo. `includes/stripe_init.php` calls `devModeBlockStripe()`, a `\Stripe\ApiRequestor::setHttpClient()` client that throws `ApiConnectionException`. `smsSendQueued()` leaves a text queued, `rmmDeviceAct()` refuses, `managedHttpJson()` answers not ok, and `logForwardQueue()` and `logForwardSend()` send nothing. `cron/ticket_email_parser.php`, `ai_triage.php`, `sms_queue.php`, `payment_provider_fees.php`, `manager_checkin.php` and `managed_instances.php` stop with `cronJobStop("Development mode - ...")`; `cron/nightly_tasks.php` skips autopay and telemetry. `includes/top_nav.php` shows `[data-dev-mode]`, a link to `admin/backup.php#dev-mode` for admins. From Mountaineer IT's build of ITFlow. +- Restoring from Maintenance > Backup (`functions/backup_restore.php`): `restore_backup` (`admin/post/backup.php`) checks the password and RESTORE, then `backupRestoreQueue()` checks the archive with `backupRestoreCheck()` (`backupInspectArchive()`, refused when its database version is newer) and the disk (three times its size), writes `restore_queued.json` beside the backups - another install's key sealed with this one's (`backupRestoreSealKey()`, AES-256-GCM) - and `restore_status.json`, and sets the Backup job's `cron_job_run_now`. `cron/backup.php` calls `backupRestoreRunQueued()` after the queued backups: it deletes the request first, refuses one older than `BACKUP_RESTORE_WAIT_MINUTES`, makes a Full Backup with source `Before restore`, runs `backupRestoreArchive()`, puts that row and the restored one back in the restored `backups` (others found again by retention), marks `Running` and `Pending` rows `Failed`, runs `scripts/update_cli.php --update_db`, records the outcome and sets `$cron_dispatch_stop_cycle`. `cron/cron.php` exits before its heartbeat while `backupRestoreInProgress()`. `cancel_restore` removes a request. Uploads: `backup_upload` in one request, or `backup_upload_start`, `backup_upload_chunk` and `backup_upload_finish` from `js/backup_restore.js` (JSON, the upload in the session, pieces of `backupRestoreChunkBytes()` appended to `upload_.part`, retried smaller when a proxy refuses), then `backupRestoreAdopt()` lists it with source `Uploaded`. The Restore window is `admin/modals/backup/backup_restore.php`, opened from a row's menu or by `?restore=`; `backupMadeBy()` words a row's source. All locked in the demo. From Mountaineer IT's build of ITFlow. ## [26.11.8] diff --git a/admin/backup.php b/admin/backup.php index daf52f0f3..b6ef9bb7b 100644 --- a/admin/backup.php +++ b/admin/backup.php @@ -114,8 +114,22 @@ if (!empty($_SESSION['backup_master_key_reveal'])) { unset($_SESSION['backup_master_key_reveal']); } -// The page reloads while a backup is on its way - but never out from under a key shown once -$reload = $state === 'working' && $master_key_reveal === ''; +// A restore - waiting for cron, being done, or how the last one went, for a week (functions/backup_restore.php) +$restore_status = backupRestoreStatus(); +$restore_state = (string) ($restore_status['state'] ?? ''); +if ($restore_state === 'queued' && !backupRestoreRequest()) { + $restore_state = 'running'; // cron took the request and is about to say so +} +if ($restore_state === 'running' && !backupRestoreInProgress() && strtotime((string) ($restore_status['started'] ?? $restore_status['at'] ?? '')) < strtotime('-6 hours')) { + $restore_state = 'lost'; +} +if (in_array($restore_state, ['done', 'failed', 'cancelled'], true) && strtotime((string) ($restore_status['at'] ?? '')) < strtotime('-7 days')) { + $restore_state = ''; +} +$restore_open = intval($_GET['restore'] ?? 0); + +// The page reloads while a backup or a restore is on its way - but never out from under a key shown once +$reload = ($state === 'working' || in_array($restore_state, ['queued', 'running'], true)) && $master_key_reveal === ''; // A time as "3 hours ago", with the date and time behind it $ago = function ($datetime) { @@ -123,13 +137,7 @@ $ago = function ($datetime) { }; // Who a backup is down to, from its row $made_by = function ($backup) { - if ($backup['backup_source'] === 'Cron') { - return 'Scheduled'; - } - if ($backup['backup_source'] === 'Adopted') { - return 'Found on this server'; - } - return (string) $backup['backup_created_by'] !== '' ? 'By ' . $backup['backup_created_by'] : 'By hand'; + return backupMadeBy($backup); }; $schedule_text = $schedule_on ? "Every night at $schedule_time" : 'Off'; // A type as it reads in a sentence - "a Full Backup", "a Database Only backup" @@ -196,6 +204,38 @@ $status_looks = [ + 'info', 'running' => 'primary', 'done' => empty($restore_status['warnings']) ? 'success' : 'warning', 'failed' => 'danger', 'lost' => 'danger', 'cancelled' => 'secondary'][$restore_state]; + $restore_icon = ['queued' => 'fa-hourglass-half', 'running' => 'fa-circle-notch fa-spin', 'done' => 'fa-check-circle', 'failed' => 'fa-exclamation-triangle', 'lost' => 'fa-exclamation-triangle', 'cancelled' => 'fa-ban'][$restore_state]; + $restore_label = (string) ($restore_status['label'] ?? 'a backup'); ?> + +
> + +
+ + A restore is waiting to start - the , asked for by . + + + + Restoring the - started by . + ITFlow is backed up as it is first, then replaced. Anyone working meanwhile may see errors, and you may be signed out - sign in again once it's done. + + The last restore didn't say it finished. The was started by - the Backup job may have been stopped part way. + Check that ITFlow works, and Maintenance > Cron and App Logs for what happened. The Before restore backup, if it's in the list, is ITFlow as it was. + + 'Last restore', 'failed' => "The last restore didn't work", 'cancelled' => 'Last restore cancelled'][$restore_state] ?> + (, ) - + + +
+ + +
+
+ +
@@ -368,6 +408,11 @@ $status_looks = [ + + + @@ -587,21 +632,40 @@ $config_dev_mail_to = '
- -
+ +

Restoring

-

Restoring replaces the database - and, from a Full Backup, everything uploaded - with what's in the backup. It isn't done from this page, on purpose: a running ITFlow shouldn't drop its own tables from a browser.

-

From the command line

-
-
- -
-

Works for a backup of any size. Add --inspect to see what's in one without changing anything, and --key= for a backup from another server, which needs that server's key. A backup's menu copies its path.

-

On a new server

-

The setup wizard at /setup offers to restore a backup before the first user exists - up to here, the most PHP takes in one upload.

+

Restoring replaces the database - and, from a Full Backup, everything uploaded - with what's in the backup. Pick Restore in a backup's menu, or upload one - from another ITFlow, say. ITFlow is backed up as it is first, as Before restore, so a restore can be undone.

+
+ + +
+ + +
+
+ + + +
+
Up to . It joins the list as Uploaded, and its Restore window opens. A restore needs free space of about three times the backup's size.
+ + +
+
+ From the command line +
+
+ +
+

For a backup of any size, or when cron isn't running. Add --inspect to see what's in one without changing anything, and --key= for a backup from another server, which needs that server's key. A backup's menu copies its path. On a new server, the setup wizard at /setup offers to restore a backup before the first user exists.

+
@@ -622,7 +686,12 @@ $config_dev_mail_to = ' + + + + + Backup - restore a backup in the list (functions/backup_restore.php). What's in + * it first, then the admin's password and RESTORE typed to confirm; admin/post/backup.php, + * restore_backup, queues it for cron. + */ + +require_once '../../includes/modal_header.php'; + +[$restore_row, $restore_path] = backupRestoreStored($mysqli, intval($_GET['backup_id'] ?? 0)); +$restore_meta = null; +$restore_problem = ''; +$restore_needs_key = false; +if ($restore_row) { + [$restore_meta, $restore_problem] = backupRestoreCheck($mysqli, $restore_path, backupRestoreKey('')); + // Made with another install's key - it opens once that key is typed + $restore_needs_key = !$restore_meta; +} +$restore_can = $restore_row && ($restore_problem === '' || $restore_needs_key) && !backupRestoreInProgress() && backupEncryptionKey() !== ''; +$restore_with_uploads = !$restore_meta || in_array('uploads.zip', $restore_meta['entries'], true); + +ob_start(); + +?> + +
+ + + + + +
+ + Cron - it's only done if cron runs within " . BACKUP_RESTORE_WAIT_MINUTES . " minutes.", 'warning'); + } else { + flashAlert("Restore queued - cron starts it within a minute. ITFlow is backed up as it is first, then the backup is restored."); + } + + redirect("backup.php"); +} + +if (isset($_GET['cancel_restore'])) { + + validateCSRFToken(); + + if (backupRestoreCancel(stripslashes((string) $session_name))) { + logAudit("Backup", "Restore", "$session_name cancelled a restore before it started"); + flashAlert("Restore cancelled - nothing was changed."); + } else { + flashAlert("There's no restore waiting to start - it may have started already.", 'error'); + } + + redirect("backup.php"); +} + +// Upload a backup to restore, in one request - the form without the script (js/backup_restore.js) +if (isset($_POST['backup_upload'])) { + + validateCSRFToken(); + + $file = $_FILES['backup_file'] ?? []; + + if (($file['error'] ?? UPLOAD_ERR_NO_FILE) !== UPLOAD_ERR_OK || !is_uploaded_file($file['tmp_name'] ?? '')) { + $too_big = in_array($file['error'] ?? 0, [UPLOAD_ERR_INI_SIZE, UPLOAD_ERR_FORM_SIZE], true); + flashAlert($too_big ? "It's bigger than this server takes in one go (" . escapeHtml(backupFormatBytes(backupMaxUploadBytes())) . ") - upload it with JavaScript on, which sends it in pieces, or restore it from the command line." : "Pick a backup .zip to upload.", 'error'); + redirect("backup.php"); + } + + [$uploaded_id, $upload_message] = backupRestoreAdopt($mysqli, $file['tmp_name'], (string) ($_POST['restore_key'] ?? ''), stripslashes((string) $session_name), true); + + if (!$uploaded_id) { + flashAlert(escapeHtml($upload_message), 'error'); + redirect("backup.php"); + } + + logAudit("Backup", "Upload", "$session_name uploaded a backup to restore"); + flashAlert(escapeHtml($upload_message)); + redirect("backup.php?restore=$uploaded_id"); +} + +/* + * Upload a backup in pieces (js/backup_restore.js), so PHP's and a proxy's request limits don't + * matter. Answers JSON. The upload going on is kept in the admin's session; the pieces are put + * together beside the backups (backupRestorePartPath()). + */ +if (isset($_POST['backup_upload_start']) || isset($_POST['backup_upload_chunk']) || isset($_POST['backup_upload_finish'])) { + + validateCSRFToken(); + + header('Content-Type: application/json'); + + try { + + if (isset($_POST['backup_upload_start'])) { + + $size = intval($_POST['size'] ?? 0); + + if (!preg_match('/\.zip$/i', (string) ($_POST['name'] ?? ''))) { + throw new RuntimeException('Pick a backup .zip to upload.'); + } + if ($size <= 0 || $size > BACKUP_RESTORE_MAX_BYTES) { + throw new RuntimeException('A backup up to ' . backupFormatBytes(BACKUP_RESTORE_MAX_BYTES) . ' can be uploaded here - restore a bigger one from the command line.'); + } + + backupRestoreTidyParts(); + if (!empty($_SESSION['backup_upload']['id'])) { + @unlink(backupRestorePartPath($_SESSION['backup_upload']['id'])); + } + + $problem = backupRestoreDiskProblem($size + 512 * 1024 * 1024, 'for the upload'); + if ($problem !== '') { + throw new RuntimeException($problem); + } + + $upload_id = bin2hex(random_bytes(16)); + if (@file_put_contents(backupRestorePartPath($upload_id), '') === false) { + throw new RuntimeException("The backups folder can't be written to."); + } + @chmod(backupRestorePartPath($upload_id), 0600); + + $_SESSION['backup_upload'] = ['id' => $upload_id, 'size' => $size, 'received' => 0]; + + echo json_encode(['id' => $upload_id, 'chunk' => backupRestoreChunkBytes()]); + + } elseif (isset($_POST['backup_upload_chunk'])) { + + $upload = $_SESSION['backup_upload'] ?? null; + $piece = $_FILES['chunk'] ?? []; + + if (!$upload || !hash_equals((string) $upload['id'], (string) ($_POST['id'] ?? ''))) { + throw new RuntimeException('This upload is no longer going on - start it again.'); + } + if (($piece['error'] ?? UPLOAD_ERR_NO_FILE) !== UPLOAD_ERR_OK || !is_uploaded_file($piece['tmp_name'] ?? '')) { + throw new RuntimeException('A piece of the file didn\'t arrive (error ' . intval($piece['error'] ?? 0) . ').'); + } + if (intval($_POST['offset'] ?? -1) !== intval($upload['received'])) { + throw new RuntimeException('A piece arrived out of order - start the upload again.'); + } + + $length = intval(filesize($piece['tmp_name'])); + if ($length <= 0 || $length > backupRestoreChunkBytes() || $upload['received'] + $length > $upload['size']) { + throw new RuntimeException('A piece of the file wasn\'t the size expected - start the upload again.'); + } + + $in = fopen($piece['tmp_name'], 'rb'); + $out = fopen(backupRestorePartPath($upload['id']), 'ab'); + $copied = ($in && $out) ? stream_copy_to_stream($in, $out) : false; + $in && fclose($in); + $out && fclose($out); + + if ($copied !== $length) { + throw new RuntimeException("A piece couldn't be written to the backups folder - is the disk full?"); + } + + $_SESSION['backup_upload']['received'] += $length; + + echo json_encode(['received' => $_SESSION['backup_upload']['received']]); + + } else { + + $upload = $_SESSION['backup_upload'] ?? null; + unset($_SESSION['backup_upload']); + + if (!$upload || !hash_equals((string) $upload['id'], (string) ($_POST['id'] ?? ''))) { + throw new RuntimeException('This upload is no longer going on - start it again.'); + } + + $part = backupRestorePartPath($upload['id']); + + if (intval($upload['received']) !== intval($upload['size']) || !is_file($part) || filesize($part) !== intval($upload['size'])) { + @unlink($part); + throw new RuntimeException("The upload didn't finish - start it again."); + } + + [$uploaded_id, $upload_message] = backupRestoreAdopt($mysqli, $part, (string) ($_POST['restore_key'] ?? ''), stripslashes((string) $session_name), false); + + if (!$uploaded_id) { + throw new RuntimeException($upload_message); + } + + logAudit("Backup", "Upload", "$session_name uploaded a backup to restore"); + flashAlert(escapeHtml($upload_message)); + + echo json_encode(['redirect' => "backup.php?restore=$uploaded_id"]); + } + + } catch (Throwable $e) { + http_response_code(422); + echo json_encode(['error' => $e->getMessage()]); + } + + exit; +} diff --git a/cron/backup.php b/cron/backup.php index 45ca35958..5af782b8b 100644 --- a/cron/backup.php +++ b/cron/backup.php @@ -38,6 +38,25 @@ if ($queued > 0) { echo "Built $queued queued backup(s)\n"; } +/* + * Then a restore asked for in Maintenance > Backup (functions/backup_restore.php). The database + * is the backup's afterwards, so nothing else is done this run, and the cron dispatch ends its + * cycle here - the jobs after this one run with the next. + */ +$restore = backupRestoreRunQueued($mysqli); + +if ($restore !== null) { + [$restore_ok, $restore_message] = $restore; + echo ($restore_ok ? '' : 'Restore FAILED: ') . "$restore_message\n"; + appNotify("Backup", ($restore_ok ? 'Restore finished: ' : 'Restore failed: ') . $restore_message, "/admin/backup.php"); + logAudit("Backup", "Restore", ($restore_ok ? 'Restored from Maintenance > Backup: ' : 'A restore from Maintenance > Backup failed: ') . $restore_message); + if (!$restore_ok) { + logApp("Backup", "error", "Restore failed: $restore_message"); + } + $cron_dispatch_stop_cycle = true; + return; +} + /* * Then the scheduled one. The dispatcher only calls this script when the schedule says so, * so reaching here means a scheduled backup is due. diff --git a/cron/cron.php b/cron/cron.php index 5a607e3d0..03aeceafa 100644 --- a/cron/cron.php +++ b/cron/cron.php @@ -191,6 +191,14 @@ function cronJobFinished($mysqli, string $job_name, string $status, ?float $dura } } +// A restore is replacing the database (functions/backup_restore.php) - no job may write to tables +// being replaced. The dispatch that runs the restore carries on; the ones that start meanwhile stop +// here, and the first after it runs everything that came due. +if (backupRestoreInProgress()) { + echo "Cron: a restore is running - nothing else runs until it's done.\n"; + exit; +} + // Proof the crontab is firing, recorded before any job runs. Maintenance > Cron reads it to tell // "no job was due" apart from "nothing has run this since the server was rebuilt". mysqli_query($mysqli, "UPDATE settings SET config_cron_last_dispatch_at = '" . date('Y-m-d H:i:s') . "' WHERE company_id = 1"); diff --git a/functions.php b/functions.php index dfffcba9a..6511765d0 100644 --- a/functions.php +++ b/functions.php @@ -56,6 +56,7 @@ require_once __DIR__ . '/functions/import.php'; require_once __DIR__ . '/functions/import_kinds.php'; require_once __DIR__ . '/functions/calendar.php'; require_once __DIR__ . '/functions/backup.php'; +require_once __DIR__ . '/functions/backup_restore.php'; require_once __DIR__ . '/functions/search.php'; require_once __DIR__ . '/functions/notifications.php'; require_once __DIR__ . '/functions/dashboard.php'; diff --git a/functions/backup_restore.php b/functions/backup_restore.php new file mode 100644 index 000000000..2439b670a --- /dev/null +++ b/functions/backup_restore.php @@ -0,0 +1,404 @@ + Backup + * + * Like a backup, a restore isn't done in the page's own request: a running ITFlow shouldn't + * drop its own tables in the middle of a page load, behind a proxy that gives up waiting. The + * page checks the backup and the admin's password, then queues it (backupRestoreQueue()) and + * cron/backup.php does it (backupRestoreRunQueued()): + * - a Full Backup of ITFlow as it is first, kept as "Before restore", so it can be undone + * - then backupRestoreArchive(), which puts the database back as it was if the import fails + * - the backups list is the backup's own afterwards, so the "Before restore" backup and the + * one restored are put back in it + * - then the database updates (scripts/update_cli.php --update_db), for a backup from an + * older version + * The cron dispatch that runs it ends its cycle there, and the dispatches started while it + * works skip theirs (cron/cron.php) - the jobs mustn't write to tables being replaced. + * + * What was asked for, and how it went, are files beside the backups rather than rows: the + * restore replaces the database they would be in. restore_queued.json is taken away as the + * job starts, so a restore that stops part way is never started again unattended, and one + * that waited longer than BACKUP_RESTORE_WAIT_MINUTES - cron was stopped - isn't done at all. + * restore_status.json says how the last one went, for the page. + * + * A backup can also be uploaded to restore - one from another server, say. The browser sends it + * in pieces (js/backup_restore.js, the backup_upload_* handlers in admin/post/backup.php), so + * PHP's and a proxy's request limits don't matter, and it joins the list as "Uploaded". + */ + +const BACKUP_SOURCE_BEFORE_RESTORE = 'Before restore'; +const BACKUP_SOURCE_UPLOADED = 'Uploaded'; +// The biggest backup the browser uploads, and the most of it in one request +const BACKUP_RESTORE_MAX_BYTES = 5 * 1024 * 1024 * 1024; +const BACKUP_RESTORE_CHUNK_BYTES = 32 * 1024 * 1024; +// A restore cron hasn't started this long after it was asked for isn't done +const BACKUP_RESTORE_WAIT_MINUTES = 60; + +/* The backups folder, found once a process - cron/cron.php asks every minute */ +function backupRestoreDir(): string +{ + static $dir = null; + return $dir ?? ($dir = rtrim(backupStorageDir(), '/')); +} + +function backupRestoreRequestFile(): string +{ + return backupRestoreDir() . '/restore_queued.json'; +} + +function backupRestoreStatusFile(): string +{ + return backupRestoreDir() . '/restore_status.json'; +} + +function backupRestoreReadJson(string $file): ?array +{ + $raw = @file_get_contents($file); + $data = $raw !== false ? json_decode($raw, true) : null; + return is_array($data) ? $data : null; +} + +function backupRestoreWriteJson(string $file, array $data): bool +{ + $ok = @file_put_contents($file, json_encode($data), LOCK_EX) !== false; + @chmod($file, 0600); + return $ok; +} + +/** How the last restore went - state queued, running, done or failed - or null. */ +function backupRestoreStatus(): ?array +{ + return backupRestoreReadJson(backupRestoreStatusFile()); +} + +function backupRestoreSetStatus(array $status): void +{ + backupRestoreWriteJson(backupRestoreStatusFile(), $status + ['at' => date('Y-m-d H:i:s')]); +} + +/** A restore asked for and not started yet - or null. */ +function backupRestoreRequest(): ?array +{ + return backupRestoreReadJson(backupRestoreRequestFile()); +} + +/** Whether a restore is replacing the database now - cron/cron.php skips its cycle while it does. */ +function backupRestoreInProgress(): bool +{ + $status = backupRestoreStatus(); + return ($status['state'] ?? '') === 'running' && strtotime((string) ($status['started'] ?? '')) > strtotime('-6 hours'); +} + +/** Who or what a backup in the list is down to, from its row - "Scheduled", "Uploaded by Sam". */ +function backupMadeBy(array $backup): string +{ + $by = (string) ($backup['backup_created_by'] ?? ''); + switch ((string) $backup['backup_source']) { + case 'Cron': + return 'Scheduled'; + case 'Adopted': + return 'Found on this server'; + case BACKUP_SOURCE_UPLOADED: + return 'Uploaded' . ($by !== '' ? " by $by" : ''); + case BACKUP_SOURCE_BEFORE_RESTORE: + return 'Before a restore' . ($by !== '' ? " by $by" : ''); + } + return $by !== '' ? "By $by" : 'By hand'; +} + +/** A backup in the list that can be restored: [row, path] or [null, why not]. */ +function backupRestoreStored(mysqli $mysqli, int $backup_id): array +{ + $row = mysqli_fetch_assoc(mysqli_query($mysqli, "SELECT backup_id, backup_type, backup_file_name, backup_size, backup_sha256, backup_status, backup_source, + backup_created_by, backup_created_at, backup_completed_at FROM backups WHERE backup_id = " . intval($backup_id))); + if (!$row || $row['backup_status'] !== 'Complete') { + return [null, 'That backup isn\'t there, or isn\'t finished.']; + } + if ($row['backup_type'] === BACKUP_TYPE_MASTER_KEY) { + return [null, 'A master key export can\'t be restored.']; + } + $path = backupResolvePath((string) $row['backup_file_name']); + if ($path === false || !is_file($path)) { + return [null, 'Its file is no longer on this server.']; + } + return [$row, $path]; +} + +/** The key to open a backup with: the one typed, else this install's. */ +function backupRestoreKey(string $typed): string +{ + $typed = trim($typed); + return $typed !== '' ? $typed : backupEncryptionKey(); +} + +/* + * What is in a backup, and whether it can be restored here: [meta, problem] - problem '' when it + * can. One from a newer database version than this code is refused - ITFlow would break after it. + */ +function backupRestoreCheck(mysqli $mysqli, string $path, string $key): array +{ + $error = null; + $meta = backupInspectArchive($path, $key, $error); + if ($meta === false) { + return [null, (string) $error]; + } + $current = backupCurrentDatabaseVersion($mysqli); + if ($meta['database_version'] !== 'Unknown' && $current !== 'Unknown' && version_compare($meta['database_version'], $current, '>')) { + return [$meta, "It's from a newer database version ({$meta['database_version']}) than this ITFlow ($current) - update ITFlow first."]; + } + return [$meta, '']; +} + +/** '' when the backups folder's disk has $bytes free, else why not. */ +function backupRestoreDiskProblem(float $bytes, string $for): string +{ + $free = @disk_free_space(backupStorageDir()); + if ($free === false || $free >= $bytes) { + return ''; + } + return "There isn't room $for: about " . backupFormatBytes($bytes) . ' is needed, and ' . backupFormatBytes($free) . ' is free.'; +} + +/* + * Another install's key, for cron to open its backup with - sealed with this install's own key + * while it waits in restore_queued.json, so the file alone doesn't give it away. + */ +function backupRestoreSealKey(string $key): string +{ + if ($key === '') { + return ''; + } + $iv = random_bytes(12); + $tag = ''; + $sealed = openssl_encrypt($key, 'aes-256-gcm', hash('sha256', backupEncryptionKey(), true), OPENSSL_RAW_DATA, $iv, $tag); + return $sealed === false ? '' : base64_encode($iv . $tag . $sealed); +} + +function backupRestoreOpenKey(string $sealed): string +{ + $raw = base64_decode($sealed, true); + if ($sealed === '' || $raw === false || strlen($raw) < 29) { + return ''; + } + $key = openssl_decrypt(substr($raw, 28), 'aes-256-gcm', hash('sha256', backupEncryptionKey(), true), OPENSSL_RAW_DATA, substr($raw, 0, 12), substr($raw, 12, 16)); + return $key === false ? '' : $key; +} + +/* + * Ask cron to restore a backup in the list. Checked here first - it opens with the key, isn't + * from a newer version, and there's room - so what the page says is what cron will find. + * Returns [ok, message]. + */ +function backupRestoreQueue(mysqli $mysqli, int $backup_id, string $typed_key, string $who, int $user_id): array +{ + if (backupEncryptionKey() === '') { + return [false, 'There\'s no encryption key, so ITFlow can\'t back itself up before the restore.']; + } + if (backupRestoreInProgress()) { + return [false, 'A restore is already running.']; + } + [$row, $path] = backupRestoreStored($mysqli, $backup_id); + if (!$row) { + return [false, $path]; + } + [$meta, $problem] = backupRestoreCheck($mysqli, $path, backupRestoreKey($typed_key)); + if (!$meta || $problem !== '') { + return [false, $problem]; + } + // The "Before restore" backup (about the size of what's here now), the unpacked backup and the import + $problem = backupRestoreDiskProblem(3 * (filesize($path) ?: 0), 'to restore it'); + if ($problem !== '') { + return [false, $problem]; + } + $request = [ + 'backup_id' => intval($row['backup_id']), + 'file' => (string) $row['backup_file_name'], + 'label' => backupTypeLabel((string) $meta['type']) . ' taken ' . $meta['generated'], + 'key' => backupRestoreSealKey(trim($typed_key)), + 'who' => $who, + 'user_id' => $user_id, + 'queued' => date('Y-m-d H:i:s'), + ]; + if (!backupRestoreWriteJson(backupRestoreRequestFile(), $request)) { + return [false, 'The backups folder can\'t be written to.']; + } + backupRestoreSetStatus(['state' => 'queued', 'who' => $who, 'file' => $request['file'], 'label' => $request['label'], 'queued' => $request['queued']]); + // run_now works on a job that's off - the Backup job ships with its schedule off + mysqli_query($mysqli, "UPDATE cron_jobs SET cron_job_run_now = 1 WHERE cron_job_name = 'backup'"); + return [true, $request['label']]; +} + +/** Take a queued restore back before cron starts it. Whether there was one. */ +function backupRestoreCancel(string $who): bool +{ + $request = backupRestoreRequest(); + if (!$request) { + return false; + } + @unlink(backupRestoreRequestFile()); + backupRestoreSetStatus(['state' => 'cancelled', 'who' => $who, 'file' => (string) $request['file'], 'label' => (string) ($request['label'] ?? ''), + 'message' => "Cancelled by $who before it started - nothing was changed."]); + return true; +} + +/* + * cron/backup.php: do the restore that was asked for, if there is one. Returns null when there + * wasn't, else [ok, message]. $mysqli may be a new connection afterwards. + */ +function backupRestoreRunQueued(mysqli &$mysqli): ?array +{ + $request = backupRestoreRequest(); + if (!$request) { + return null; + } + // Taken away first: a restore that stops part way - the server restarted - is looked at, not started again unattended + @unlink(backupRestoreRequestFile()); + + $who = (string) ($request['who'] ?? 'An admin'); + $base = ['who' => $who, 'file' => (string) ($request['file'] ?? ''), 'label' => (string) ($request['label'] ?? ''), 'queued' => (string) ($request['queued'] ?? '')]; + $fail = function ($message) use ($base) { + backupRestoreSetStatus($base + ['state' => 'failed', 'message' => $message]); + return [false, $message]; + }; + + if (strtotime((string) ($request['queued'] ?? '')) < strtotime('-' . BACKUP_RESTORE_WAIT_MINUTES . ' minutes')) { + return $fail('Not restored - cron didn\'t start it within ' . BACKUP_RESTORE_WAIT_MINUTES . ' minutes of being asked, so nothing was changed. Ask again if you still want it.'); + } + $row = mysqli_fetch_assoc(mysqli_query($mysqli, "SELECT backup_id FROM backups WHERE backup_file_name = '" . escapeSql($base['file']) . "'")); + [$backup, $path] = $row ? backupRestoreStored($mysqli, intval($row['backup_id'])) : [null, 'That backup is no longer in the list.']; + if (!$backup) { + return $fail("Nothing was changed - $path"); + } + $key = !empty($request['key']) ? backupRestoreOpenKey((string) $request['key']) : backupEncryptionKey(); + [$meta, $problem] = backupRestoreCheck($mysqli, $path, $key); + if (!$meta || $problem !== '') { + return $fail("Nothing was changed - $problem"); + } + $problem = backupRestoreDiskProblem(3 * (filesize($path) ?: 0), 'to restore it'); + if ($problem !== '') { + return $fail("Nothing was changed - $problem"); + } + + @set_time_limit(0); + backupRestoreSetStatus($base + ['state' => 'running', 'started' => date('Y-m-d H:i:s')]); + + // ITFlow as it is now, kept - so the restore can be undone + $error = null; + $safety_id = backupCreate($mysqli, BACKUP_TYPE_FULL, $who, BACKUP_SOURCE_BEFORE_RESTORE, $error); + if (!$safety_id) { + return $fail("Nothing was changed - the backup of ITFlow as it is now didn't work: $error"); + } + $safety = mysqli_fetch_assoc(mysqli_query($mysqli, "SELECT backup_type, backup_file_name, backup_size, backup_sha256, backup_status, backup_source, + backup_created_by, backup_created_at, backup_completed_at FROM backups WHERE backup_id = " . intval($safety_id))); + + $error = null; + $ok = backupRestoreArchive($mysqli, $path, $key, $error); + $mysqli = backupDbEnsure($mysqli); + + // The list is the backup's own now, as it was while that backup was made - put back the one + // restored and the "Before restore" one (the rest on this server are found again by the + // nightly tidy-up). Done when it failed too: the database may have been restored before + // what came after it failed. + foreach (array_filter([$backup, $safety]) as $keep) { + $set = []; + foreach (['backup_type', 'backup_file_name', 'backup_size', 'backup_sha256', 'backup_status', 'backup_source', 'backup_created_by', 'backup_created_at', 'backup_completed_at'] as $column) { + $set[] = "$column = " . ($keep[$column] === null ? 'NULL' : "'" . escapeSql((string) $keep[$column]) . "'"); + } + $name = escapeSql((string) $keep['backup_file_name']); + if (mysqli_num_rows(mysqli_query($mysqli, "SELECT 1 FROM backups WHERE backup_file_name = '$name'"))) { + mysqli_query($mysqli, "UPDATE backups SET " . implode(', ', $set) . ", backup_error = NULL WHERE backup_file_name = '$name'"); + } else { + mysqli_query($mysqli, "INSERT INTO backups SET " . implode(', ', $set)); + } + } + // The backup lists the one being made as it was made, or one that was waiting - not left as if still going + $waiting = mysqli_query($mysqli, "SELECT backup_id FROM backups WHERE backup_status IN ('Running', 'Pending')"); + while ($waiting_row = mysqli_fetch_assoc($waiting)) { + mysqli_query($mysqli, "UPDATE backups SET backup_status = 'Failed', backup_error = 'Came with a restored backup - it was being made, or waiting, when that backup was taken', backup_completed_at = NOW() + WHERE backup_id = " . intval($waiting_row['backup_id'])); + } + + // A backup from an older version - bring the database up to this code's + $warnings = []; + $update = backupRestoreUpdateDatabase(); + if ($update !== '') { + $warnings[] = $update; + } + + if (!$ok) { + backupRestoreSetStatus($base + ['state' => 'failed', 'message' => "It didn't restore: $error", 'warnings' => $warnings]); + return [false, "It didn't restore: $error"]; + } + $message = $base['label'] . ' was restored. ITFlow as it was is kept as the "' . BACKUP_SOURCE_BEFORE_RESTORE . '" backup.'; + backupRestoreSetStatus($base + ['state' => 'done', 'message' => $message, 'warnings' => $warnings]); + return [true, $message]; +} + +/** Run the database updates after a restore. '' when it worked, or nothing needed it, else what to tell the admin. */ +function backupRestoreUpdateDatabase(): string +{ + $script = realpath(__DIR__ . '/../scripts/update_cli.php'); + if ($script === false || !function_exists('exec') || PHP_BINARY === '') { + return 'The database updates couldn\'t be run from here - finish them from Maintenance > Update, or with php scripts/update_cli.php --update_db.'; + } + $output = []; + $return = 0; + exec(escapeshellarg(PHP_BINARY) . ' ' . escapeshellarg($script) . ' --update_db 2>&1', $output, $return); + if ($return !== 0) { + $lines = array_values(array_filter(array_map('trim', $output), 'strlen')); + return 'The database update stopped: ' . implode(' | ', array_slice($lines, -2)) . ' - finish it from Maintenance > Update.'; + } + return ''; +} + +/* + * A backup file on this server - an upload - checked with its key, moved in with the other + * backups under a backup's name, and listed as Uploaded. Removed when it isn't a backup. + * Returns [backup_id, message]. + */ +function backupRestoreAdopt(mysqli $mysqli, string $tmp, string $typed_key, string $who, bool $uploaded): array +{ + $error = null; + $meta = backupInspectArchive($tmp, backupRestoreKey($typed_key), $error); + if ($meta === false) { + $uploaded || @unlink($tmp); + return [0, "That isn't a backup that can be opened: $error"]; + } + $type = in_array($meta['type'], backupUnattendedTypes(), true) ? $meta['type'] : BACKUP_TYPE_FULL; + $name = backupBuildFileName($type, randomString(32)); + $path = backupRestoreDir() . '/' . $name; + if (!($uploaded ? move_uploaded_file($tmp, $path) : @rename($tmp, $path))) { + $uploaded || @unlink($tmp); + return [0, 'It couldn\'t be saved with the backups - can the backups folder be written to?']; + } + @chmod($path, 0600); + $size = intval(filesize($path) ?: 0); + $sha = (string) hash_file('sha256', $path); + mysqli_query($mysqli, "INSERT INTO backups SET backup_type = '" . escapeSql($type) . "', backup_file_name = '" . escapeSql($name) . "', backup_size = $size, + backup_sha256 = '$sha', backup_status = 'Complete', backup_source = '" . BACKUP_SOURCE_UPLOADED . "', backup_created_by = '" . escapeSql($who) . "', backup_completed_at = NOW()"); + return [intval(mysqli_insert_id($mysqli)), backupTypeLabel($type) . ' taken ' . $meta['generated'] . ' uploaded (' . backupFormatBytes($size) . ').']; +} + +/** One piece of a browser upload - under PHP's request limit on this server. */ +function backupRestoreChunkBytes(): int +{ + $limit = backupMaxUploadBytes(); + return $limit > 0 ? max(512 * 1024, min(BACKUP_RESTORE_CHUNK_BYTES, $limit - 256 * 1024)) : BACKUP_RESTORE_CHUNK_BYTES; +} + +/** Where a browser upload's pieces are put together. */ +function backupRestorePartPath(string $id): string +{ + return backupRestoreDir() . '/upload_' . preg_replace('/[^a-f0-9]/', '', $id) . '.part'; +} + +/* Pieces of uploads that never finished - a browser closed part way - go after a day */ +function backupRestoreTidyParts(): void +{ + foreach (glob(backupRestoreDir() . '/upload_*.part') ?: [] as $old) { + if (filemtime($old) < time() - 86400) { + @unlink($old); + } + } +} diff --git a/functions/demo.php b/functions/demo.php index 0c534898a..1b1fc690d 100644 --- a/functions/demo.php +++ b/functions/demo.php @@ -75,7 +75,7 @@ const DEMO_MODE_LOCKS = [ ], [ 'message' => "Backups and updates are turned off in the demo.", - 'keys' => ['backup_master_key', 'delete_backup', 'edit_backup_settings', 'edit_dev_mode', 'queue_backup', 'check_update', 'queue_update'], + 'keys' => ['backup_master_key', 'delete_backup', 'edit_backup_settings', 'edit_dev_mode', 'queue_backup', 'restore_backup', 'cancel_restore', 'backup_upload', 'backup_upload_start', 'backup_upload_chunk', 'backup_upload_finish', 'check_update', 'queue_update'], ], [ 'message' => "Cron can't be changed in the demo - it runs the nightly reset.", diff --git a/js/backup_restore.js b/js/backup_restore.js new file mode 100644 index 000000000..5231a8def --- /dev/null +++ b/js/backup_restore.js @@ -0,0 +1,138 @@ +/** + * Maintenance > Backup - uploading a backup to restore, and opening its Restore window. + * + * The file goes in pieces (backup_upload_start, backup_upload_chunk and backup_upload_finish in + * admin/post/backup.php), so a backup bigger than PHP's or a proxy's request limit still gets + * there. A piece refused before it reached ITFlow - a proxy's limit - is sent again smaller, + * down to 1 MB. Without this script the form posts the file in one go (backup_upload). + * + * After an upload the page opens with ?restore=, and the hidden button for its Restore + * window is clicked. + */ +(function () { + 'use strict'; + + document.addEventListener('DOMContentLoaded', function () { + + const autoOpen = document.querySelector('[data-backup-restore-autoopen]'); + if (autoOpen) { + autoOpen.click(); + } + + const form = document.querySelector('[data-backup-upload]'); + if (!form || !window.fetch || !window.FormData || !window.Blob) { + return; + } + + const box = form.querySelector('[data-backup-upload-progress]'); + const bar = box.querySelector('.progress-bar'); + const text = box.querySelector('[data-backup-upload-text]'); + const button = form.querySelector('[data-backup-upload-button]'); + const megabytes = function (n) { + return (n / 1048576).toFixed(n < 104857600 ? 1 : 0) + ' MB'; + }; + + // What answered when it wasn't ITFlow's JSON - a proxy, the web server, or a sign-in that ran out + const refused = function (status, body) { + const doc = new DOMParser().parseFromString(body || '', 'text/html'); + const said = (doc.body ? doc.body.textContent : '').replace(/\s+/g, ' ').trim().slice(0, 200); + let why = 'The server answered ' + status + (said ? ': "' + said + '"' : '') + '.'; + if (status === 413) { + why += ' A piece is over the web server\'s request limit.'; + } else if (/csrf|sign in|log in/i.test(body || '')) { + why += ' Your sign-in no longer matches this page - reload it and upload again.'; + } + return why; + }; + + const post = function (fields, piece) { + const data = new FormData(); + data.append('csrf_token', form.querySelector('[name=csrf_token]').value); + Object.keys(fields).forEach(function (name) { + data.append(name, fields[name]); + }); + if (piece) { + data.append('chunk', piece, 'chunk'); + } + return fetch(form.getAttribute('action'), { method: 'POST', body: data, credentials: 'same-origin' }).then(function (response) { + return response.text().then(function (body) { + let json = null; + try { + json = JSON.parse(body); + } catch (e) { + json = null; + } + if (json && response.ok && !json.error) { + return json; + } + const error = new Error(json && json.error ? json.error : refused(response.status, body)); + error.status = response.status; + error.ours = json !== null; + throw error; + }); + }); + }; + + form.addEventListener('submit', function (event) { + const file = form.querySelector('[name=backup_file]').files[0]; + if (!file) { + return; + } + event.preventDefault(); + + bar.classList.remove('bg-danger'); + bar.style.width = '0%'; + box.hidden = false; + if (file.size > Number(form.dataset.max)) { + text.textContent = 'A backup up to ' + megabytes(Number(form.dataset.max)) + ' can be uploaded here - restore a bigger one from the command line.'; + bar.classList.add('bg-danger'); + return; + } + button.disabled = true; + + // js/auto_reload.js waits while one of these is on the page, so the upload isn't cut off by a reload + const busy = document.createElement('div'); + busy.className = 'swal2-container'; + busy.hidden = true; + busy.style.setProperty('display', 'none', 'important'); + document.body.appendChild(busy); + + const key = form.querySelector('[name=restore_key]').value; + let id = ''; + let size = 0; + + const send = function (offset) { + if (offset >= file.size) { + text.textContent = 'Checking the backup...'; + return post({ backup_upload_finish: 1, id: id, restore_key: key }); + } + return post({ backup_upload_chunk: 1, id: id, offset: offset }, file.slice(offset, offset + size)).catch(function (error) { + // Refused before it reached ITFlow: smaller pieces, down to 1 MB + if (!error.ours && (error.status === 403 || error.status === 413) && size > 1048576) { + size = Math.max(1048576, Math.floor(size / 4)); + text.textContent = 'The server refused pieces that size - sending ' + megabytes(size) + ' pieces instead'; + return { received: offset }; + } + throw new Error('At ' + megabytes(offset) + ' of ' + megabytes(file.size) + ': ' + error.message); + }).then(function (answer) { + bar.style.width = Math.floor(100 * answer.received / file.size) + '%'; + text.textContent = megabytes(answer.received) + ' of ' + megabytes(file.size) + ' sent - keep this page open'; + return send(answer.received); + }); + }; + + post({ backup_upload_start: 1, name: file.name, size: file.size }).then(function (start) { + id = start.id; + size = start.chunk; + return send(0); + }).then(function (done) { + window.location.href = done.redirect; + }).catch(function (error) { + busy.remove(); + button.disabled = false; + bar.classList.add('bg-danger'); + text.textContent = 'The upload stopped: ' + error.message; + }); + }); + }); +})();