diff --git a/CHANGELOG.md b/CHANGELOG.md index 0ed697000..764de99ae 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -67,6 +67,7 @@ Update from Maintenance > Update — Queue Update hands the job to cron, which u ### Bug Fixes - Ticket emails: a picture of about 750 KB or more pasted into a reply, or into a ticket's details, left the email without its message - the picture went missing, and the words with it. The email has all of it now, whatever the picture's size. The same picture made a document come out empty in a client export, and a document template come out empty when it was exported or imported. +- Ticket emails: a picture pasted into a reply, or into a ticket's details, now goes with the email as a picture attached inline, where the message shows it, rather than as data inside the message. Gmail, among others, showed no picture sent the old way. The mail queue also keeps each picture once, instead of a copy inside every email it sends. Admin > Mail Queue still shows the email with its pictures. - App Logs: the email parser wrote an entry every minute that mail came in or was left in the inbox ("Cron Email Parser executed in 0.16 seconds. Processed: 0 email(s), Unprocessed: 2 email(s)"), and the mail queue and email parser wrote one every minute while no mail server was set up. Neither does now. Each email is still a ticket or a reply, one that can't be is still left flagged and unread in the inbox, and an email that fails is still logged, once. With no mail server set up, Maintenance > Cron says so instead. The Mail Queue, Ticket SLA Monitor, Domain Refresher and Certificate Refresher also no longer log an error each time they run while cron is switched off - Maintenance > Cron says why they stopped, as the email parser already did. - Assets: a virtual machine can have a vendor - the provider a hosted one or VPS is rented from - with its order reference and date, on the Purchase tab like other assets. Before, only one added from a vendor's page kept its vendor. - Hosting panels: on a server where Apache hands PHP to PHP-FPM - as HestiaCP, cPanel, Plesk and most hosting control panels set it up - your logo, the favicon, avatars and the photos on contacts, locations, assets and racks didn't load, and the server answered them with an error. They load now. The folders kept private, such as backups and receipts, stay private. Installs made with the install script, which runs PHP inside Apache, were not affected. @@ -115,6 +116,7 @@ Update from Maintenance > Update — Queue Update hands the job to cron, which u - Database 3.7.0: `api_keys.api_key_address_at`, when a key last came in a call's address (`apiKeyNoteAddress()`, at most once a minute), and `settings.config_api_key_in_address` - on where a key that hadn't expired was there at the update, off otherwise and in `db.sql`. `apiKeyInAddress()` says whether a call's query string has an `api_key`. With `apiKeyInAddressAccepted()` false (read guarded, so true before the update), `api/v1/validate_api_key.php` answers a working key in the address with a 401 and a message, and logs it as an `API` `Failed` against the key without counting it toward `apiKeyWatch()` - even when a header sends the key too, as the address is logged all the same. Admin > API Keys reads `api_key_address_at` for its flag and note, and Admin > Settings > Security saves the switch. - Database 3.7.1: `email_queue.email_sensitive`, set by `emailQueue()`'s new `sensitive` option (`addToMailQueue()` writes it once the column is there, `mailQueueHasSensitive()`). It is passed by `shareEmailLink()`, `shareEmailCode()`, `portalInviteMail()`, `portalMailInvite()` and `portalMailSignupExists()` when the account is local, `portalMailPasswordReset()`, `userInviteMail()`, `contractSignatureRequestEmail()` and `ticketMailTaskApproval()`. `cron/mail_queue.php` marks an email sent with `mailQueueSentSql()`, which clears `email_content`, `email_cal_str` and `email_attachments` only where `email_sensitive` is 1 - every email before the update, as before. `mailQueueClearSensitive()` (nightly) clears a failed or cancelled one a day after it was queued. Retry (`mailQueueCan()`, `mailQueueAct()`) needs `email_content`, and Admin > Mail Queue reads `email_content = '' AS email_cleared` for it. The update marks the emails waiting or being sent as sensitive, as they would have been cleared. - `emailMessageHtml()` styled images and paragraphs with a lookahead across each tag (`(?![^>]*\bstyle=)`). On a data: image of 750 KB or more - its base64 is a million characters - that ran out of `pcre.backtrack_limit`, `preg_replace()` returned `null` and the message came out ''. It now works a tag at a time (`/<(img|p)\b([^>]*)>/`, which never backtracks into a tag) and keeps its input should a pattern fail. The same kind of scan through a tag to an attribute is replaced in `documentTemplateInlineImages()` and in the template import's alt and uploads clean-up (`functions/document_template_transfer.php`), and `clientExportLocalImages()` stops at the first `src=` and keeps its input on failure. +- `emailInlineImages()` (`functions/email.php`), called by `emailQueue()` for each email it queues, takes every `` whose `src` is a data: picture (checked with `getimagesizefromstring()`) out to `uploads/tmp/mail_images/.` - written once, whoever it goes to - and points it at `cid:@itflow`. Each goes in the email's attachment manifest with a `cid` (`addToMailQueue()` keeps it), and `cron/mail_queue.php` attaches it with `addEmbeddedImage()`, so the message is `multipart/related`. A meeting request with pictures goes as an attached `invite.ics`, as one with files already did, since PHPMailer leaves the calendar part out of a message with inline parts. `mailQueueAttachments()` splits a manifest into files and pictures - the list's paperclip and the email's attachments are files only - and `mailQueueShowImages()` puts the pictures back as data to show the email in Admin > Mail Queue. `mailQueueImageCleanup()` (nightly) deletes a picture no queued email points at once it is a day old. ## [26.11.8] diff --git a/admin/mail_queue.php b/admin/mail_queue.php index 489f3b261..fe8cbc29b 100644 --- a/admin/mail_queue.php +++ b/admin/mail_queue.php @@ -39,7 +39,7 @@ $search = "(email_id LIKE '%$q%' OR email_from LIKE '%$q%' OR email_from_name LI AND DATE(email_queued_at) BETWEEN '$dtf' AND '$dtt'"; $sql = mysqli_query($mysqli, "SELECT SQL_CALC_FOUND_ROWS email_id, email_status, email_attempts, email_from, email_from_name, email_recipient, email_recipient_name, - email_subject, email_queued_at, email_failed_at, email_sent_at, " . mailQueueErrorColumn() . ", COALESCE(email_attachments, '') <> '' AS has_attachments, COALESCE(email_cal_str, '') <> '' AS has_invite, email_content = '' AS email_cleared + email_subject, email_queued_at, email_failed_at, email_sent_at, " . mailQueueErrorColumn() . ", email_attachments, COALESCE(email_cal_str, '') <> '' AS has_invite, email_content = '' AS email_cleared FROM email_queue WHERE $search AND {$tabs[$show][2]} ORDER BY {$sortable_columns[$sort]}, email_id DESC @@ -182,6 +182,8 @@ $filtered_dates = isset($_GET['dtf']) && $_GET['dtf'] !== '' && $_GET['dtf'] !== (no subject)'; $view = 'data-modal-size="lg" data-modal-url="modals/mail_queue/mail_queue_message_view.php?id=' . $email_id . '"'; $from = $person($row['email_from_name'], $row['email_from']); diff --git a/admin/modals/mail_queue/mail_queue_message_view.php b/admin/modals/mail_queue/mail_queue_message_view.php index b70a2a810..3dcdfdfac 100644 --- a/admin/modals/mail_queue/mail_queue_message_view.php +++ b/admin/modals/mail_queue/mail_queue_message_view.php @@ -45,14 +45,10 @@ if (!$row) { ?> } return escapeHtml($name) . ' <' . escapeHtml($address) . '>'; }; - $attachments = []; - foreach ((array) json_decode((string) $row['email_attachments'], true) as $attachment) { - if (is_array($attachment) && trim((string) ($attachment['name'] ?? '')) !== '') { - $attachments[] = basename((string) $attachment['name']); - } - } + // The files attached; the pictures in its message are shown in it + $attachments = mailQueueAttachments($row['email_attachments'])['files']; $has_invite = trim((string) $row['email_cal_str']) !== ''; - $content = (string) $row['email_content']; + $content = mailQueueShowImages((string) $row['email_content'], $row['email_attachments']); $subject = trim((string) $row['email_subject']) !== '' ? escapeHtml($row['email_subject']) : '(no subject)'; // How many tries, in words diff --git a/cron/mail_queue.php b/cron/mail_queue.php index d30356231..bbbd445e3 100644 --- a/cron/mail_queue.php +++ b/cron/mail_queue.php @@ -285,8 +285,8 @@ function sendQueueEmail( // A calendar invitation or cancellation goes in as its own text/calendar part // beside the HTML - the way calendar apps send them, and the only way Outlook // shows one as a meeting rather than a file. PHPMailer only builds that part - // when there are no file attachments, so anything else (and a published - // event to add) travels as an attached invite.ics. + // when there are no file attachments or pictures in the message, so anything + // else (and a published event to add) travels as an attached invite.ics. if (!empty($ics_str)) { if (preg_match('/^METHOD:(REQUEST|CANCEL)\r?$/m', $ics_str) && empty($attachments_json)) { $mail->Ical = $ics_str; @@ -318,7 +318,12 @@ function sendQueueEmail( continue; } - $mail->addAttachment($attachment_path, basename($attachment['name'])); + // A picture that was in the message as data goes in inline, where its cid: points (emailInlineImages()) + if (!empty($attachment['cid'])) { + $mail->addEmbeddedImage($attachment_path, (string) $attachment['cid'], basename($attachment['name'])); + } else { + $mail->addAttachment($attachment_path, basename($attachment['name'])); + } } } } diff --git a/cron/nightly_tasks.php b/cron/nightly_tasks.php index c09ef181f..81f06ea23 100644 --- a/cron/nightly_tasks.php +++ b/cron/nightly_tasks.php @@ -164,6 +164,7 @@ mysqli_query($mysqli, "DELETE FROM notifications WHERE notification_dismissed_at // Clean-up mail queue - a failed or cancelled email with a link or code that opens something is cleared after a day, everything is deleted after 90 mailQueueClearSensitive(); +mailQueueImageCleanup(); mysqli_query($mysqli, "DELETE FROM email_queue WHERE email_queued_at < CURDATE() - INTERVAL " . MAIL_QUEUE_KEEP_DAYS . " DAY"); // Clean-up the PDFs made to go with billing emails - sent or given up on by now diff --git a/functions/app.php b/functions/app.php index b3443ac42..d550de767 100644 --- a/functions/app.php +++ b/functions/app.php @@ -829,10 +829,11 @@ function addToMailQueue($data) { if (empty($attachment['path']) || empty($attachment['name'])) { continue; } + // A picture in the message goes in inline, where its cid: points (emailInlineImages()) $attachment_manifest[] = [ 'path' => strval($attachment['path']), 'name' => strval($attachment['name']) - ]; + ] + (!empty($attachment['cid']) ? ['cid' => preg_replace('/[^A-Za-z0-9@._-]/', '', (string) $attachment['cid'])] : []); } if ($attachment_manifest) { $attachments = mysqli_escape_string($mysqli, json_encode($attachment_manifest)); diff --git a/functions/email.php b/functions/email.php index b84f1bdec..1f504ba1c 100644 --- a/functions/email.php +++ b/functions/email.php @@ -336,6 +336,49 @@ function emailPlainText($html, $limit = 0) { return $text; } +/* + * The pictures in an email's HTML as data - pasted into a ticket reply, say - taken out to files + * the mail queue attaches inline, each pointing at its cid: instead. Gmail, among others, + * shows no picture carried as data, and the queue keeps each picture once rather than a copy in + * every email. [html, [['path', 'name', 'cid'], ...]]. A tag at a time, as a scan through a + * large picture's data runs out of PCRE's backtracking (emailMessageHtml()); a picture that isn't + * one, or can't be saved, stays as it was. + */ +function emailInlineImages($html) { + $images = []; + $html = (string) $html; + if (stripos($html, 'data:image/') === false) { + return [$html, []]; + } + $types = [IMAGETYPE_PNG => 'png', IMAGETYPE_JPEG => 'jpg', IMAGETYPE_GIF => 'gif', IMAGETYPE_WEBP => 'webp']; + $out = preg_replace_callback('/]*>/i', function ($tag) use (&$images, $types) { + if (!preg_match('/\ssrc\s*=\s*(["\'])data:image\/[a-z+.-]+;base64,([A-Za-z0-9+\/=\s]+)\1/i', $tag[0], $m)) { + return $tag[0]; + } + $data = base64_decode(preg_replace('/\s+/', '', $m[2]), true); + $info = $data !== false ? @getimagesizefromstring($data) : false; + $ext = $info ? ($types[$info[2]] ?? '') : ''; + if ($ext === '') { + return $tag[0]; + } + $hash = hash('sha256', $data); + $path = MAIL_QUEUE_IMAGE_DIR . "/$hash.$ext"; + $file = dirname(__DIR__) . '/' . $path; + if (!is_file($file)) { + if (!is_dir(dirname($file))) { + @mkdir(dirname($file), 0775, true); + } + if (@file_put_contents($file, $data) === false) { + return $tag[0]; + } + } + $cid = substr($hash, 0, 32) . '@itflow'; + $images[$cid] = ['path' => $path, 'name' => 'picture-' . substr($hash, 0, 8) . ".$ext", 'cid' => $cid]; + return str_replace($m[0], ' src=' . $m[1] . 'cid:' . $cid . $m[1], $tag[0]); + }, $html); + return $out === null ? [$html, []] : [$out, array_values($images)]; +} + /** * Queue an email - one per recipient, each greeted by name, each address once. * $recipients: [['email' => ..., 'name' => ..., 'greet' => name to greet (or @@ -380,6 +423,8 @@ function emailQueue(array $recipients, $subject, array $email, array $opts = []) 'subject' => (string) $subject, 'html' => $html, 'attachments' => $attachments, 'cal_str' => (string) ($opts['cal_str'] ?? '')]; continue; } + // Its pictures attached inline, not carried as data + [$html, $images] = emailInlineImages($html); $data[] = [ 'from' => mysqli_real_escape_string($mysqli, (string) $from['email']), 'from_name' => mysqli_real_escape_string($mysqli, (string) $from['name']), @@ -387,7 +432,7 @@ function emailQueue(array $recipients, $subject, array $email, array $opts = []) 'recipient_name' => mysqli_real_escape_string($mysqli, $name !== '' ? $name : $address), 'subject' => mysqli_real_escape_string($mysqli, (string) $subject), 'body' => mysqli_real_escape_string($mysqli, $html), - 'attachments' => $attachments, + 'attachments' => array_merge($attachments, $images), ] + (!empty($opts['cal_str']) ? ['cal_str' => (string) $opts['cal_str']] : []) + ($queued_at !== '' ? ['queued_at' => $queued_at] : []) + (intval($opts['ticket_reply_id'] ?? 0) > 0 && ticketReplyEmailReady() ? ['ticket_reply_id' => intval($opts['ticket_reply_id'])] : []) diff --git a/functions/mail_queue.php b/functions/mail_queue.php index ead12d764..4cb1ea54b 100644 --- a/functions/mail_queue.php +++ b/functions/mail_queue.php @@ -25,6 +25,8 @@ const MAIL_QUEUE_TRIES = 4; const MAIL_QUEUE_RETRY_MINUTES = 30; const MAIL_QUEUE_KEEP_DAYS = 90; const MAIL_QUEUE_CANCELLED = 4; +// Pictures that were in an email as data, kept once each for the queue to attach inline (emailInlineImages()) - the web can't reach uploads/tmp +const MAIL_QUEUE_IMAGE_DIR = 'uploads/tmp/mail_images'; /* The tabs of the list - key => [label, icon, SQL for what is in it]. */ function mailQueueTabs() { @@ -246,6 +248,67 @@ function mailQueueErrorColumn() { return mailQueueHasError() ? 'email_error' : 'NULL AS email_error'; } +/* + * What an email's manifest (email_attachments) holds - ['files' => names of the files attached, + * 'images' => [cid => app-relative path] of the pictures that go in its message (emailInlineImages())]. + */ +function mailQueueAttachments($json) { + $out = ['files' => [], 'images' => []]; + foreach ((array) json_decode((string) $json, true) as $attachment) { + if (!is_array($attachment) || trim((string) ($attachment['name'] ?? '')) === '') { + continue; + } + if (!empty($attachment['cid'])) { + $out['images'][(string) $attachment['cid']] = (string) ($attachment['path'] ?? ''); + } else { + $out['files'][] = basename((string) $attachment['name']); + } + } + return $out; +} + +/* + * An email's message with its pictures back in it as data, to show in Admin > Mail Queue - the + * message points at them by cid:, as the mail program that gets it reads them. One gone stays a cid:. + */ +function mailQueueShowImages($html, $json) { + $dir = realpath(dirname(__DIR__) . '/' . MAIL_QUEUE_IMAGE_DIR); + foreach (mailQueueAttachments($json)['images'] as $cid => $path) { + $file = $dir ? realpath(dirname(__DIR__) . '/' . $path) : false; + $info = $file && strpos($file, $dir . DIRECTORY_SEPARATOR) === 0 ? @getimagesize($file) : false; + if ($info) { + $html = str_replace('cid:' . $cid, 'data:' . $info['mime'] . ';base64,' . base64_encode((string) file_get_contents($file)), $html); + } + } + return $html; +} + +/* + * Delete the pictures no email in the queue points at any more - left a day first, so one saved + * for an email being queued this minute stays (nightly). How many went. + */ +function mailQueueImageCleanup() { + global $mysqli; + $dir = dirname(__DIR__) . '/' . MAIL_QUEUE_IMAGE_DIR; + if (!is_dir($dir)) { + return 0; + } + $kept = []; + $sql = mysqli_query($mysqli, "SELECT email_attachments FROM email_queue WHERE email_attachments LIKE '%mail\\_images%'"); + while ($row = mysqli_fetch_row($sql)) { + foreach (mailQueueAttachments($row[0])['images'] as $path) { + $kept[basename($path)] = true; + } + } + $removed = 0; + foreach (glob("$dir/*") ?: [] as $file) { + if (is_file($file) && !isset($kept[basename($file)]) && filemtime($file) < time() - 86400 && @unlink($file)) { + $removed++; + } + } + return $removed; +} + /* Is there an email_sensitive column yet? Database 3.7.1 adds it. */ function mailQueueHasSensitive() { global $mysqli;