diff --git a/CHANGELOG.md b/CHANGELOG.md index 7bdf419e7..308e8933b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,7 +6,7 @@ This file documents all notable changes made to ITFlow. ### Upgrading -Update from Maintenance > Update — Update ITFlow hands the job to cron, which updates the files and applies this release's five database changes (3.5.4 to 3.5.9) in the same pass. Nothing else is required. Everyone is signed out once after the update, agents and client portal contacts alike - see Security. The new Payment Provider Fees job in Maintenance > Cron is set up by itself, and so are Manager Check-in and Managed Instances on an install whose config.php turns managed mode on (see New Features & Updates). +Update from Maintenance > Update — Update ITFlow hands the job to cron, which updates the files and applies this release's eight database changes (3.5.4 to 3.6.1) in the same pass. Nothing else is required. Everyone is signed out once after the update, agents and client portal contacts alike - see Security. The new Payment Provider Fees and Text Messages jobs in Maintenance > Cron are set up by themselves, and so are Manager Check-in and Managed Instances on an install whose config.php turns managed mode on (see New Features & Updates). ### Security @@ -27,6 +27,12 @@ Update from Maintenance > Update — Update ITFlow hands the job to cron, which - API keys: agents are notified in ITFlow a week before an API key expires, so an integration doesn't stop without warning. - Payment Providers: Stripe's fee on a card payment is now recorded as an expense about 15 minutes after the payment, instead of the next morning. A new Payment Provider Fees job in Maintenance > Cron does it, and you can run it more often there. Record fees now still works any time. - Payment Providers: clients can pay the card fee. Turn on Clients pay the card fee on Stripe in Admin > Payment Providers and enter what Stripe charges you, as 2.9% plus 0.30. Each card payment is then made bigger by Stripe's fee on it, so you get the full invoice. The fee goes on the invoice as a Card Fee line, with no tax, when the payment goes through. Clients see it before they pay: on Pay invoices and the checkout page in the client portal, on the page an invoice's link opens, and next to autopay. It applies to payments in the client portal, through an invoice's link, by autopay, and when you charge a saved card, where the Charge Saved Card window shows it too. One payment for several invoices has one fee, shared over them. A client can be let off it with No card fee on their Billing tab. Refunding a card payment from ITFlow takes its Card Fee line back off the invoice. It is off until you turn it on. Card surcharges have rules where you and your clients are - in the US, for one, the card networks cap them and don't allow them on debit cards - so check them before turning it on. +- Text messaging: ITFlow can text through Skyetel, from one support number. Set it up in Admin > Integrations > Text Messaging with your Skyetel SID, secret and number, paste the address it shows into the number's SMS Callback URL in Skyetel's portal, and send a test text. A US number has to be registered for texting in Skyetel's portal, or carriers block what it sends. +- Tickets: a text from one of your contacts opens a ticket for them, and they're texted back its number. What they text next goes on it as their reply, with any photos attached, and opens it again if it was resolved. Pick **Text reply** on a ticket to answer by text - it goes to the contact's mobile as plain text and shows in the conversation with whether it went. Turn this off, or open tickets for unknown numbers too, in Admin > Settings > Tickets > Text to ticket. +- Texts: a new page under Support for what comes to your number that isn't a contact. Unmatched holds texts from numbers that aren't a contact's - open a ticket from one (saving the number on the contact as you go), dismiss it, or mark it spam, which blocks the number. Codes holds verification and MFA codes, kept out of tickets: only agents who can view credentials see them, each code is masked until revealed (every reveal is in the audit log), it can be revealed for 15 minutes, and its words are wiped after a day. Spam holds texts from blocked numbers and the usual bait. +- Texts to clients: contacts who agree to texts hear about their work by text - a ticket opened for them and resolved (and, if you choose, each emailed reply), a visit or remote work booked or cancelled, a reminder before it, and invoices and payment reminders with a link to pay. Replying to a text about a ticket adds to the ticket, and a reply to the resolved text opens it again. Choose which in Admin > Settings > Notifications > Texts to clients. Texts ITFlow sends by itself that would land between 9pm and 8am wait for the morning. +- Contacts: nobody is texted these until they agree. Turn on **Texts** on a contact (with their say-so), use **Ask them by text** on their page - a YES turns them on - or they turn it on under Your account in the client portal. The contact's page says when and how they agreed, or that they replied STOP. +- Text messaging: STOP, START and HELP are answered as carriers require. A number that replies STOP gets nothing more until it replies START - Text reply is greyed out for it - and the ticket says so. Admin > Text Messaging lists who opted out and what is blocked; Maintenance > Text Messages has every text sent and received, with retry and cancel. - Managed mode: one ITFlow can now manage others - a client's, a reseller's or your own hosted ones. It's off until config.php turns it on, each side on its own: `$config_enable_managed_instances = true;` on the ITFlow that does the managing adds Admin > Managed Instances, and `$config_enable_managed_by = true;` on each one it manages adds Admin > Managed By. An install with neither line sees nothing of it - no menu, pages, cron jobs or sign-in. Then add one in Admin > Managed Instances, and paste the code it gives you into the other ITFlow's Admin > Managed By. From then on that ITFlow checks in every minute, and you see how each one is doing: its version and whether an update is waiting (a security one stands out), whether cron is running, its last backup, the errors it logged today, and its agents against its seat limit - and you're notified when one stops checking in. From the manager you can update an instance, or let it update itself a set number of days after a release comes out and straight away for a security release; set how many agents it can have and its white-label key; make it read-only, where its people can still sign in, look and export, or suspend it, with a message of your own; and sign in to it as support in a new tab, for two hours, with no password or seat of its own and every sign-in in its audit log. What the manager sets is locked on the instance, which says so - everything else stays the instance's own, and its admins can disconnect it unless its config.php keeps it managed (`$config_manager_required = true`, for a provider hosting it, which turns Managed By on as well). Instances only ever call the manager, so they can sit behind a firewall, and they send how they're doing - never client data. - Export Data: the documentation PDF can go out as a secure link, like a login, document or file. Share a link next to Download PDF - or Share the IT documentation on a client's Shared Links - makes the PDF from the parts you tick and gives you a link that works for as long and as many times as you set, with a passphrase or an emailed code first if you like, and tells you when it's downloaded. The PDF is made when the link is, so later changes aren't in it, and it's kept encrypted until the link stops working - then it's deleted. With passwords and 2FA secrets in it, the PDF is locked with a password of its own too, as a download is, and the page the link opens says so. - Maintenance > Update is easier to follow. It opens with where your install stands - up to date, an update available, an update running, or the database still to finish - and the one button for it, now Update ITFlow instead of Queue Update. Below it is what's new: the release notes for the versions between yours and the latest, read from the changelog each time ITFlow checks for updates, with the developer notes folded away and the commits behind them a click further. Before you update it lists what to do first, and says whether any of ITFlow's own files have been changed on your install, which updating would put back. Once you click Update ITFlow, the page shows the update running, refreshes itself until it's done, and then says when ITFlow was updated, or what stopped it. Alongside are this install's version, branch, database, when it last checked and updated, whether cron is running, and the command to run the same update from a shell, with the folder and user filled in for your server. @@ -43,6 +49,15 @@ Update from Maintenance > Update — Update ITFlow hands the job to cron, which ### Developer Updates +- Text messaging is `functions/sms.php`. Database 3.6.0 adds `sms_providers` (the service, its SID and secret, the number in E.164 digits, the key in its inbound address), `sms_messages` (every text both ways - `sms_message_direction` `out`/`in`, `sms_message_status` queued/sending/sent/failed/blocked/cancelled going out and ticket/unmatched/code/spam/keyword/dismissed coming in) and `sms_numbers` (a number's name, `sms_number_opted_out_at`, `sms_number_blocked_at`), and `settings.config_ticket_sms_parse`, `config_ticket_sms_unknown_senders` and `config_ticket_sms_acknowledge`, read where they're used by `smsTicketSettings()`. +- Sending: `smsQueue($e164, $text, $meta)` splits at `SMS_TEXT_LIMIT` and refuses a number that replied STOP (a `blocked` row) unless `force` - only the STOP answer itself. `smsSendQueued()` claims a row before sending, so a request and the new Text Messages job (`cron/sms_queue.php`, `smsQueueRun()`, every minute) can't both send it; a 401 fails at once, a timeout, 403/429 or 5xx is tried again up to `SMS_MAX_ATTEMPTS`. The job keeps under Skyetel's 300 texts in five minutes and a second apart, and `smsTidy()` wipes codes after `SMS_CODE_KEEP_HOURS`, deletes spam and dismissed texts after `SMS_SPAM_KEEP_DAYS` and the rest after `SMS_KEEP_DAYS`. Providers are `smsProviderKinds()` - another service is an entry there with its own send function and `api/sms/.php`. +- Receiving: `api/sms/skyetel.php?key=` (the key compared with `hash_equals`, else 404) hands `{to, from, text, media}` to `smsInbound()`, in this order: a repeat within two minutes is dropped; STOP/START/HELP (`smsKeyword()`) are answered; a blocked number is Spam; a number with a text on an open ticket in `SMS_CONVERSATION_DAYS` replies on it (`smsConversationTicket()`, `smsTicketReply()`); a contact's mobile or phone (`smsContactsFor()`, matched on the digits with `REGEXP_REPLACE`) opens a ticket (`smsTicketOpen()`, source `Text`); then `smsVerificationCode()`, `smsLooksLikeSpam()`, and Unmatched with one in-app notification an hour per number. `smsSaveMedia()` fetches media over https only, from public addresses only, `SMS_MEDIA_MAX_BYTES` each, as `TICKET_ATTACHMENT_EXTENSIONS`. Texts and HTML go into queries through `smsSql()` (mysqli_real_escape_string) - `escapeSql()` strips anything that looks like a tag. +- Tickets: `public_reply_type` 3 on `add_ticket_reply` texts the reply through `smsTicketReplySend()` (`smsTextFromHtml()` keeps lines, writes links out and drops formatting) to `smsTicketRecipient()` - the contact's mobile, else the number texting on the ticket. `agent/ticket.php` reads each reply's texts with `smsRepliesForTicket()`. `ticketSourceIcon()`, the ticket summary report and the agent emails' `how` know `Text`/`text`, and `aiTicketDraft()` takes `text` for a short plain draft. +- Texts to clients: database 3.6.1 adds `contacts.contact_sms_opt_in`, `contact_sms_opt_in_at` and `contact_sms_opt_in_by` (a record of how they agreed), `sms_messages.sms_message_send_after` with a `(sms_message_purpose, sms_message_ticket_id, sms_message_created_at)` index, and `settings.config_sms_client_tickets` (0 off, 1 opened and resolved, 2 and each emailed reply), `config_sms_client_visits`, `config_sms_visit_reminder_hours` (0, 1, 2, 4, 24) and `config_sms_client_invoices`, read by `smsClientSettings()`. Everything checks `smsColumnExists()` first, so files ahead of the database do nothing. +- `triggerCustomAction()` calls `smsClientNotify()` on `ticket_create` (not for a ticket that came in by text), `ticket_resolve`, `ticket_schedule` and `ticket_unschedule`; `add_ticket_reply` calls `smsTicketReplyNotify()` after an emailed reply; and `smsInvoiceNotify($invoice, $kind, $recipients)` follows `billingEmailQueue()` in Send (`email_invoice`), the nightly payment reminders, recurring invoices sending themselves and `billingInvoiceIssue()`, texting those recipients who agreed and adding "and texted to ..." to the invoice's history. Each goes through `smsNotifyContact()`: only to `smsContactTextable()` contacts, queued with `send_after` from `smsSendAfter()` (`SMS_QUIET_FROM`/`SMS_QUIET_UNTIL`, your time - unless the morning is too late for a visit), and sent by the Text Messages job, whose `smsQueueRun()` waits for `send_after`. Texts about a ticket carry its id, so `smsConversationTicket()` puts the reply on it. +- The Text Messages job also runs `smsVisitReminders()`: tickets booked within `config_sms_visit_reminder_hours`, open, whose contact agreed, with no `reminder` or `notify_scheduled` text for the ticket in that window. +- Opting in: `smsOptInSet($contact_id, $on, $by, $by_them)` - turned on by the contact (portal, YES) it lifts their STOP; by an agent it doesn't. `smsOptInAsk()` texts an `opt_in_request`; `smsInbound()` takes YES (or OK, Sure) from a number asked in `SMS_OPT_IN_ASK_DAYS` as agreeing (`smsOptInPending()`). The contact window posts `sms_opt_in_shown` with the switch, so `smsContactOptInFromForm()` leaves a contact saved from elsewhere alone. `smsContactTextState()` is what the contact page, the contact window and the portal show. +- Texts > Codes reveals through `ajax.php?sms_reveal_code` (POST, CSRF, `module_credential`), logged as Text Message / View every time; the page asks `ajax.php?sms_latest_code` every five seconds and comes again when a code lands. Codes are masked in Maintenance > Text Messages, and their words aren't searched on either page. - `backupRestoreArchive()` no longer extracts the inner `uploads.zip` wholesale. It builds an allow-list of entries and extracts with `extractTo($dir, $entries)`, leaving out any entry whose basename is `.htaccess`, `.user.ini` or `.htpasswd` and any entry that resolves inside `backupStorageDir()`, which the restore preserves while it empties the rest of `uploads/`. Unsafe paths (`..`, absolute) still fail the restore as before. `scripts/restore_cli.php` gets the same, as it calls the same function. - `backupHardenStorageDir()` writes its `.htaccess` every time instead of only when there is none, so a guard carried in by an archive can't survive. `backupAssertUploadsGuards()` also rewrites the deny-all guards in `uploads/backups`, `canned_responses`, `expenses` and `tmp`, so a restored `uploads/` has the same guards as a fresh install. Both use the new `BACKUP_UPLOADS_DENY_GUARD` constant. - `accountLedgerSql()` reads its lines first and joins each to its account with `STRAIGHT_JOIN`. Left to choose, the optimizer started from the accounts and found their lines through an index it built on the lines' temporary table (`derived_with_keys`), which was two to three times slower. diff --git a/admin/database_updates/3.6.0.php b/admin/database_updates/3.6.0.php new file mode 100644 index 000000000..83a9e983e --- /dev/null +++ b/admin/database_updates/3.6.0.php @@ -0,0 +1,91 @@ + Tickets > Text to ticket + * + * Re-runnable: each table and column is made only where it isn't there yet. + */ + +defined('FROM_DB_UPDATER') || die("Direct file access is not allowed"); + + mysqli_query($mysqli, "CREATE TABLE IF NOT EXISTS `sms_providers` ( + `sms_provider_id` int(11) NOT NULL AUTO_INCREMENT, + `sms_provider_name` varchar(100) NOT NULL, + `sms_provider_description` varchar(255) DEFAULT NULL, + `sms_provider_username` varchar(255) NOT NULL DEFAULT '', + `sms_provider_secret` varchar(500) NOT NULL DEFAULT '', + `sms_provider_number` varchar(20) NOT NULL DEFAULT '', + `sms_provider_inbound_key` varchar(64) NOT NULL DEFAULT '', + `sms_provider_active` tinyint(1) NOT NULL DEFAULT 1, + `sms_provider_last_inbound_at` datetime DEFAULT NULL, + `sms_provider_created_at` datetime NOT NULL DEFAULT current_timestamp(), + `sms_provider_updated_at` datetime DEFAULT NULL ON UPDATE current_timestamp(), + PRIMARY KEY (`sms_provider_id`) + ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci"); + + mysqli_query($mysqli, "CREATE TABLE IF NOT EXISTS `sms_messages` ( + `sms_message_id` int(11) NOT NULL AUTO_INCREMENT, + `sms_message_direction` varchar(3) NOT NULL, + `sms_message_status` varchar(20) NOT NULL, + `sms_message_number` varchar(20) NOT NULL, + `sms_message_our_number` varchar(20) NOT NULL DEFAULT '', + `sms_message_body` text DEFAULT NULL, + `sms_message_media` text DEFAULT NULL, + `sms_message_code` varchar(20) DEFAULT NULL, + `sms_message_purpose` varchar(30) NOT NULL DEFAULT '', + `sms_message_attempts` tinyint(4) NOT NULL DEFAULT 0, + `sms_message_error` varchar(500) DEFAULT NULL, + `sms_message_provider_ref` varchar(100) DEFAULT NULL, + `sms_message_contact_id` int(11) NOT NULL DEFAULT 0, + `sms_message_client_id` int(11) NOT NULL DEFAULT 0, + `sms_message_ticket_id` int(11) NOT NULL DEFAULT 0, + `sms_message_ticket_reply_id` int(11) NOT NULL DEFAULT 0, + `sms_message_user_id` int(11) NOT NULL DEFAULT 0, + `sms_message_created_at` datetime NOT NULL DEFAULT current_timestamp(), + `sms_message_attempted_at` datetime DEFAULT NULL, + `sms_message_sent_at` datetime DEFAULT NULL, + `sms_message_handled_at` datetime DEFAULT NULL, + `sms_message_handled_by` int(11) NOT NULL DEFAULT 0, + PRIMARY KEY (`sms_message_id`), + KEY `sms_message_status` (`sms_message_status`,`sms_message_id`), + KEY `sms_message_number` (`sms_message_number`,`sms_message_created_at`), + KEY `sms_message_ticket_id` (`sms_message_ticket_id`), + KEY `sms_message_created_at` (`sms_message_created_at`) + ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci"); + + mysqli_query($mysqli, "CREATE TABLE IF NOT EXISTS `sms_numbers` ( + `sms_number_phone` varchar(20) NOT NULL, + `sms_number_label` varchar(100) DEFAULT NULL, + `sms_number_opted_out_at` datetime DEFAULT NULL, + `sms_number_blocked_at` datetime DEFAULT NULL, + `sms_number_blocked_by` int(11) NOT NULL DEFAULT 0, + `sms_number_updated_at` datetime DEFAULT NULL ON UPDATE current_timestamp(), + PRIMARY KEY (`sms_number_phone`) + ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci"); + + // Settings > Tickets > Text to ticket - after the email-to-ticket switches, in the order the page shows them + $itflow_sms_settings = [ + 'config_ticket_sms_parse' => "tinyint(1) NOT NULL DEFAULT 1 AFTER `config_ticket_email_parse_unknown_senders`", + 'config_ticket_sms_unknown_senders' => "tinyint(1) NOT NULL DEFAULT 0 AFTER `config_ticket_sms_parse`", + 'config_ticket_sms_acknowledge' => "tinyint(1) NOT NULL DEFAULT 1 AFTER `config_ticket_sms_unknown_senders`", + ]; + foreach ($itflow_sms_settings as $itflow_sms_column => $itflow_sms_definition) { + $itflow_column_exists = mysqli_query($mysqli, "SELECT 1 FROM information_schema.COLUMNS + WHERE TABLE_SCHEMA = DATABASE() AND TABLE_NAME = 'settings' AND COLUMN_NAME = '$itflow_sms_column' LIMIT 1"); + if (!$itflow_column_exists || mysqli_num_rows($itflow_column_exists) === 0) { + mysqli_query($mysqli, "ALTER TABLE `settings` ADD `$itflow_sms_column` $itflow_sms_definition"); + } + } + + unset($itflow_sms_settings, $itflow_sms_column, $itflow_sms_definition, $itflow_column_exists); diff --git a/admin/database_updates/3.6.1.php b/admin/database_updates/3.6.1.php new file mode 100644 index 000000000..da717340e --- /dev/null +++ b/admin/database_updates/3.6.1.php @@ -0,0 +1,45 @@ + Notifications > Texts to clients + * + * Re-runnable: each column is made only where it isn't there yet. + */ + +defined('FROM_DB_UPDATER') || die("Direct file access is not allowed"); + + $itflow_sms_columns = [ + ['contacts', 'contact_sms_opt_in', "tinyint(1) NOT NULL DEFAULT 0 AFTER `contact_company_ticket_emails`"], + ['contacts', 'contact_sms_opt_in_at', "datetime DEFAULT NULL AFTER `contact_sms_opt_in`"], + ['contacts', 'contact_sms_opt_in_by', "varchar(200) DEFAULT NULL AFTER `contact_sms_opt_in_at`"], + ['sms_messages', 'sms_message_send_after', "datetime DEFAULT NULL AFTER `sms_message_created_at`"], + ['settings', 'config_sms_client_tickets', "tinyint(1) NOT NULL DEFAULT 1 AFTER `config_ticket_sms_acknowledge`"], + ['settings', 'config_sms_client_visits', "tinyint(1) NOT NULL DEFAULT 1 AFTER `config_sms_client_tickets`"], + ['settings', 'config_sms_visit_reminder_hours', "int(11) NOT NULL DEFAULT 24 AFTER `config_sms_client_visits`"], + ['settings', 'config_sms_client_invoices', "tinyint(1) NOT NULL DEFAULT 1 AFTER `config_sms_visit_reminder_hours`"], + ]; + foreach ($itflow_sms_columns as [$itflow_sms_table, $itflow_sms_column, $itflow_sms_definition]) { + $itflow_column_exists = mysqli_query($mysqli, "SELECT 1 FROM information_schema.COLUMNS + WHERE TABLE_SCHEMA = DATABASE() AND TABLE_NAME = '$itflow_sms_table' AND COLUMN_NAME = '$itflow_sms_column' LIMIT 1"); + if (!$itflow_column_exists || mysqli_num_rows($itflow_column_exists) === 0) { + mysqli_query($mysqli, "ALTER TABLE `$itflow_sms_table` ADD `$itflow_sms_column` $itflow_sms_definition"); + } + } + + // Whether a reminder or a booking went for a ticket is looked up by what it was for and the ticket + $itflow_index_exists = mysqli_query($mysqli, "SELECT 1 FROM information_schema.STATISTICS + WHERE TABLE_SCHEMA = DATABASE() AND TABLE_NAME = 'sms_messages' AND INDEX_NAME = 'sms_message_purpose' LIMIT 1"); + if (!$itflow_index_exists || mysqli_num_rows($itflow_index_exists) === 0) { + mysqli_query($mysqli, "ALTER TABLE `sms_messages` ADD KEY `sms_message_purpose` (`sms_message_purpose`,`sms_message_ticket_id`,`sms_message_created_at`)"); + } + + unset($itflow_sms_columns, $itflow_sms_table, $itflow_sms_column, $itflow_sms_definition, $itflow_column_exists, $itflow_index_exists); diff --git a/admin/includes/side_nav.php b/admin/includes/side_nav.php index 5c7800347..844176605 100644 --- a/admin/includes/side_nav.php +++ b/admin/includes/side_nav.php @@ -121,6 +121,12 @@

AI Providers

+ + + + + + +