refactor(mimefactory): separate rendering of message payload and sendable message#8345
refactor(mimefactory): separate rendering of message payload and sendable message#8345link2xt wants to merge 2 commits into
Conversation
45102bc to
1956c42
Compare
5f3c57c to
f9c33de
Compare
f9c33de to
e314fc0
Compare
3d4610c to
b1b0e3f
Compare
ce43887 to
6abd90c
Compare
6abd90c to
e350dd6
Compare
a18ab11 to
2e20218
Compare
a39da96 to
1c19dd8
Compare
1c19dd8 to
efe7d0f
Compare
28522ff to
b34bc43
Compare
606d7c7 to
23c8c8a
Compare
b34bc43 to
c85e622
Compare
| /// Consumes a `MimeFactory` and renders it into a message which is then stored in | ||
| /// `smtp`-table to be used by the SMTP loop | ||
| #[expect(clippy::arithmetic_side_effects)] | ||
| pub(crate) async fn pre_render( |
There was a problem hiding this comment.
Maybe render_queued() or smth like this? pre_render() is somewhat "vague", and i'm afraid it may be confused with pre-messages.
EDIT: render_queued() sounds incorrect because it should mention the source object (MimeFactory), not the returned QueuedMail. Maybe into_queued_mail()?
There was a problem hiding this comment.
into_queued_mail() sounds best to me. I agree that pre_render could be confused with pre-messages.
| from_header.write_header(&mut inner_headers, 6)?; | ||
|
|
||
| if is_encrypted { | ||
| let unencrypted_from = Address::new_address(None::<&'static str>, from_addr.to_string()); |
There was a problem hiding this comment.
| let unencrypted_from = Address::new_address(None::<&'static str>, from_addr.to_string()); | |
| let unencrypted_from = Address::new_address(None::<&'static str>, from_addr.clone()); |
|
|
||
| if should_attach_pubkey { | ||
| let aheader = Aheader { | ||
| addr: from_addr.clone(), |
There was a problem hiding this comment.
| addr: from_addr.clone(), | |
| addr: from_addr, |
| verified: false, | ||
| }; | ||
| let autocrypt_header = mail_builder::headers::raw::Raw::new(aheader.to_string()); | ||
| if is_encrypted { |
There was a problem hiding this comment.
Could be
let headers = match is_encrypted {
true => &mut inner_headers,
false => &mut outer_headers,
};
...to deduplicate the code a bit
| }; | ||
|
|
||
| // Encrypt to self unconditionally, | ||
| // even for a single-device setup. |
There was a problem hiding this comment.
"... to not reveal which setup we have", IIRC.
23c8c8a to
9351ae3
Compare
c85e622 to
908b761
Compare
…able message This change separates rendering into two separate steps: 1. Rendering of the message payload without the From, Date and Autocrypt headers. 2. Adding the From, Date and Autocrypt headers and possibly encrypting the message. The goal is to have serializable result of the first step that can be persisted in the database and sent later with any email address. This way it will be possible to send queued messages over any relay. This will make it possible not to remove all messages from the queue when the sending relay is changed. Currently changing `configured_addr` deletes everything from `smtp` table. This change is however only a refactoring and does not implement any features.
9351ae3 to
598ac45
Compare
| } else { | ||
| SeipdVersion::V1 | ||
| }; | ||
| let display_name = if is_securejoin_message && !is_encrypted { |
There was a problem hiding this comment.
Btw, this is apparently not the correct implementation of #7396 , see the protocol flow chart -- Alice should attach her profile (avatar, displayname, etc.) only at the final step, but not in the "vc-pubkey" message which is encrypted with AUTH. It seems this doesn't create any security issue, but it's just enough to send the profile at the final step. Also if we let Alice review "Securejoin requests" in the future before fully executing them, showing Bob's profile to Alice first (not otherwise) makes sense:
Since we want to give Alice the opportunity to review join requests (as a future improvement that doesn't require breaking the protocol), Alice can't send any private data in the second message. [...]
EDIT: For Bob it's even worse because he sends his displayname already in "vc-request-pubkey" apparently, so e.g. other broadcast subscribers can see it.
CC @Hocuri
This is not related to the refactoring, but i'd like this to be reviewed by others.
There was a problem hiding this comment.
Ok, this misunderstanding comes from improper naming of is_encrypted which is evaluated by this function:
pub fn will_be_encrypted(&self) -> bool {
self.encryption_pubkeys.is_some()
}Could it be renamed in a separate commit to e.g. is_pubkey_encrypted? I've created #8399 and tested it with this PR as well, everything works as expected actually.
There was a problem hiding this comment.
IIUC, the naming of will_be_encrypted is correct. If the message is symmetrically-encrypted, then encryption_pubkeys is Some(vec![]). As @link2xt said elsewhere, would be a nice refactoring to put this into an enum instead in order to prevent such confusion.
The "vc-pubkey" and "vc-request-pubkey" messages don't use the function here. Instead, they are created by render_symm_encrypted_securejoin_message. This is why the code works as correctly. Thanks for writing a test for it!
There was a problem hiding this comment.
Thanks for making it clear! I think that just renaming encryption_pubkeys to encryption would make the code more understandable (because it's actually not only about pubkeys). And as for this check, it could use self.encryption.is_none_or(|pubkeys| pubkeys.is_empty()), but maybe it's better to add an assertion that we're not rendering a symmetrically encrypted SecureJoin message here.
| /// - `Message-ID` | ||
| /// | ||
| /// Encrypts and signs the message if necessary. | ||
| pub(crate) fn render_queued_mail( |
There was a problem hiding this comment.
May be a member function of QueuedMail, even if there are some extra params
There was a problem hiding this comment.
Seems equally fine as a standalone function
| .await?; | ||
| let side_effects = RenderSideEffects { | ||
| subject: "Secure-Join".to_string(), | ||
|
|
Agreed, there is no need to support chat-like features in unencrypted messages anymore. Hidden headers are from a time when we wanted to enable nice DC-to-DC communication via unencrypted messages with avatars etc. |
|
ftr, some history: we had late-mime-generation until 2018 and switched to early-mime-generation for various reasons, mainly speed and disappearing-messages, thid old core-c issue shows more detailes deltachat/deltachat-core#427 while performance might still be a concern, esp when there are only short time windows to get things out, iiuc, in this PR most data to be sent out are still generated early, so eg it is fine if database records the message originates from are deleted, this is important eg. for disappearing messages that time, however, mime-generation included encryption, so doing that late uses the state of the database when message goes out, not the state when send button is pressed, which led to all kind of issues. if i get it corectly, this is not the case with this PR, the mime |
|
I made some measurements on my 5 year old middle-class phone (Fairphone 3), compiled in release mode. FTR, this PR here doesn't switch to late encryption yet (it's just refactoring that will enable us to switch to late encryption). And yes, the plan is to switch to late encryption, but keep early mime generation. Once we switch to late encryption, the function Click to see the diff I used for measuringdiff --git a/src/chat.rs b/src/chat.rs
index 0ad2fd49d..9ee6bfc65 100644
--- a/src/chat.rs
+++ b/src/chat.rs
@@ -6,7 +6,7 @@
use std::io::Cursor;
use std::marker::Sync;
use std::path::{Path, PathBuf};
-use std::time::Duration;
+use std::time::{Duration, Instant};
use anyhow::{Context as _, Result, anyhow, bail, ensure};
use chrono::TimeZone;
@@ -2780,10 +2780,17 @@ async fn render_mime_message_and_pre_message(
let mut mimefactory_post_msg = mimefactory.clone();
mimefactory_post_msg.set_as_post_message();
+ let start = Instant::now();
let (queued_msg, side_effects) = Box::pin(mimefactory_post_msg.pre_render(context))
.await
.context("Failed to render post-message")?;
+ info!(
+ context,
+ "post-message pre_render took {:?}",
+ start.elapsed()
+ );
+ let start = Instant::now();
let rendered_msg = mimefactory::render_queued_mail(
queued_msg,
&public_key,
@@ -2792,12 +2799,20 @@ async fn render_mime_message_and_pre_message(
timestamp,
side_effects,
)?;
+ info!(
+ context,
+ "post-message render_queued_mail took {:?}",
+ start.elapsed()
+ );
let mut mimefactory_pre_msg = mimefactory;
mimefactory_pre_msg.set_as_pre_message_for(&rendered_msg);
+ let start = Instant::now();
let (queued_pre_msg, pre_side_effects) = Box::pin(mimefactory_pre_msg.pre_render(context))
.await
.context("pre-message failed to render")?;
+ info!(context, "pre-message pre_render took {:?}", start.elapsed());
+ let start = Instant::now();
let rendered_pre_msg = mimefactory::render_queued_mail(
queued_pre_msg,
&public_key,
@@ -2806,6 +2821,11 @@ async fn render_mime_message_and_pre_message(
timestamp,
pre_side_effects,
)?;
+ info!(
+ context,
+ "pre-message render_queued_mail took {:?}",
+ start.elapsed()
+ );
if rendered_pre_msg.message.len() > PRE_MSG_SIZE_WARNING_THRESHOLD {
warn!(
@@ -2818,7 +2838,10 @@ async fn render_mime_message_and_pre_message(
Ok((Some(rendered_pre_msg), rendered_msg))
} else {
+ let start = Instant::now();
let (queued_msg, side_effects) = Box::pin(mimefactory.pre_render(context)).await?;
+ info!(context, "pre_render took {:?}", start.elapsed());
+ let start = Instant::now();
let rendered_msg = mimefactory::render_queued_mail(
queued_msg,
&public_key,
@@ -2827,6 +2850,7 @@ async fn render_mime_message_and_pre_message(
timestamp,
side_effects,
)?;
+ info!(context, "render_queued_mail took {:?}", start.elapsed());
Ok((None, rendered_msg))
} |
| let (queued_mail, side_effects) = Box::pin(self.pre_render(context)).await?; | ||
| let rendered_mail = render_queued_mail( | ||
| queued_mail, | ||
| &public_key, | ||
| &secret_key, | ||
| from_addr, | ||
| timestamp, | ||
| side_effects, | ||
| )?; |
There was a problem hiding this comment.
render_queued_mail() is a blocking function (possibly blocking for multiple seconds) that is called without spawn_blocking() or block_in_place() both here and in multiple other places. This needs to be fixed in order not to block the executor
There was a problem hiding this comment.
Indeed, spawn_blocking() was removed from pk_encrypt(). I think, to protect from blocking the executor in such cases, we should use block_in_place() at the lower level, i.e. in pk_encrypt() while still calling spawn_blocking() here and in other high-level functions. So, even if spawn_blocking() is forgotten, the worst thing is other futures in the same task getting blocked, not other tasks. Nested block_in_place() should be fine (no-op), see discussion in tokio-rs/tokio#2327
| /// Unencrypted queued message. | ||
| /// | ||
| /// This message has both the headers and the body, | ||
| /// but without the From, Autocrypt and Date headers. |
There was a problem hiding this comment.
without the [...] Date header
Didn't we say that the Date header should continue to contain the time of the user pressing "Send", and if we add the time of sending the message out then it should go into a new header?
| /// - `Message-ID` | ||
| /// | ||
| /// Encrypts and signs the message if necessary. | ||
| pub(crate) fn render_queued_mail( |
There was a problem hiding this comment.
Seems equally fine as a standalone function
| from_header.write_header(&mut inner_headers, 6)?; | ||
|
|
||
| if is_encrypted { | ||
| let unencrypted_from = Address::new_address(None::<&'static str>, from_addr.to_string()); | ||
| outer_headers.extend_from_slice(b"From: "); | ||
| unencrypted_from.write_header(&mut outer_headers, 6)?; | ||
|
|
||
| inner_headers.extend_from_slice(b"HP-Outer: From: "); | ||
| unencrypted_from.write_header(&mut inner_headers, 16)?; | ||
| } else { | ||
| outer_headers.extend_from_slice(b"From: "); | ||
| from_header.write_header(&mut outer_headers, 6)?; | ||
| } |
There was a problem hiding this comment.
Those are a lot of magic numbers (6, 6, 16, 6, ...). Would be nice to extract a function like:
fn add_header(
name: &str,
value: impl mail_builder::headers::Header,
headers: &mut Vec<u8>,
) -> Result<()> {
headers.extend_from_slice(name.as_bytes());
value.write_header(headers, name.len())?;
Ok(())
}| inner_headers.extend_from_slice(b"Date: "); | ||
| inner_headers.extend_from_slice(date.as_bytes()); | ||
| inner_headers.extend_from_slice(b"\r\n"); |
There was a problem hiding this comment.
Is there a reason for using extend_from_slice() rather than extend() everywhere? From what I understand, when both of them compile then they do the same except that extend() can be slightly faster? (because it uses copy rather than clone). And while performance doesn't matter, extend() would read more nicely than extend_from_slice() because it's shorter
| .unwrap() | ||
| .to_rfc2822(); | ||
| outer_headers.extend_from_slice(b"Date: "); | ||
| outer_headers.extend_from_slice(unprotected_date.as_bytes()); |
There was a problem hiding this comment.
I'm mildly concerned about building everything ourselves here rather than using mail-builder, because we may forget to escape some special characters. But probably it's fine.
|
|
||
| if is_encrypted { | ||
| // Copy not protected headers to outer headers. | ||
| let (parsed_headers, _index) = mailparse::parse_headers(&raw_message)?; |
There was a problem hiding this comment.
FTR, we're first building the message and then parsing it again, but this parse_headers call takes just 5µs-80µs in my measurements (5 year old middle-class phone, release mode), and code-wise it seems like the easiest solution, so, it's good as-is.
There was a problem hiding this comment.
...although the only header we actually need is Chat-Is-Post-Message. For all the others (Subject, To, Chat-Version) it's already clear that they should be added and with which value. So, the code here could be slightly simplified by putting pre_message_mode on QueuedMail, and then always mechanically adding these four outer headers. Then we wouldn't need to parse the message here and iterate over all headers.
But it's fine as-is, too
| } else if header_name == "subject" { | ||
| &b"[...]"[..] | ||
| } else if header_name == "to" { | ||
| &b"To: \"hidden-recipients\""[..] |
There was a problem hiding this comment.
This will lead to a header that looks like To: To: "hidden-recipients". IIRC it needs to be To: "hidden-recipients": ;, i.e. a group of recipients with 0 entries. I just tested it, GMX rejects mails sent with this PR because of a malformed To header.
Apparently we have no tests at all for how a rendered outer message should look like?
| let mut full_raw_message = inner_headers.clone(); | ||
| full_raw_message.append(&mut raw_message); |
There was a problem hiding this comment.
This leaves raw_message empty but still accessible (not dropped), so that someone might accidentally use it. Using extend would consume it instead, which seems nicer. Also, there is no need to clone inner_headers since it's equally not needed afterwards.
| let mut full_raw_message = inner_headers.clone(); | |
| full_raw_message.append(&mut raw_message); | |
| let mut full_raw_message = inner_headers; | |
| full_raw_message.extend(raw_message); |
| }; | ||
|
|
||
| let mut full_message = outer_headers; | ||
| full_message.append(&mut message); |
There was a problem hiding this comment.
Same here:
| full_message.append(&mut message); | |
| full_message.extend(message); |
| // <https://www.rfc-editor.org/rfc/rfc9787.html#structural-header-fields>. | ||
| continue; | ||
| } | ||
| let header_value = if header_name == "mime-version" |
There was a problem hiding this comment.
header_name can't be mime-version at this place. We checked just a few lines above, and continueed if so.
I guess it can just eb removed from here.
|
|
||
| if is_encrypted { | ||
| // Copy not protected headers to outer headers. | ||
| let (parsed_headers, _index) = mailparse::parse_headers(&raw_message)?; |
There was a problem hiding this comment.
...although the only header we actually need is Chat-Is-Post-Message. For all the others (Subject, To, Chat-Version) it's already clear that they should be added and with which value. So, the code here could be slightly simplified by putting pre_message_mode on QueuedMail, and then always mechanically adding these four outer headers. Then we wouldn't need to parse the message here and iterate over all headers.
But it's fine as-is, too
| /// Helper function render the messages that are not queued. | ||
| /// | ||
| /// Used for MDNs because their payload is rendered right before sending. |
There was a problem hiding this comment.
| /// Helper function render the messages that are not queued. | |
| /// | |
| /// Used for MDNs because their payload is rendered right before sending. | |
| /// Helper function that renders a message without queuing it first. | |
| /// | |
| /// Used for MDNs because they are fully rendered and sent in one go, | |
| /// rather than first creating a [`QueuedMail`] and sending it later. |
| } else { | ||
| SeipdVersion::V1 | ||
| }; | ||
| let display_name = if is_securejoin_message && !is_encrypted { |
There was a problem hiding this comment.
IIUC, the naming of will_be_encrypted is correct. If the message is symmetrically-encrypted, then encryption_pubkeys is Some(vec![]). As @link2xt said elsewhere, would be a nice refactoring to put this into an enum instead in order to prevent such confusion.
The "vc-pubkey" and "vc-request-pubkey" messages don't use the function here. Instead, they are created by render_symm_encrypted_securejoin_message. This is why the code works as correctly. Thanks for writing a test for it!
| let side_effects = RenderSideEffects { | ||
| subject: "Secure-Join".to_string(), | ||
|
|
||
| wrap_encrypted_part(encrypted) | ||
| ..Default::default() | ||
| }; | ||
|
|
There was a problem hiding this comment.
I don't think we need to care about the subject of securejoin messages anymore?
| let side_effects = RenderSideEffects { | |
| subject: "Secure-Join".to_string(), | |
| wrap_encrypted_part(encrypted) | |
| ..Default::default() | |
| }; | |
| let side_effects = RenderSideEffects::default(); |
| @@ -2022,84 +2293,26 @@ struct HeadersByConfidentiality { | |||
| /// See [`HeadersByConfidentiality`] for more info. | |||
| fn group_headers_by_confidentiality( | |||
There was a problem hiding this comment.
Maybe put a comment here that this function isn't really needed anymore and can be removed once we remove hidden headers?
|
|
||
| Ok(message) | ||
| /// Renders MIME part into a vector. | ||
| pub(crate) fn part_to_vec(message: MimePart<'static>) -> Vec<u8> { |
There was a problem hiding this comment.
Maybe rename this function to part_to_bytes? To better describe what this function does without looking at the function signature (a "vec" could also be a vec of lines, or a vec of headers, or whatever)
| inner_headers.extend_from_slice(b"Date: "); | ||
| inner_headers.extend_from_slice(date.as_bytes()); |
There was a problem hiding this comment.
Wouldn't it be easier to already add the Date in pre_render()? Then the timestamp wouldn't need to be passed through. For the randomized outer date, the current time can be used rather than the time of clicking "Send".
| let message = message.header( | ||
| "Message-ID", | ||
| mail_builder::headers::message_id::MessageId::new(rfc724_mid.clone()), | ||
| ); |
There was a problem hiding this comment.
Why is this needed here? Does it add the Message-Id header to the inner part in order to reduce problems with MUAs messing with it?
There was a problem hiding this comment.
Just in case, currently we also put Message-Id into hidden headers, so this PR doesn't change this, and i think this is indeed to protect from servers messing with it
This change separates rendering into two separate steps:
The goal is to have serializable result of the first step
that can be persisted in the database and sent later with any email address.
This way it will be possible to send queued messages over any relay.
This will make it possible not to remove all messages from the queue
when the sending relay is changed.
Currently changing
configured_addrdeletes everything fromsmtptable.This change is however only a refactoring and does not implement any features.
This is a refactoring PR in preparation for automatic relay failover.
As a side effect it also makes possible to change the Date of the message for #8112 if we decide on this approach (unlikely).
Serializable mail is currently called
mimefactory::QueuedMail. Everything except the public keys is trivially serializable, public keys should likely be serialized as recipient fingerprints rather than as OpenPGP certificates.I also noticed that we likely can remove the concept of "hidden headers" which are headers that are sent on the mulipart/mixed level of unencrypted messages. They are used to send
Chat-Editheaders and avatars in unencrypted messages. Sending avatars in unencrypted messages is not useful because they are not displayed anyway. And we can decide to make it impossible to edit and delete unencrypted messages. I have not changed anything in this PR, however, hidden headers work as before.