docs: more additions to chat protocol (#4355)

* docs: more additions to chat protocol

* schema wip

* schema

* update diagram

* remove comments

* punctuation

* diagram text

* update threat model

---------

Co-authored-by: Evgeny Poberezkin <2769109+epoberezkin@users.noreply.github.com>

docs/protocol/diagrams/group.mmd

@@ -4,9 +4,16 @@ sequenceDiagram
   participant B as Bob
   participant C as Existing<br>contact
 
-  note over A, B: 1. send and accept group invitation
+  alt invite contact
-  A ->> B: x.grp.inv<br>invite Bob to group<br>(via contact connection)
+    note over A, B: 1a. send and accept group invitation
-  B ->> A: x.grp.acpt<br>accept invitation<br>(via member connection)<br>establish group member connection
+    A ->> B: x.grp.inv<br>invite Bob to group<br>(via contact connection)
+    B ->> A: x.grp.acpt<br>accept invitation<br>(via member connection)<br>establish group member connection
+  else join via group link
+    note over A, B: 1b. join via group link and accept request
+    B ->> A: join via group link<br>SimpleX contact address
+    A ->> B: x.grp.link.inv in SMP confirmation<br>accept joining member request,<br>sending group profile, etc.<br>establish group member connection
+    A ->> B: x.grp.link.mem<br>send inviting member profile
+  end
 
   note over M, B: 2. introduce new member Bob to all existing members
   A ->> M: x.grp.mem.new<br>"announce" Bob<br>to existing members<br>(via member connections)
@@ -20,14 +27,25 @@ sequenceDiagram
   end
   A ->> M: x.grp.mem.fwd<br>forward "invitations" and<br>Bob's chat protocol version<br>to all members<br>(via member connections)
 
+  note over M, B: group message forwarding<br>(while connections between members are being established)
+  M -->> B: messages between members and Bob are forwarded by Alice
+  B -->> M: 
+
   note over M, B: 3. establish direct and group member connections
   M ->> B: establish group member connection
 
   opt chat protocol compatible version < 2
     M ->> B: establish direct connection
-    note over M, C: 4. deduplicate new contact
+    note over M, C: 3*. deduplicate new contact
     B ->> M: x.info.probe<br>"probe" is sent to all new members
     B ->> C: x.info.probe.check<br>"probe" hash,<br>in case contact and<br>member profiles match
     C ->> B: x.info.probe.ok<br> original "probe",<br> in case contact and member<br>are the same user
     note over B: merge existing and new contacts if received and sent probe hashes match
   end
+
+  note over M, B: 4. notify inviting member that connection is established
+  M ->> A: x.grp.mem.con
+  B ->> A: x.grp.mem.con
+  note over A: stops forwarding messages
+  M -->> B: messages are sent via group connection without forwarding
+  B -->> M: 

docs/protocol/simplex-chat.md

@@ -2,7 +2,7 @@
 title: SimpleX Chat Protocol
 revision: 08.08.2022
 ---
-DRAFT Revision 0.1, 2022-08-08
+Revision 2, 2024-06-24
 
 Evgeny Poberezkin
 
@@ -157,7 +157,7 @@ This message is sent by both sides of the connection during the connection hands
 
 ### Probing for duplicate contacts
 
-As there are no globally unique user identitifiers, when the contact a user is already connected to is added to the group by some other group member, this contact will be added to user's list of contacts as a new contact. To allow merging such contacts, "a probe" (random base64url-encoded 32 bytes) SHOULD be sent to all new members as part of `x.info.probe` message and, in case there is a contact with the same profile, the hash of the probe MAY be sent to it as part of `x.info.probe.check` message. In case both the new member and the existing contact are the same user (they would receive both the probe and its hash), the contact would send back the original probe as part of `x.info.probe.ok` message via the previously existing contact connection – proving to the sender that this new member and the existing contact are the same user, in which case the sender SHOULD merge these two contacts.
+As there are no globally unique user identifiers, when the contact a user is already connected to is added to the group by some other group member, this contact will be added to user's list of contacts as a new contact. To allow merging such contacts, "a probe" (random base64url-encoded 32 bytes) SHOULD be sent to all new members as part of `x.info.probe` message and, in case there is a contact with the same profile, the hash of the probe MAY be sent to it as part of `x.info.probe.check` message. In case both the new member and the existing contact are the same user (they would receive both the probe and its hash), the contact would send back the original probe as part of `x.info.probe.ok` message via the previously existing contact connection – proving to the sender that this new member and the existing contact are the same user, in which case the sender SHOULD merge these two contacts.
 
 Sending clients MAY disable this functionality, and receiving clients MAY ignore probe messages.
 
@@ -210,23 +210,23 @@ File attachment can optionally include connection address to receive the file -
 
 ### Decentralized design for chat groups
 
-SimpleX Chat groups are fully decentralized and do not have any globally unique group identifiers - they are only defined on client devices as a group profile and a set of bi-directional SimpleX connections with other group members. When a new member accepts group invitation, the inviting member introduces a new member to all existing members and forwards the connection addresses so that they can establish direct and group member connections.
+SimpleX Chat groups are fully decentralized and do not have any globally unique group identifiers - they are only defined on client devices as a group profile and a set of bi-directional SimpleX connections with other group members. When a new member accepts group invitation or joins via group link, the inviting member introduces a new member to all existing members and forwards the connection addresses so that they can establish direct and group member connections.
 
 There is a possibility of the attack here: as the introducing member forwards the addresses, they can substitute them with other addresses, performing MITM attack on the communication between existing and introduced members - this is similar to the communication operator being able to perform MITM on any connection between the users. To mitigate this attack this group sub-protocol will be extended to allow validating security of the connection by sending connection verification out-of-band.
 
-Clients are RECOMMENDED to indicate in the UI whether the connection to a group member or contact was made directly or via annother user.
+Clients are RECOMMENDED to indicate in the UI whether the connection to a group member or contact was made directly or via another user.
 
 Each member in the group is identified by a group-wide unique identifier used by all members in the group. This is to allow referencing members in the messages and to allow group message integrity validation.
 
 The diagram below shows the sequence of messages sent between the users' clients to add the new member to the group.
 
-![Adding member to the group](./diagrams/group.svg)
-
 While introduced members establish connection inside group, inviting member forwards messages between them by sending `x.grp.msg.forward` messages. When introduced members finalize connection, they notify inviting member to stop forwarding via `x.grp.mem.con` message.
 
+![Adding member to the group](./diagrams/group.svg)
+
 ### Member roles
 
-Currently members can have one of three roles - `owner`, `admin` and `member`. The user that created the group is self-assigned owner role, the new members are assigned role by the member who adds them - only `owner` and `admin` members can add new members; only `owner` members can add members with `owner` role.
+Currently members can have one of three roles - `owner`, `admin`, `member` and `observer`. The user that created the group is self-assigned owner role, the new members are assigned role by the member who adds them - only `owner` and `admin` members can add new members; only `owner` members can add members with `owner` role. `Observer` members only receive messages and aren't allowed to send messages.
 
 ### Messages to manage groups and add members
 
@@ -279,3 +279,66 @@ These message are used for WebRTC calls:
 3. `x.call.answer`: to continue with call connection the initiating clients must reply with `x.call.answer` message. This message contains WebRTC answer and collected ICE candidates. Additional ICE candidates can be sent in `x.call.extra` message.
 
 4. `x.call.end` message is sent to notify the other party that the call is terminated.
+
+## Threat model
+
+This threat model compliments SMP, XFTP, push notifications and XRCP protocols threat models:
+
+- [SimpleX Messaging Protocol threat model](https://github.com/simplex-chat/simplexmq/blob/master/protocol/overview-tjr.md#threat-model);
+- [SimpleX File Transfer Protocol threat model](https://github.com/simplex-chat/simplexmq/blob/master/protocol/xftp.md#threat-model);
+- [Push notifications threat model](https://github.com/simplex-chat/simplexmq/blob/master/protocol/push-notifications.md#threat-model);
+- [SimpleX Remote Control Protocol threat model](https://github.com/simplex-chat/simplexmq/blob/master/protocol/xrcp.md#threat-model).
+
+#### A user's contact
+
+*can:*
+
+- send messages prohibited by user's preferences or otherwise act non-compliantly with user's preferences (for example, if message with updated preferences was lost or failed to be processed, or with modified client), in which case user client should treat such messages and actions as prohibited.
+
+- by exchanging special messages with user's client, match user's contact with existing group members and/or contacts that have identical user profile (see [Probing for duplicate contacts](#probing-for-duplicate-contacts)).
+
+- identify that and when a user is using SimpleX, in case user has delivery receipts enabled, or based on other automated client responses.
+
+*cannot:*
+
+- match user's contact with existing group members and/or contacts with different or with incognito profiles.
+
+- match user's contact without communicating with the user's client.
+
+#### A group member
+
+*can:*
+
+- send messages prohibited by group's preferences and member restrictions or otherwise act non-compliantly with preferences and restrictions (for example, if decentralized group state diverged, or with modified client), in which case user client should treat such messages and actions as prohibited.
+
+- create a direct contact with a user if group permissions allow it.
+
+- by exchanging special messages with user's client, match user's group member record with the existing group members and/or contacts that have identical user profile.
+
+- undetectably send different messages to different group members, or selectively send messages to some members and not send to others.
+
+- identify that and when a user is using SimpleX, in case user has delivery receipts enabled, or based on other automated client responses.
+
+- join the same group several times, from the same or from different user profile, and pretend to be different members.
+
+*cannot:*
+
+- match user's contact with existing group members and/or contacts with different or with incognito profiles.
+
+- match user's group member record with existing group members and/or contacts without communication of user's client.
+
+- determine whether two group members with different or with incognito profiles are the same user.
+
+#### A group admin
+
+*can:*
+
+- carry out MITM attack between user and other group member(s) when forwarding invitations for group connections (user can detect such attack by verifying connection security codes out-of-band).
+
+- undetectably forward different messages to different group members, selectively adding, modifying, and dropping forwarded messages.
+
+- disrupt decentralized group state by sending different messages that change group state (such as adding or removing members, member role changes, etc.) to different group members, or sending such messages selectively.
+
+*cannot:*
+
+- prove that two group members with incognito profiles is the same user.

docs/protocol/simplex-chat.schema.json

@@ -8,7 +8,7 @@
         "displayName": {
           "type": "string",
           "metadata": {
-            "format": "non-empty string without spaces, the first character must not be # or @"
+            "format": "non-empty string, the first character must not be # or @"
           }
         },
         "fullName": {"type": "string"}
@@ -19,6 +19,39 @@
           "metadata": {
             "format": "data URI format for base64 encoded image"
           }
+        },
+        "contactLink": {"ref": "connReqUri"},
+        "preferences": {
+          "type": "string",
+          "metadata": {
+            "format": "JSON encoded user preferences"
+          }
+        }
+      },
+      "additionalProperties": true
+    },
+    "groupProfile": {
+      "properties": {
+        "displayName": {
+          "type": "string",
+          "metadata": {
+            "format": "non-empty string, the first character must not be # or @"
+          }
+        },
+        "fullName": {"type": "string"}
+      },
+      "optionalProperties": {
+        "image": {
+          "type": "string",
+          "metadata": {
+            "format": "data URI format for base64 encoded image"
+          }
+        },
+        "groupPreferences": {
+          "type": "string",
+          "metadata": {
+            "format": "JSON encoded user preferences"
+          }
         }
       },
       "additionalProperties": true
@@ -29,6 +62,8 @@
       },
       "optionalProperties": {
         "file": {"ref": "fileInvitation"},
+        "ttl": {"type": "integer"},
+        "live": {"type": "boolean"},
         "quote": {
           "properties": {
             "msgRef": {"ref": "msgRef"},
@@ -56,17 +91,47 @@
           }
         },
         "image": {
-          "text": {"type": "string", "metadata": {"comment": "can be empty"}},
+          "properties": {
-          "image": {"ref": "base64url"}
+            "text": {"type": "string", "metadata": {"comment": "can be empty"}},
+            "image": {"ref": "base64url"}
+          }
+        },
+        "video": {
+          "properties": {
+            "text": {"type": "string", "metadata": {"comment": "can be empty"}},
+            "image": {"ref": "base64url"},
+            "duration": {"type": "integer"}
+          }
+        },
+        "voice": {
+          "properties": {
+            "text": {"type": "string", "metadata": {"comment": "can be empty"}},
+            "duration": {"type": "integer"}
+          }
         },
         "file": {
-          "text": {"type": "string", "metadata": {"comment": "can be empty"}}
+          "properties": {
+            "text": {"type": "string", "metadata": {"comment": "can be empty"}}
+          }
         }
       },
       "metadata": {
         "comment": "it is RECOMMENDED that the clients support other values in `type` properties showing them as text messages in case `text` property is present"
       }
     },
+    "msgReaction" : {
+      "discriminator": "type",
+      "mapping": {
+        "emoji": {
+          "properties": {
+            "emoji": {
+              "type": "string",
+              "metadata": {"comment": "emoji character"}
+            }
+          }
+        }
+      }
+    },
     "msgRef": {
       "properties": {
         "msgId": {"ref": "base64url"},
@@ -91,7 +156,31 @@
         "fileSize": {"type": "uint32"}
       },
       "optionalProperties": {
-        "fileConnReq": {"ref": "connReqUri"}
+        "fileDigest": {"ref": "base64url"},
+        "fileConnReq": {"ref": "connReqUri"},
+        "fileDescr": {"ref": "fileDescription"}
+      }
+    },
+    "fileDescription": {
+      "properties": {
+        "fileDescrText": {
+          "type": "string",
+          "metadata": {
+            "format": "XFTP file description part text"
+          }
+        },
+        "fileDescrPartNo": {
+          "type": "integer",
+          "metadata": {
+            "format": "XFTP file description part number"
+          }
+        },
+        "fileDescrComplete": {
+          "type": "boolean",
+          "metadata": {
+            "format": "XFTP file description completion marker"
+          }
+        }
       }
     },
     "linkPreview": {
@@ -100,6 +189,21 @@
         "title": {"type": "string"},
         "description": {"type": "string"},
         "image": {"ref": "base64url"}
+      },
+      "optionalProperties": {
+        "content": {"ref": "linkContent"}
+      }
+    },
+    "linkContent": {
+      "discriminator": "type",
+      "mapping": {
+        "page": {},
+        "image": {},
+        "video": {
+          "optionalProperties": {
+            "duration": {"type": "integer"}
+          }
+        }
       }
     },
     "groupInvitation": {
@@ -107,15 +211,27 @@
         "fromMember": {"ref": "memberIdRole"},
         "invitedMember": {"ref": "memberIdRole"},
         "connRequest": {"ref": "connReqUri"},
-        "groupProfile": {"ref": "profile"}
+        "groupProfile": {"ref": "groupProfile"}
       },
       "optionalProperties": {
         "groupLinkId": {"ref": "base64url"},
+        "groupSize": {"type": "integer"},
         "metadata": {
-          "comment": "used to identify invitation via group link"
+          "comment": "groupLinkId is used to identify invitation via group link"
         }
       }
     },
+    "groupLinkInvitation": {
+      "properties": {
+        "fromMember": {"ref": "memberIdRole"},
+        "fromMemberName": {"type": "string"},
+        "invitedMember": {"ref": "memberIdRole"},
+        "groupProfile": {"ref": "groupProfile"}
+      },
+      "optionalProperties": {
+        "groupSize": {"type": "integer"}
+      }
+    },
     "memberIdRole": {
       "properties": {
         "memberId": {"ref": "base64url"},
@@ -127,16 +243,35 @@
         "memberId": {"ref": "base64url"},
         "memberRole": {"ref": "groupMemberRole"},
         "profile": {"ref": "profile"}
+      },
+      "optionalProperties": {
+        "v": {"ref": "chatVersionRange"}
+      }
+    },
+    "memberRestrictions": {
+      "properties": {
+        "restriction": {"ref": "memberRestrictionStatus"}
+      }
+    },
+    "memberRestrictionStatus": {
+      "enum": ["blocked", "unrestricted"]
+    },
+    "chatVersionRange": {
+      "type": "string",
+      "metadata": {
+        "format": "chat version range string encoded as `<min>-<max>`, or as `<number>` if min = max"
       }
     },
     "introInvitation": {
       "properties": {
-        "groupConnReq": {"ref": "connReqUri"},
+        "groupConnReq": {"ref": "connReqUri"}
+      },
+      "optionalProperties": {
         "directConnReq": {"ref": "connReqUri"}
       }
     },
     "groupMemberRole": {
-      "enum": ["author", "member", "admin", "owner"]
+      "enum": ["observer", "author", "member", "admin", "owner"]
     },
     "callInvitation": {
       "properties": {
@@ -257,6 +392,17 @@
         "params": {"ref": "msgContainer"}
       }
     },
+    "x.msg.file.descr": {
+      "properties": {
+        "msgId": {"ref": "base64url"},
+        "params": {
+          "properties": {
+            "msgId": {"ref": "base64url"},
+            "fileDescr": {"ref": "fileDescription"}
+          }
+        }
+      }
+    },
     "x.msg.update": {
       "properties": {
         "msgId": {"ref": "base64url"},
@@ -264,6 +410,10 @@
           "properties": {
             "msgId": {"ref": "base64url"},
             "content": {"ref": "msgContent"}
+          },
+          "optionalProperties": {
+            "ttl": {"type": "integer"},
+            "live": {"type": "boolean"}
           }
         }
       }
@@ -274,6 +424,24 @@
         "params": {
           "properties": {
             "msgId": {"ref": "base64url"}
+          },
+          "optionalProperties": {
+            "memberId": {"ref": "base64url"}
+          }
+        }
+      }
+    },
+    "x.msg.react": {
+      "properties": {
+        "msgId": {"ref": "base64url"},
+        "params": {
+          "properties": {
+            "msgId": {"ref": "base64url"},
+            "reaction": {"ref": "msgReaction"},
+            "add": {"type": "boolean"}
+          },
+          "optionalProperties": {
+            "memberId": {"ref": "base64url"}
           }
         }
       }
@@ -294,8 +462,10 @@
         "params": {
           "properties": {
             "msgId": {"ref": "base64url"},
-            "fileConnReq": {"ref": "connReqUri"},
             "fileName": {"type": "string"}
+          },
+          "optionalProperties": {
+            "fileConnReq": {"ref": "connReqUri"}
           }
         }
       }
@@ -310,6 +480,14 @@
         }
       }
     },
+    "x.direct.del": {
+      "properties": {
+        "msgId": {"ref": "base64url"},
+        "params": {
+          "properties": {}
+        }
+      }
+    },
     "x.grp.inv": {
       "properties": {
         "msgId": {"ref": "base64url"},
@@ -330,6 +508,26 @@
         }
       }
     },
+    "x.grp.link.inv": {
+      "properties": {
+        "msgId": {"ref": "base64url"},
+        "params": {
+          "properties": {
+            "groupLinkInvitation": {"ref": "groupLinkInvitation"}
+          }
+        }
+      }
+    },
+    "x.grp.link.mem": {
+      "properties": {
+        "msgId": {"ref": "base64url"},
+        "params": {
+          "properties": {
+            "profile": {"ref": "profile"}
+          }
+        }
+      }
+    },
     "x.grp.mem.new": {
       "properties": {
         "msgId": {"ref": "base64url"},
@@ -346,6 +544,9 @@
         "params": {
           "properties": {
             "memberInfo": {"ref": "memberInfo"}
+          },
+          "optionalProperties": {
+            "memberRestrictions": {"ref": "memberRestrictions"}
           }
         }
       }
@@ -394,6 +595,27 @@
         }
       }
     },
+    "x.grp.mem.restrict": {
+      "properties": {
+        "msgId": {"ref": "base64url"},
+        "params": {
+          "properties": {
+            "memberId": {"ref": "base64url"},
+            "memberRestrictions": {"ref": "memberRestrictions"}
+          }
+        }
+      }
+    },
+    "x.grp.mem.con": {
+      "properties": {
+        "msgId": {"ref": "base64url"},
+        "params": {
+          "properties": {
+            "memberId": {"ref": "base64url"}
+          }
+        }
+      }
+    },
     "x.grp.mem.del": {
       "properties": {
         "msgId": {"ref": "base64url"},
@@ -425,7 +647,42 @@
         "msgId": {"ref": "base64url"},
         "params": {
           "properties": {
-            "groupProfile": {"ref": "profile"}
+            "groupProfile": {"ref": "groupProfile"}
+          }
+        }
+      }
+    },
+    "x.grp.direct.inv": {
+      "properties": {
+        "msgId": {"ref": "base64url"},
+        "params": {
+          "properties": {
+            "connReq": {"ref": "connReqUri"}
+          },
+          "optionalProperties": {
+            "content": {"ref": "msgContent"}
+          }
+        }
+      }
+    },
+    "x.grp.msg.forward": {
+      "properties": {
+        "msgId": {"ref": "base64url"},
+        "params": {
+          "properties": {
+            "memberId": {"ref": "base64url"},
+            "msg": {
+              "type": "string",
+              "metadata": {
+                "format": "JSON encoded chat message"
+              }
+            },
+            "msgTs": {
+              "type": "string",
+              "metadata": {
+                "format": "ISO8601 UTC time of the message"
+              }
+            }
           }
         }
       }
@@ -436,7 +693,7 @@
         "params": {
           "properties": {
             "callId": {"ref": "base64url"},
-            "invitation": {}
+            "invitation": {"ref": "callInvitation"}
           }
         }
       }