nips/29.md

NIP-29

Time-Based Sync

draft optional author:vitorpamplona

This NIP describes a simple event database reconciliation procedure for clients and relays (peers). Both sides hash the same groups of event ids and compare the resulting hashes to determine which subsets should be downloaded or uploaded to each other. The procedure is ideal for clients that use a local database and must guarantee the database is in sync with each relay without downloading or uploading all events again.

Sync Protocol

The syncing peer (client or a relay) sends a HASH-REQ message to a relay with a subscription ID and filters for the content to be synced.

Request:

[
    "HASH-REQ",
    "<subscription ID string>",
    "<WINDOW-SIZE>"	
    <nostr filter>, <nostr filter2>, <nostr filter3>
]

Upon receiving a HASH-REQ the relay MUST:

1. Apply all nostr filters in the subscription to the database
2. Sort all resulting events by .created_at descending (as usual)
3. Group by the first WINDOW-SIZE chars of the stringified .created_at
4. For each group, create an array of event ids: ["idHex1", "idHex2", "idHex3"] in the same order as before, JSON-serialize the array, and hash it using SHA-256
5. Return the group list, with their truncated .created_at identifier and hash to the peer.

After calculating the hashes, the relay responds with an EOSE-ended sequence of

Response:

[
    "HASH-RES",
    "<subscription ID string>",
    "<TRUNCATED_CREATED_AT>",
    "<SHA256(JSON.stringify([event1.id, event2.id, event3.id, ...])) in hex>"
]

The peer then compares the receiving hashes with those stored locally and, if different, creates a new filter to either refine the window size OR to download all events within the missing window.

The choice of window size is use case dependent. Clients can start with a large window after the first round of results adjust the filters and reduce the window size to further reduce the group to be re-downloaded.

Appendix A: Window Size explanation

The WINDOW-SIZE is a number between 0 and 10 and represents each order of magnitude of .created_at. It defines the size of each group as well as their start and end times for the .created_at field.

By truncating the .created_at to the first WINDOW-SIZE chars, it effectively creates the groups below:

• WINDOW-SIZE=0: Returns only one hash for the entire subscription
• WINDOW-SIZE=1: Groups by periods of 1000000000 seconds (~31.7 years): Periods start with *000000000 and end with *999999999
• WINDOW-SIZE=2: Groups by periods of 100000000 seconds (~3.17 years): Periods start with **00000000 and end with **99999999
• WINDOW-SIZE=3: Groups by periods of 10000000 seconds (~16.53 weeks): Periods start with ***0000000 and end with ***9999999
• WINDOW-SIZE=4: Groups by periods of 1000000 seconds (~11.57 days): Periods start with ****000000 and end with ****999999
• WINDOW-SIZE=5: Groups by periods of 100000 seconds (~1.16 days): Periods start with *****00000 and end with *****99999
• WINDOW-SIZE=6: Groups by periods of 10000 seconds (~2.77 hours): Periods start with ******0000 and end with ******9999
• WINDOW-SIZE=7: Groups by periods of 1000 seconds (~16.66 minutes): Periods start with *******000 and end with *******999
• WINDOW-SIZE=8: Groups by periods of 100 seconds (~1.66 minutes): Periods start with ********00 and end with ********99
• WINDOW-SIZE=9: Groups by periods of 10 seconds: Periods start with *********0 and end with *********9
• WINDOW-SIZE=10: Groups by periods of 1 second: Periods start with ********** and end with **********

Notice that each group starts at 0 (inclusive) and ends at their last possible number 9 (inclusive).