nostr/nips
1
0
Fork 0
mirror of https://github.com/nostr-protocol/nips.git synced 2024-09-19 22:35:50 -04:00
Code Issues Actions Packages Projects Releases Wiki Activity
f83c49fc39
nips/66.md
sandwich f83c49fc39 draft6
2024-02-11 20:25:19 +00:00

24 KiB
Raw Blame History

NIP-66: Relay Meta and Monitoring System

draft optional

Table of Kinds

This NIP defines three (3) event kinds, 30066, 30166 and 10166

kind name description
10166 Relay Monitor Registration An RE that stores data to publish intent to monitor and details around their activities
30166 Relay Discoverability A PRE composed of only indexed tags used to discover relays [d]
30066 Relay Metadata A PRE composed of human readable, non-indexed tags used to find data on a relay [d]

High Level

nostr is a network composed of dumb relays that exhibit different behaviors that are mostly, but not entirely self-reported by their NIP-11 document. However, discoverabililty of relays, and acquiring metadata for relays at the protocol level is not presently implemented. Additionally, self-reported data has an extremely low cieling and is not always reliable.

History

  • draft1 implemented in January 2023 and proposed February 2023. "Too many one-letter indexable tags"
  • draft2 experimented with March 2023 "Everything should be expirable"
  • draft3 experimented, never proposed
  • draft4 experimented, never proposed
  • draft5 proposed December 2023, "Needs to be indexed"
  • draft6 proposed February 2024, "Just use one-letter indexable tags" (lol) and split discoverability and parse cases.

Status

Starting in NIP-66@draft6 Relay Metadata and Relay Discoverability are considered as two different cases and are split into two different events. However, there remains a pattern for monitors to flag that their 30066 events are indexable.

Ontology

  • Relay Operator: someone who operates a relay
  • Monitor Service: a group or individual that monitors the network
  • Monitor: a piece of software that aggregates network data and publishes that data at a specified frequency.
  • Check: a specific data point that is tested or aggregated by a monitor.

10166: "Relay Monitor Registration" Events

Summary

10166 is a replacable event herein referred to as "Relay Monitor Registration" events. These events contain information about a publisher's intent to monitor and publish data as NIP-66 events.

Purpose

To provide a directory of monitors, their intent to publish, their criteria and parameters of monitoring activities.

Kind Usage

NIP-66 kinds can be used by both "Relay Operators" and "Relay Monitors," with the exception that "Relay Operators" SHOULD NOT use 10166

Kind Monitors Operators
10166 x
30066 x x
30166 x x

Schema

Summary

Non-Indexed Tags

  • url A URL with human readable information about the monitor's activities. If not included, it is assumed these details are included in the monitor's website member in their kind: 0 JSON for the pubkey that signed the 10066 event.

  • timeout The timeout values for various tests. Index 1 is the monitor's timeout in milliseconds. Index 2 describes what test the timeout is used for, for example open, read, write, info, etc. If no index 2 is provided, it is inferred that the timeout provided applies to all tests.

    [ "timeout", "open", "2000" ],
[ "timeout", "read", "2000" ],
[ "timeout", "write", "3000" ],
[ "timeout", "nip11", "2000" ],
[ "timeout", "ssl", "4000" ]

  • frequency The frequency at which the monitor publishes events. A string-integer at index 1 represents the frequency in seconds of the checks. There should only be 1 frequency per monitor.

    [ "frequency", "3600" ]

Indexed Tags

  • k "Kinds" should be a the string-representation of an integer that describes the NIP-66 kinds this monitor publishes. Note: Kind 10166 does not need to be listed here.

    [ "k", "30066" ],
[ "k", "30166" ]

    While 30066 and 30166 have a high-level separation of concerns by Metadata and Discoverability, a monitor can decide to flag 30066 as indexable, and include indexed tags defined in 30166 within their 30066 Relay Metadata events.

    [ "k", "30066", "indexed"],

  • c "Checks" should be a lowercase string describing the check(s) conducted by a monitor. Some examples are: rtt, nip11, ssl, dns, geo and count. Other checks can be added via consensus.

    [ "c", "rtt" ],
[ "c", "nip11" ],
[ "c", "dns" ],
[ "c", "geo" ],
[ "c", "dns" ],
[ "c", "count" ]

  • n "Counts" should be a lower-snake-case string describing the counts(s) conducted by a monitor at index 1. Some examples are: total_active_users, total_note_zaps, total_note_reactions, events_per_minute, note_publish_rate. Counts are not atomized, their values are entirely ad-hoc. The time period of the count is defined in 30066

    [ "n", "users_total_active" ],
[ "n", "note_total_zaps" ],
[ "n", "note_total_reactions" ],
[ "n", "note_publish_rate" ]

  • o The hex pubkey of the owner of the monitor. A "Monitor Service" will likely run several monitors.

    [ "o", "b3b0d247f66bf40c4c9f4ce721abfe1fd3b7529fbc1ea5e64d5f0f8df3a4b6e6" ]

  • g: NIP-66 leverages a draft NIP for geo tags, which is backwards compatible with 0 collisions for legacy g tags from NIP-52. See YAGT

    [ "G", "countryCode" ]
[ "g", "US", "countryCode"]
[ "g", "USA", "countryCode" ]
[ "G", "regionCode" ]
[ "g", "US-CA", "regionCode"]
[ "g", "9r1652whz" ]
[ "g", "9r1652wh" ]
[ "g", "9r1652w" ]
[ "g", "9r1652" ]
[ "g", "9r165" ]
[ "g", "9r16" ]
[ "g", "9r1" ]
[ "g", "9r" ]
[ "g", "9" ]

Other Requirements

Monitors should have the following

  • A published 0 (NIP-1) event
  • A published 10002 (NIP-65) event that defines the relays the monitor publishes to.

Example

{
  "id": "<eventid>",
  "pubkey": "<monitor's pubkey>",
  "created_at": "<created_at  [some recent date ...]>",
  "signature": "<signature>",
  "content": "",
  "tags": [  
    [ "url", "https://a.good.monitor"],

    [ "timeout", "open", "5000"  ],
    [ "timeout", "read", "3000"  ],
    [ "timeout", "write", "3000" ],
    [ "timeout", "nip11", "3000" ],

    [ "frequency", "3600" ],

    [ "c", "ws" ],
    [ "c", "nip11" ],
    [ "c", "ssl" ],
    [ "c", "dns" ],
    [ "c", "geo" ],
    [ "c", "count" ],

    [ "n", "total_active_users" ],
    [ "n", "total_note_zaps" ],
    [ "n", "total_note_reactions" ],
    [ "n", "events_per_minute" ],

    [ "k", "30066" ],
    [ "k", "30166" ],

    ["g", "ww8p1r4t8"],
    ["G", "countryCode"],
    ["g", "NL", "countryCode"]
  ]
}

30166: "Relay Discoverability"

Summary

30166 is a NIP-33 Parameterized-Replaceable Event [PRE], referred to as a "Relay Discoverability" event. These events are optimized with a small footprint for protocol-level relay discoverability, and contain no unindexed tags.

Purpose

Enables protocol-level discovery of relays on a variety of parameters while ommitting the large footprint of metadata in the "Discovery Case"

Schema

Summary

30166 events contain only indexable tags.

Indexed Tags

  • d The relay URL. The #d tag must be included in the event.tags[] array. Index position 1 must be the relay websocket URL. The URL SHOULD be normalized

    [ "d", "wss://history.nostr.watch/"]

  • n: Network

    [ "n", "clearnet" ]

  • N: Supported Nips From NIP-11 "Informational Document" nip11.supported_nips[]

    [ "N", "33" ]

  • R: Requirements NIP-11 "Informational Document" nip11.limitations.payment_required, nip11.limitations.auth_required and/or any other boolean value within nip11.limitations[] that is added in the future

    [ "R", "payment" ],
[ "R", "auth" ],

    Since nostr protocol does not currently support filtering on whether an indexed tag IS or IS NOT set, to make "public" and "no auth" relays discoverable requires a ! flag

    [ "R", "!payment" ], //is public
[ "R", "!auth" ], //no auth

  • l: "Language Tags" From NIP-11 "Informational Document" nip11.language_tags[]

    [ "l", "en" ],
[ "l", "en-419" ]

  • t: "Tags" From NIP-11 "Informational Document" nip11.tags[]

    [ "t", "nsfw" ]

  • k: Accepted Kinds (Only possible with self-reporting until accepted and rejected kinds are added to NIP-11)

    [ "k", "0" ],
[ "k", "3" ],
[ "k", "10002" ]

  • g: NIP-66 leverages a draft NIP for geo tags, which is backwards compatible with 0 collisions for legacy g tags from NIP-52. See YAGT

    [ "G", "countryCode" ],
[ "g", "US", "countryCode"],
[ "g", "USA", "countryCode" ],
[ "G", "regionCode" ],
[ "g", "US-CA", "regionCode"],
[ "g", "9r1652whz" ],
[ "g", "9r1652wh" ],
[ "g", "9r1652w" ],
[ "g", "9r1652" ],
[ "g", "9r165" ],
[ "g", "9r16" ],
[ "g", "9r1" ],
[ "g", "9r" ],
[ "g", "9" ]

Required Tags

In order for discoverability to work as intended, please review the following table.

tag should be set notes
d yes NIP-33
R yes Required so that public and no-auth relays are discoverable
n no
N no
l no
t no
o no
k no

Example

Relay was online, and you can find it using a bunch of different indexable tags

{
  "id": "<eventid>",
  "pubkey": "<monitor's pubkey>",
  "created_at": "<created_at  [some recent date ...]>",
  "signature": "<signature>",
  "content": "{}",
  "kind": 30166,
  "tags": [  
    ["d","wss://some.relay/"],
    ["n", "clearnet"],
    ["N", "7"],
    ["N", "33"],
    ["k", "0" ],
    ["k", "3" ],
    ["k", "10002" ],
    ["R", "!payment"],
    ["R", "!auth"],
    ["l", "en"],
    ["l", "es"],
    ["l", "fr"],
    ["g", "ww8p1r4t8"],
    ["G", "countryCode"],
    ["g", "NL", "countryCode"]
  ]
}

Tag availability mapping

When a monitor flags check support [c tag] in their 10166 kind, related indexable fields SHOULD be present in their published 30166 event kinds.

check indexable tags
rtt n/a
nip11 N R l t
ssl n/a
dns n/a
geo n/a
Important Notes
  • Relay Monitors that publish 30166 events SHOULD at a minimum be conducting rtt checks, namely websocket open, so that clients can determine liveness.
  • k indexable tags are not associated to any other standard and thus cannot be flagged for support in 10166 until NIP-11 support kind flagging.

30066: "Relay Metadata"

Summary

30066 is a NIP-33 Parameterized-Replaceable Event [PRE], referred to as a "Relay Metadata" event. These events store the existence and optionally some data relating to the relay. These events can be published by a monitor or self published by relay operators

Purpose

To store useful, computationally expensive data derived from checks and other metadata about relays. This data is found by filtering the normalized relay URL with the d tag. Determining liveness 30066 can be used by filtering against since with respect to the frequency value set in a Relay Monitor's 10166 event kind.

Schema

event.content

content MAY contain stringified NIP-11 JSON, since the NIP-11 cannot be serialized into tags with any sane patterns.

event.tags

Summary
  1. "Relay Metadata" events have one (1) required tag
  2. Unindexed Tags use grouping by defining the group at index position 1, the key at index position 2 and the value at index position 3.

Indexed Tags

  • d The relay URL. The #d tag MUST be included in the event.tags[] array. Index position 1 SHOULD be the relay websocket URL. The URL SHOULD be normalized 
    [ "d", "wss://history.nostr.watch/"]
  • 30066 events MAY contain any of the indexable tags included in 30166 kind events (See instructions for this case in 10166's schema, specifically under k)

Unindexed Tags

  • rtt group contains key/value pairs that describe the round trip time of a particular operation. Values at position 2... SHOULD treated as an array by consumers. This array can be used to mathematically find min, max, average and median values. For example Math.min(...["190", "201", "540"].map( n => parseInt(n) )). Some obvious index 1 keys are websocket open, read and write round-trip times, as well as nip11 https request round-trip time.

    ["rtt", "open", "190", "201", "540"],
["rtt", "read", "35"],
["rtt", "write", "701"]

  • ssl group contains key/value pairs that describe the checked status of a relay's SSL Certificate. Index 1 keys for ssl can be atomized and should be one of the following valid_from, valid_to, subject_alt_name, fingerprint, fingerprint256, fingerprint512, ext_key_usage, exponent, serial_number, modulus and/or pem_encoded. There SHOULD NOT be duplicate keys.

    ["ssl", "valid_from", "<timestamp_seconds>"],
["ssl", "valid_to", "<timestamp_seconds>"]

  • dns group contains key/value pairs that describe a DNS lookup for the relay from the perspective of a given monitor. It can provide insights on other meta data in the event. Index 1 keys for dns can be atomized and should be one of the following as, asn, ipv4 or ipv6. There MAY be duplicate keys.

    ["dns", "asn", "19281"],
["dns", "as", "QUAD9-AS-1, CH"],
["dns", "ipv4", "1.1.1.1"],
["dns", "ipv6", "2001:db8:3333:4444:5555:6666:7777:8888"]

  • nip11 group is a special group that extracts specific values as long as they are no more than 1 level deep and do not contain arrays of objects. This data MAY be duplicated as stringified NIP-11 JSON in content.

    ["nip11", "name", "A nostr relay"],
["nip11", "description", "It stores notes and other stuff"],
["nip11", "owner", "<pubkey:hex>"],
["nip11", "software", "<software>"],
["nip11", "version", "<version>"]

    For arrays

    ["nip11", "supported_nips", "3", "33"],

  • count Meta values for arbitrary counts related to a relay. Index position 1 is the value expressed as the string representation of an integer or float. Index position 2 is the key and describes the count, such as total_users or total_events. Index position 3 MAY be set and defines the "period" in seconds that the count at position 2 was calculated. Counts SHOULD only be included when representing unique or computationally expensive counts, not ones that can be easily achieved via NIP-45 counts. A count's key SHOULD be snake case. There MAY be zero (0) to many count tags.

    ["count", "users_total_active", "201", "86400"],
["count", "note_total_zaps", "87", "86400"],
["count", "note_total_reactions", "125", "86400"],
["count", "note_publish_rate", "24.1", "3600"]

  • other: A group for other type checks that could be a derivative of another check or unassociated to a specific check. A good example is network which is is determined by parsing the URL to attempt to identify which network it can be found on, and would likely already be in the monitor's cache as it was determined for discoverability purposes.

    ["other", "network", "clearnet"]

Tag Usage

For self-reported 30066 events by "Relay Operators", certain tags SHOULD NOT be self-published or otherwised ignored by clients if present. Please reference the table below.

Monitor Relay Operators
rtt x
nip11 x x
dns x x
geo x x
ssl x
count x x
other x x

Methodology

A Relay Monitor publishes 30066 events exclusively when a relay is reachable. This approach ensures that the last known active state of the relay is maintained and recorded in created_at value of event. Based on this data, several inferences can be drawn about the relay's status and characteristics.

  1. Clients and/or users can set a custom threshold to establish a cutoff timestamp for filtering events using since. This timestamp helps in identifying which relays are currently online. Selecting a lower threshold value results in a stricter criterion for relay uptime, making the filter more sensitive to brief downtimes. Conversely, choosing a higher threshold value creates a more lenient filter, allowing relays with longer downtimes to still be considered as online.

  2. In determining whether a relay is 'dead,' the decision is solely at the discretion of the client or user. The are responsible for setting and applying arbitrary thresholds using until filters or post-processing to make this determination. This approach underscores that the classification of a relay as 'dead' is a subjective decision, varying according to each client's or user's assessment criteria, rather than a fixed status provided by the monitor.

  3. For Relay Metadata events that have become outdated, the retained data points remain valuable. They offer insights and information about the relay's characteristics, performance and impact on the network, which might not be currently accessible due to the relay being offline.

Important Notes

  • Relay Monitors SHOULD run checks and publish events according to the frequency value set in their respective 10166 kind event.
  • If a relay monitor publishes both 30166 and 30066 kind events, they SHOULD publish both on every check.
  • Clients SHOULD only use monitors reporting rtt values for websockets (particularly open) for determining the "liveness" or "deadness" of a relay.
  • Clients SHOULD consider the frequency value in a monitor's 10166 when trying to reach determination of "liveness".
  • NIP-11 values are provided as means to filter and discover relays, however **SHOULD NOT be used as a replacement to NIP-11.
  • A particular relay's retention policy could conflict with subjective thresholds used in determination of "liveness" or "deadness" of relays and so care SHOULD be taken with regards to chosen relays for NIP-66 by both monitors and consumers.

Examples

Minimum Requirements

"Relay was online at <created_at>"

{
  "id": "<eventid>",
  "pubkey": "<pubkey>",
  "created_at": "<created_at>",
  "signature": "<signature>",
  "content": "",
  "kind": 30066,
  "tags": [  
    ["d","wss://some.relay/"]
  ]
}
Relay is checked and online

"Relay was online at <created_at> and here's some meta-data this monitor believes is accurate"

{
  "id": "<eventid>",
  "pubkey": "<monitor's pubkey>",
  "created_at": "<created_at>",
  "signature": "<signature>",
  "content": "{\"contact\":\"pablof7z.com\",\"description\":\"Nostr's Purple Pages\",\"name\":\"purplepag.es\",\"pubkey\":\"fa984bd7dbb282f07e16e7ae87b26a2a7b9b90b7246a44771f0cf5ae58018f52\",\"software\":\"git+https://github.com/hoytech/strfry.git\",\"supported_nips\":[1,2,4,9,11,12,16,20,22,28,33,40],\"version\":\"0.9.6-7-g7196547\"}",
  "kind": 30066,
  "tags": [  
    ["d","wss://some.relay/"],
    ["other", "network", "clearnet"]
    ["rtt", "open", "190", "201", "540"],
    ["rtt", "read", "35"],
    ["rtt", "write", "701"],
    ["nip11", "name", "A nostr relay"],
    ["nip11", "description", "It stores notes and other stuff"],
    ["nip11", "owner", "<pubkey:hex>"],
    ["nip11", "software", "<software>"],
    ["nip11", "version", "<version>"],
    ["ssl", "valid_from", "<timestamp>"],
    ["ssl", "valid_to", "<timestamp>"],
    ["geo", "geohash", "ww8p1r4t8"],
    ["geo", "countryCode", "NL"],
    ["dns", "asn", "19281"],
    ["dns", "as", "QUAD9-AS-1, CH"],
    ["dns", "ipv4", "1.1.1.1"],
    ["dns", "ipv6", "2001:db8:3333:4444:5555:6666:7777:8888"],
    ["count", "users_total_active", "201", "86400"],
    ["count", "note_total_zaps", "87", "86400"],
    ["count", "note_total_reactions", "125", "86400"],
    ["count", "note_publish_rate", "24.1", "3600"]
  ]
}

Testing Criteria

The testing criteria to determine conditions defined in event's tags MAY be subjective and MAY vary between monitors.

Limitations

The data in 30066 MAY be erroneous, intentionally or otherwise. Where accuracy is required, the data in 30066 events SHOULD be subscribed to by populating the authors filter array with the public keys of subjectively trusted monitors and where security or privacy is a concern any republished values (such as NIP-11 values) should instead be attained from the source (https). All data is for informational purposes and to make finding and filtering through relays through nostr a possiblity.

Ammending NIP-66

Any test results that cannot be expressed through NIP-66 should be ammended to the nip following discussion and general consensus

Use Cases

  1. Geographic Relay Discovery: Identify relays situated near a specific geographic location or within a particular country, facilitating localized network interactions.

  2. NIP Support Filtering: Search for relays based on their support for specific Nostr Improvement Proposals (NIPs), ensuring compatibility with desired protocol features.

  3. Accessibility Search: Locate relays that are free to use, helping users find cost-effective or no-cost options for their network interactions.

  4. Real-Time Status Monitoring: Utilize a status client to display up-to-date statuses of various relays, providing insights into their current operational state.

  5. Relay Network Analysis: Analyze connections and patterns between relays using their IP addresses, aiding in network topology understanding and security assessments.

  6. Error Detection in Relay Listings: Spot and rectify erroneous entries in relay lists, ensuring the accuracy and reliability of relay data.

  7. Performance Benchmarking: Compare relays based on performance metrics like round-trip times and uptime, aiding in the selection of the most efficient relays for specific needs.

  8. Security and Compliance Checks: Evaluate relays for adherence to security standards and regulatory compliance, essential for users with specific security and privacy requirements.

  9. Language and Content Filtering: Identify relays catering to specific languages or content types, enabling users to engage in a more targeted and relevant social networking experience.

  10. Data-Driven Relay Selection: Make informed choices about which relays to connect to, based on comprehensive metadata including user counts, event frequencies, network types and more.