Commit Graph

132 Commits

Author SHA1 Message Date
Greg Heartsfield
ea204761c9 fix: do not show slow queries more than once per sub 2022-12-20 15:41:50 -06:00
Greg Heartsfield
c270ae1434 improvement: reduce event count for db writer pauses 2022-12-20 15:25:24 -06:00
Greg Heartsfield
64bd983cb6 perf: every 5000 persisted events, pause for 500ms for backups
I have observed backups running for a very long time under heavy load,
this introduces some artificial delay to give the online backup enough
time to make progress.
2022-12-20 15:05:04 -06:00
Greg Heartsfield
1c153bc784 perf: shed DB query load when queue gets large 2022-12-20 13:23:21 -06:00
Greg Heartsfield
dc11d9a619 improvement: explicitly rollback transaction on duplicate event 2022-12-20 13:23:04 -06:00
Greg Heartsfield
cd1557787b improvement: log write pool 2022-12-20 13:21:57 -06:00
Greg Heartsfield
86bb7aeb9a improvement: function to check pool capacity 2022-12-20 10:07:01 -06:00
Greg Heartsfield
2cfd384339 perf: drop db handles that are not quickly read 2022-12-19 00:18:39 -06:00
Greg Heartsfield
6f7c080180 improvement: reduce number of writer blocking threads from 4->2 2022-12-18 22:32:31 -06:00
Greg Heartsfield
d833a3e40d perf: reduce logging 2022-12-18 22:11:46 -06:00
Greg Heartsfield
cf144d503d perf: reduce logging for slow queries 2022-12-18 20:47:11 -06:00
Greg Heartsfield
677b7d39e9 improvement: log slow requests that return zero results 2022-12-18 13:42:31 -06:00
Greg Heartsfield
43222d44e5 feat: perform optimization after seeing many events 2022-12-17 23:18:54 -06:00
Greg Heartsfield
b6e33f044f improvement: limit db connection max lifetime 2022-12-17 10:47:35 -06:00
Greg Heartsfield
3c880b2f49 perf: pull distinct to outermost SQL 2022-12-17 07:49:28 -06:00
Greg Heartsfield
7ca9c864f2 improvement: DB pool logging shows used connections directly 2022-12-16 17:01:49 -06:00
Greg Heartsfield
838aafd079 improvement: consistent log messages for client/sub ids 2022-12-16 15:22:27 -06:00
Greg Heartsfield
e554b10ac2 improvement: tweak sub/sql logging for slow queries 2022-12-16 14:55:45 -06:00
Greg Heartsfield
e2de162931 feat: only show SQL in logs for slow queries unless tracing 2022-12-16 08:17:39 -06:00
Greg Heartsfield
bbe359364a refactor: clippy warnings 2022-12-15 08:43:36 -06:00
Greg Heartsfield
5fa24bc9f1 fix: send EOSE when ids list is empty in subscriptions
Fixes: https://todo.sr.ht/~gheartsfield/nostr-rs-relay/50
2022-11-19 10:35:00 -06:00
Greg Heartsfield
4de7490d97 fix: send EOSE when authors list is empty in subscriptions
Fixes: https://todo.sr.ht/~gheartsfield/nostr-rs-relay/49
2022-11-19 10:00:38 -06:00
William Casarin
5a91419d34 feat(NIP-20): send command results to clients
When submitting events to relays, clients currently have no way to know
if an event was successfully committed to the database. This NIP
introduces the concept of command results which are like NOTICE's except
provide more information about if an event was accepted or rejected.

A command result is a JSON object with the following structure that is
returned when an event is successfully saved to the database or
rejected:

	["OK", <event_id>, <true|false>, <message>]

nip20: https://github.com/nostr-protocol/nips/pull/62
2022-11-12 09:12:35 -06:00
Greg Heartsfield
9dd4571bee refactor: reduce level of some common DB logs 2022-11-06 13:49:32 -06:00
Greg Heartsfield
9db5a26b9c refactor: more consistent logging messages 2022-11-05 16:11:20 -05:00
Greg Heartsfield
14b5a51e3a fix: log ephemeral events after send 2022-11-04 07:55:38 -05:00
Greg Heartsfield
8ecce3f566 feat: show client IP in logs 2022-11-02 18:33:44 -05:00
Greg Heartsfield
72f8a1aa5c feat(NIP-26): allow searches for delegated public keys
Implements core NIP-26 delegated event functionality.  Events can
include a `delegation` tag that provides a signature and restrictions
on which events can be delegated.

Notable points on the implementation so far:

* Schema has been upgraded to include an index and new column.
* Basic rune parsing/evaluation to implement the example event in the
  NIP, but no more.
* No special logic for deletion.
* No migration logic for determining delegated authors for
  already-stored events.
2022-10-16 15:25:06 -05:00
Greg Heartsfield
f9693f7ac3 fix(NIP-9): hide events received after their deletions
fixes: https://todo.sr.ht/~gheartsfield/nostr-rs-relay/40
2022-10-08 12:12:41 -05:00
Greg Heartsfield
8a63d88b0b fix: prevent deletion of deletion events 2022-10-08 08:02:16 -05:00
Greg Heartsfield
6633f8b472 feat: replace logging with tracing 2022-09-28 07:19:59 -05:00
Greg Heartsfield
93dfed0a87 refactor: misc clippy suggestions 2022-09-24 19:28:02 -05:00
Greg Heartsfield
bef7ca7e27 refactor: misc clippy suggestions 2022-09-24 09:19:16 -05:00
Greg Heartsfield
8fa58de49a refactor: clippy suggestions 2022-09-24 08:30:22 -05:00
Greg Heartsfield
9ce5057af8 improvement: better log formatting 2022-09-11 10:22:01 -05:00
Greg Heartsfield
2b03f11e5e refactor: remove global/singleton settings object 2022-09-06 06:12:07 -05:00
Greg Heartsfield
e48bae10e6 feat: support in-memory SQLite database 2022-09-06 06:06:01 -05:00
Greg Heartsfield
1b9f364e15 chore: rustfmt 2022-09-02 12:38:31 -05:00
Greg Heartsfield
a843eaa939 improvement: db.rs from clippy 2022-09-02 10:30:51 -05:00
Greg Heartsfield
e6fef37d4e chore: rustfmt 2022-08-21 09:10:19 -07:00
Greg Heartsfield
035cf34673
fix(NIP-12): correctly search for mixed-case hex-like tags
Only lowercase and even-length tag values are stored as binary BLOBs.
Previously there was an error which search results from being returned
if the tag value was mixed-case and could be interpreted as hex.

A new database migration has been created to repair the `tag` table
for existing relays.

fixes: https://todo.sr.ht/~gheartsfield/nostr-rs-relay/37
2022-08-17 16:34:11 -07:00
Greg Heartsfield
be8170342e
fix(NIP-12): multi-tag searches returns correct results
Logic of generated SQL was incorrect, causing multiple tag searches
(as defined in NIP-12) to produce no results.

fixes: https://todo.sr.ht/~gheartsfield/nostr-rs-relay/36
2022-08-11 22:16:10 -07:00
Greg Heartsfield
5058d98ad6
fix(NIP-12): only allow single-char tag filters 2022-08-07 10:15:36 -05:00
Greg Heartsfield
1c14adc766 fix(NIP-01): allow limits on a per-filter basis
The original implementation of subscription limit applied to the
entire query, instead of the specific filter.  Now, each filter gets
its own query limit.  When a limit is applied, the most recent N
events will be returned, otherwise the default is to return the
earliest events (in order), for all matching events.
2022-07-04 17:25:32 -05:00
Semisol
168cfc3b26 feat(NIP-16): Implement NIP16
NIP16 introduces a replaceable and ephemeral event range:
[10000..20000) for replaceable and [20000..30000) for
ephemeral.
2022-05-30 21:43:06 -05:00
Semisol
a36ad378f6 feat(NIP-15): Implement NIP15
NIP15 sends an EOSE notice to clients after all stored events are sent
to allow loading indicators and other use cases.
2022-05-30 21:43:00 -05:00
Greg Heartsfield
9ed3391b46 fix(NIP-09): correct WHERE clause for event deletion 2022-05-10 16:50:52 -05:00
William Casarin
4ad483090e feat(NIP-01): Implement limit
This was quickly sneaked in by fiatjaf per my request[0], it makes many
queries more efficient and allows for paging when combined with until.

It is a bit weird to have multiple limits on each filter... for now we
just choose any or the last limit seen.

[0]: a4aea5337f

Signed-off-by: William Casarin <jb55@jb55.com>
2022-05-10 16:47:56 -05:00
Greg Heartsfield
1d499cf12b feat: handle NIP-09 for deletion events 2022-02-27 11:35:23 -06:00
Greg Heartsfield
53990672ae improvement: move db pool operations closer to query, do not panic on failure 2022-02-23 16:38:16 -06:00
Greg Heartsfield
9c1b21cbfe improvement: more granular perf logging for SQL queries 2022-02-21 09:03:05 -06:00
Greg Heartsfield
5913b9f87a feat: send notices when authorization checks fail 2022-02-13 09:35:54 -06:00
Greg Heartsfield
7056aae227 refactor: create schema module 2022-02-12 09:58:42 -06:00
Greg Heartsfield
753df47443 refactor: create utils/hexrange utility modules 2022-02-12 09:29:38 -06:00
Greg Heartsfield
26a0ce2b32 docs: function/struct comments 2022-02-12 09:29:35 -06:00
Greg Heartsfield
234a8ba0ac feat: limit event publishing to NIP-05 verified users
This adds a new configurable feature to restrict event publishing to
only users with NIP-05 verified metadata.  Domains can be whitelisted
or blacklisted.  Verification expiration and schedules are
configurable.

This upgrades the database to add a table for tracking verification
records.
2022-02-12 09:29:25 -06:00
Greg Heartsfield
98c6fa6f39 feat: allow whitelisting of pubkeys for new events
This adds a configuration option, `authorization.pubkey_whitelist`
which is an array of pubkeys that are allowed to publish events on
this relay.
2022-01-26 21:39:03 -06:00
Greg Heartsfield
ee0de6f875 improvement: clearer and less verbose database logging 2022-01-25 21:42:43 -06:00
Greg Heartsfield
a72eaec3b8 fix: never display hidden events 2022-01-25 20:48:46 -06:00
Greg Heartsfield
f1206e76f2 feat: database reader connection pooling
Added connection pooling for queries, as well as basic configuration
options for min/max connections.
2022-01-25 20:39:24 -06:00
Greg Heartsfield
af453548ee feat: allow author and event id prefix search
This is an experimental non-NIP feature that allows a subscription
filter to include a prefix for authors and events.
2022-01-25 18:23:08 -06:00
Greg Heartsfield
2d28a95ff7 feat: allow arbitrary tag queries
This is an experimental feature, outside of any NIP, that demonstrates
generic tag queries.

Instead of limiting subscription filters to just querying only "e" or
"p" tags (via `#e` or `#p` attributes), any tag can be queried.

As an example, consider an event which uses a tag "url".  With this
modification, a subscription filter could add a top-level field
"#url", with an array of strings as the key.  Exact matches would be
returned.

A NIP is forthcoming to formalize this.
2022-01-22 21:29:15 -06:00
Greg Heartsfield
a3124ccea4 improvement: better sql error handling 2022-01-15 09:42:53 -06:00
Greg Heartsfield
4e51e61d16 improvement: display rate limit messages max once per sec 2022-01-15 09:42:17 -06:00
Greg Heartsfield
afc9a0096a improvement: logging failed queries and timing 2022-01-01 19:25:09 -06:00
Greg Heartsfield
6673fcfd11 feat: implement multi-valued filter searching
NIP-01 now uses arrays instead of scalars.

Fixes https://todo.sr.ht/~gheartsfield/nostr-rs-relay/17
2022-01-01 18:38:52 -06:00
Greg Heartsfield
5ad383f257 fix: incorrect logic on empty filters for hidden events 2021-12-31 16:34:10 -06:00
Greg Heartsfield
4171a8870e feat: reject events that are too large
A new configuration setting controls the maximum size of event
messages, and sends a notice to the client if they exceed it.

Fixes https://todo.sr.ht/~gheartsfield/nostr-rs-relay/14
2021-12-31 15:19:35 -06:00
Greg Heartsfield
5a19a8876f feat: allow database directory configuration
Adds configuration options for database directory, either on command
line through (--db dir-name) or the config.toml file.

Fixes: https://todo.sr.ht/~gheartsfield/nostr-rs-relay/13
2021-12-31 11:51:57 -06:00
Greg Heartsfield
20ee5a054c feat: rate limit event creation
A configuration option, `messages_per_sec`, imposes a global limit on
the rate for which new events can be stored.

Fixes https://todo.sr.ht/~gheartsfield/nostr-rs-relay/6
2021-12-30 21:07:21 -06:00
Greg Heartsfield
c60519de23 feat: hide older contact update events
Type 3 (NIP-02) contact lists are hidden when newer ones are submitted
for the same author.

Fixes https://todo.sr.ht/~gheartsfield/nostr-rs-relay/4
2021-12-30 15:45:03 -06:00
Greg Heartsfield
d72e7a57b6 feat: hide older metadata update events
This updates the database schema to support hiding events.  In this
case, we are hiding older metadata updates when an author provides an
updated event.

Fixes https://todo.sr.ht/~gheartsfield/nostr-rs-relay/11
2021-12-30 13:55:05 -06:00
Greg Heartsfield
100f890284 feat: add until for request filters
This implements an additional filter criteria for selecting events
prior to some timestamp.

See https://github.com/fiatjaf/nostr/issues/39x
2021-12-23 21:38:32 -06:00
Greg Heartsfield
0e288fe678 feat: send messages in order of oldest to newest 2021-12-23 21:36:46 -06:00
Greg Heartsfield
5b6a20dfa6 feat: remove author from subscriptions (NIP-01 Spec)
The `authors` field is sufficient to represent all queries that
`author` could have been used in.  See
https://github.com/fiatjaf/nostr/issues/34 for the discussion leading
to this removal.
2021-12-16 18:53:53 -06:00
Greg Heartsfield
850064b871 fix: handle filters with no criteria 2021-12-12 14:52:55 -06:00
Greg Heartsfield
b3c7852b19 fix: correct SQL query for inserting pubkeys 2021-12-12 14:34:52 -06:00
Greg Heartsfield
1589268eba fix: use database to publish all events
This fixes a race condition where a publisher might send an event, and
immediately after issue a subscription for the same event ID.  Prior
to this change, that event would have been published on the broadcast
channel (and ignored by our publisher, because they had not yet issued
the subscription), but not yet committed to the database.  Their
subscription would trigger a database query which would return zero
results.  Therefore, they would never see the event they published.
The noscl tool is one client that would suffer from this.

Now, all events are broadcast only after they exist in the database,
so a late subscription will always return the event.
2021-12-12 10:20:23 -06:00
Greg Heartsfield
56c40f2be9 refactor: improve error messages 2021-12-12 10:03:28 -06:00
Greg Heartsfield
e732f918f9 refactor: clippy suggestions 2021-12-11 21:56:52 -06:00
Greg Heartsfield
ca0f01c94b docs: add rustdoc comments 2021-12-11 21:43:41 -06:00
Greg Heartsfield
65969a4121 feat: store events in SQLite and allow querying
Events are persisted in a local database, and can be queried through
subscriptions.
2021-12-11 15:48:59 -06:00