From c06139ec990eb216f66404c535eefa53d67f3327 Mon Sep 17 00:00:00 2001 From: Greg Heartsfield Date: Tue, 27 Dec 2022 17:00:14 -0600 Subject: [PATCH] docs: start of database maintenance tips --- docs/database-maintenance.md | 125 +++++++++++++++++++++++++++++++++++ 1 file changed, 125 insertions(+) create mode 100644 docs/database-maintenance.md diff --git a/docs/database-maintenance.md b/docs/database-maintenance.md new file mode 100644 index 0000000..184b6ff --- /dev/null +++ b/docs/database-maintenance.md @@ -0,0 +1,125 @@ +# Database Maintenance + +`nostr-rs-relay` uses the SQLite embedded database to minimize +dependencies and overall footprint of running a relay. If traffic is +light, the relay should just run with very little need for +intervention. For heavily trafficked relays, there are a number of +steps that the operator may need to take to maintain performance and +limit disk usage. + +This maintenance guide is current as of version `0.7.14`. Future +versions may incorporate and automate some of these steps. + +## Backing Up the Database + +To prevent data loss, the database should be backed up regularly. The +recommended method is to use the `sqlite3` command to perform an +"Online Backup". This can be done while the relay is running, queries +can still run and events will be persisted during the backup. + +The following commands will perform a backup of the database to a +dated file, and then compress to minimize size: + +```console +BACKUP_FILE=/var/backups/nostr/`date +%Y%m%d_%H%M`.db +sqlite3 -readonly /apps/nostr-relay/nostr.db ".backup $BACKUP_FILE +sqlite3 $BACKUP_FILE "vacuum;" +bzip2 -9 $BACKUP_FILE +``` + +Nostr events are very compressible. Expect a compression ratio on the +order of 4:1, resulting in a 75% space saving. + +## Vacuuming the Database + +As the database is updated, it can become fragmented. Performing a +full `vacuum` will rebuild the entire database file, and can reduce +space. Running this may reduce the size of the database file, +especially if a large amount of data was updated or deleted. + +```console +vacuum; +``` + +## Clearing Hidden Events + +When events are deleted, either through deletion events, metadata or +follower updates, or a replaceable event kind, the event is not +actually removed from the database. Instead, a flag `HIDDEN` is set +to true for the event, which excludes it from search results. The +original intent was to ensure that subsequent rebroadcasts of the +event would be easily detected as having been deleted, and would not +need to be stored again. In practice, this decision causes excessive +growth of the `tags` table, since all the previous followers are +retained for those `HIDDEN` events. + +The `event` and especially the `tag` table can be significantly +reduced in size by running these commands: + +```console +PRAGMA foreign_keys = ON; +delete from event where HIDDEN=true; +``` + +## Manually Removing Events + +For a variety of reasons, an operator may wish to remove some events +from the database. The only way of achieving this today is with +manually run SQL commands. + +It is recommended to have a good backup prior to manually running SQL +commands! + +In all cases, it is mandatory to enable foreign keys, and this must be +done for every connection. Otherwise, you will likely orphan rows in +the `tag` table. + +### Deleting Specific Event + +```console +PRAGMA foreign_keys = ON; +delete from event where event_hash=x'00000000000c1271675dc86e3e1dd1336827bccabb90dc4c9d3b4465efefe00e'; +``` + +### Deleting All Events for Pubkey + +```console +PRAGMA foreign_keys = ON; +delete from event where author=x'000000000002c7831d9c5a99f183afc2813a6f69a16edda7f6fc0ed8110566e6'; +``` + +### Deleting All Events of a Kind + + +```console +PRAGMA foreign_keys = ON; +delete from event where kind=70202; +``` + +### Deleting Old Events + +In this scenario, we wish to delete any event that has been stored by +our relay for more than 1 month. Crucially, this is based on when the +event was stored, not when the event says it was created. If an event +has a `created` field of 2 years ago, but was first sent to our relay +yesterday, it would not be deleted in this scenario. Keep in mind, we +do not track anything for re-broadcast events that we already have, so +this is not a very effective way of implementing a "least recently +seen" policy. + +```console +PRAGMA foreign_keys = ON; +TODO! +``` + +### Delete Profile Events with No Recent Events + +Many users create profiles, post a "hello world" event, and then never +appear again (likely using an ephemeral keypair that was lost in the +browser cache). We can find these accounts and remove them after some +time. + +```console +PRAGMA foreign_keys = ON; +TODO! +```