This page is part of a static HTML representation of the TiddlyWiki at https://tiddlywiki.com/

2025-02-18 - User Management

7th May 2025 at 12:31pm

Described here is a full blown role-based identity & access management system. However we are unlikely to need this level of sophistication immediately.

Logical groupings and definitions

  • A user is a logical grouping of logins.
  • An organization ("content space") is a logical grouping of users and the content they interact with.
  • The organization owner is a user who sets the privileges other users have in the organization.
    • If enabled, every user is the owner of their own personal organization.
    • If enabled, they may own extra organizations.
    • The organization owner has all privileges in the organization except those they explicitly deny themself.
    • They cannot deny themself the ability to change privileges in the organization.
    • They may delegate to managers.
  • Admins are users with site-wide privileges.
    • The admin role is separate from the owner role. They are not site-wide owners.
    • Admins can take various actions on extra orginizations, depending on site settings.
    • Admins can manage users access to the site and take related actions.
    • Depending on settings, admins cannot view content that does not belong to them, unless shared (the exact privacy details here aren't important, it's the technical features that I'm including this for).
  • Site admins can define a public content space which everyone can view, and an additional content space which authenticated users can view. These act as implicit organizations with admins given permissions as managers.

Even though it sounds like I'm expecting this to be some kind of public document sharing platform or online collaboration, the setup actually has multiple use-cases within a single organization that are just as complicated.

To recap the various roles in an organization

  • admin - Granted site-wide permissions.
  • owner - Granted organization level permissions.
  • manager - Delegated permission from owners.
  • user - Explicitly invited and added to the organization user list by owners or managers.
  • auth - Visitor signed in and not in the user list.
  • anon - Visitor not signed into the site.

There are two permission levels within the wiki

  • writer - can edit tiddlers, optionally filtered
  • reader - can view tiddlers, optionally filtered

Owners can create permission profiles which define reader and writer filters, then assign them to users or groups (or to their access to a specific wiki), and when they update the permission profile the changes apply everywhere.

I mean look, at some point I'm just implementing an entire Identity Access Management service.

JSON settings file

A short list of options in a JSON file alongside the database (or with the database settings) which determines some permanent settings that depend on the use case.

  • Whether admins can change user email address or oauth identity and set user passwords (account takeover).
  • Whether admins can view site content unrestricted (account privacy).
  • Sets the max visibility owners and admins can set (since that shouldn't need to change).
  • The default default visibility (before orgs change it).

settings for content spaces (organizations)

  • Whether new users get their own personal content space (personal organization).
  • Whether non-admin users can be given additional content spaces (extra organizations).
  • Whether non-admin users can create their own additional content spaces (extra organizations).
  • Whether additional content spaces created by non-admin users can be removed from them by admins (this will depend on the use case).

2025-04-03 - Phases of Development

7th May 2025 at 12:32pm

Welcome to the MultiWikiServer wiki!

Phases of development

Phase 1 is to get a fully working system that can be extended.

The features here should cover all of the core requirements.

  • Database fully async and concurrent with transactions and row locking.
  • Recipes and bags fully supported.
    • Support for both server-wide (wiki-like) and per-user (doc-like) recipes.
    • Shadow bags to hold each user's drafts and customizations for server-wide or shared recipes.
    • Possible integration of various tiddler namespace behaviors, like private User:{username} tiddlers.
    • Option on the server for users to propose edits to existing tiddlers, probably by moving a tiddler to a different bag.
  • The concept of multiple users fully supported. Auth state has a userID property.
  • Auth with username and password by default, but allowing custom integrations to be built that use third-party auth libraries or services. Several of us already have different ideas in mind, and we need to build all the required mechanisms in phase 1 in order to make sure we're properly supporting them, even if we don't actually support the third-party integrations themselves until phase 2.
  • Multiple sessions editing the same wiki supported by the server, with revision mechanisms to prevent overwriting edits with an old version.
  • Sharing read or write access with other users via invite links.

Phase 2 is to add all the other important features

How the server classes work together

This implements a server that abstracts the various HTTP protocols into a single request state object which it then hands out to the routes.

  • Streamer abstracts away the difference between HTTP/1.1 and HTTP/2 to allow both to be handled transparently.
    • Normalizes the request and response into a single class.
    • Currently it doesn't support push streams, but we could change that.
  • Listeners handle the server listener and send requests they recieve to the router.
    • ListenerHTTP: Sets up and handles HTTP requests using the http module.
    • ListenerHTTPS: Sets up and handles HTTPS requests using the http2 module (with http1.1 support).
  • Router: The main server instance. Handles route matching and sets up a state object for each request. It also instantiates subsystems.
    • AuthState (subsystem): Contains all authentication logic and is called at various points in the request process. It may throw to abort the request.
  • StateObject: Contains everything routes have access to, including a database connection. Subsystems may create their own state classes which take the StateObject as a constructor argument.

Thoughts on routing

Routing generally works well, but the current routes are all strictly path based. They mix various concerns into one module based on path similarities.

I think it would be a lot more streamlined to separate the concerns into different sections. Each of those would operate as their own submodule.

Why? Because this is how complicated permissions can get:

  • Auth determines who the user is.
    • Auth adapters that connect third-party auth services, with fields like profile pic, display name, email, etc.
    • Auth strategies like session cookies or access tokens.
    • Login and registration forms, email and phone interactions, etc.
  • Users contains all the code for changing what the user is allowed to do.
    • List of users, with tabs for pending, roles, etc.
    • Handle user registrations and initial setup.
    • Manage sharing and collaboration permissions: Roles (eventually)
  • Recipes shows the user their available recipes and lets them create or modify.
    • (Phase 1): Users create their own recipes and bags and specify read/write permissions for those recipes and bags.
    • (Phase 2): Admins can define scoped bags which are added to recipes based on the user's role.
  • Wikis contains all the code that runs when accessing the wiki itself.
    • Tiddler saving and loading based on the recipe instructions.
    • Uploads and other third-party integrations requiring server support.

2025-04-13 - Plugins

7th May 2025 at 12:35pm

We need the MWS server to be as modular as possible. Obviously certain things, like the route implementations, are difficult to make modular. But various sections are much more self-contained, or interact with the rest of the codebase in a small, well-defined way, and these can be turned into pluggable services.

Server routes for various sections of the UI can be kept separate, so the recipe/bag manger, user manager, and actual wiki routes can be put in separate modules so that one of them can easily be rewritten without affecting the others.

All of them interact with the database via the prisma adapter, which can be modified to handle different databases. We should support mariadb, sqlite, and postgresql to cover the main ones. Making sure the prisma client covers all three is probably important. There may be small differences between the prisma client for different database types, although this will be reflected in the types, so we should be able to figure that out pretty easily.

The attachment store isn't something I've taken a close look at.

The tiddlywiki instance on the server might stay. At this point I don't think we actually need it for anything besides rendering the index wiki. That might be moved to the start command instead of being part of the mws runtime. But it's also used for various import and export commands which still need to be available on the cli. We could make loading optional, or unload it after commands have completed, or once mws-listen starts.

I don't think things are quite where they should be with the client either. Currently we're adding recipe tiddlers into the wiki page dynamically, which is expected, but there are like six tiddlers that are being rendered statically, which doesn't really make sense. It also doesn't make sense that we're dumping some plugins into the database but rendering core from tiddlywiki, since this results in a version mismatch.

At the same time, we really don't want to be rendering plugins every time. They do need to be cached somewhere so they can be loaded quickly. I wonder if it would work to cache them in the wiki folder, per tiddlywiki version, so if you upgrade it would just create a new folder. The boot tiddlers would be cached in the folder as well. If we do it right, we wouldn't even have to parse the file, just read it onto the wire. We'd probably need an index file to keep everything straight. We could add plugins/themes/languages support to the wiki folder as well, which would also get cached in the same way. We could make a way for caching to be disabled, perhaps by adding a field to plugin.info.

It would be useful if the plugin syntax could specify NPM modules. We already have the + and ++ syntax. I'm not sure exactly how it'd work, but the NPM package would need to determine it's own path, which is fairly simple, and then export that so it can be imported via the standard import mechanism. Obviously the package would need to be installed, and it should probably be imported into the run file and then added as an absolute path to the list of imports. Actually, I guess that's already possible, so I just need to add the list of imports part of it.

-- Arlen22

2025-05-04

7th May 2025 at 12:35pm

2025-05-06

7th May 2025 at 3:43am

There have been several different posts complaining about the long wait times when importing huge volumes of tiddlers. I took a look at the syncer and syncadaptor this evening, and realized that most of this is caused by the designed of the syncer itself, not the syncadaptor. The syncer only saves tiddlers one at a time. There's no batching. I'm not even sure if there is an option for batching.

It's probably going to be a good idea to redesign the syncer from the ground up. The syncer hooks directly into the change events on the wiki. From there it calculates which tiddlers need to be synced. I think it would be good to get as close to the wiki as possible, so hooking directly into the change events is probably a good idea.

For the purposes of MWS, the syncer is actually pretty basic, and most of the features are probably not used, however, the syncer presents a standard API surface between the UI internals and the sync adapter. If I can modify it to handle more bulk updates, that would help.

I think the syncer also needs to be updated to be more aware of the recipe. Being able to understand the recipe is kind of important in understanding how to deal with deletions efficiently. It needs to be aware of read-only tiddlers.

Speaking of recipes, I keep bouncing this idea around in my mind of recipes which are multi-layered. Each level of the recipe could be opened by admins in order to easily make changes to the bag without creating a separate recipe. You can't just edit an individual bag because you need the bags below it in order to make sense out of it.

There also isn't a good way to move tiddlers between levels. Admins will need to be given some extra tools to do this with.

I'm not good at wikitext, but I can provide the endpoints for all of these operations as well as the client-side adapters and actions to use them.

I also don't know why we aren't using the wiki mount point for the API, and putting the react UI under admin, since then it would free up the entire path space to use for a tree path structure.

I'm working on simplifying the internal code structure yet again, still focusing on getting everything up to speed with my latest ideas.

-- Arlen22

Access Control

28th April 2025 at 5:46pm

Access Control is implemented separately for both recipes and bags, but bags can inherit the ACL of recipes they are added to.

Permission Inheritance

  • Users receive combined permissions from all assigned roles
  • When roles grant different permission levels for the same resource, the higher access level is granted. For example, if one role grants "read" and another grants "write" access to a recipe, the user receives "write" access since it includes all lower-level permissions. If a role grants "admin", it inherits both "read" and "write".

"Readonly" permission

If this were reversed, and users with explicit "read" access were forbidden from writing, it would significantly complicate roles.

Imagine a group of engineers working on several projects. Each project has people who are responsible for editing the documentation for everyone else. So everyone needs to be granted the read permission on all projects, but only a few people are granted the write permission on each project.

The easiest approach is to maintain one role which grants "read" access to all users, and then maintain additional roles to grant "write" access to the users responsible for each project.

If granting read access explicitly prevented a user from writing, we would need to create two roles for each project, one which which grants write access for the project, and another which grants read access to all other projects.

Every time a new project is added, all other projects would need to grant "read" access to that new "all projects but one" role on every single wiki for every single project in the entire organization.

Application Access & Security

28th April 2025 at 3:26pm

Users can be administered through two interfaces:

  1. Web-based Administrative Interface
    • Accessible to all users.
    • Provides graphical interface for user operations
    • Real-time validation and feedback
  2. Command-line Interface (CLI) Tools
    • Only available to server owner.
    • Suitable for automation and scripting
    • Enables batch operations
    • Useful for system initialization

Each user account contains the following essential components:

  • Username
    • Must be unique across the system
    • Case-sensitive
    • Used for authentication and audit trails
  • Password
    • Stored using secure hashing algorithms
    • Never stored in plaintext
    • Subject to complexity requirements
  • Email
    • Eventually used for the obvious password reset.
  • Role Assignments
    • Multiple roles can be assigned
    • Inherited permissions from all assigned roles
    • Dynamic permission calculation based on role combination

Access Levels

Conceptually, there are 6 access levels.

  • Site owner - has file system access to the server, and can run CLI commands. Presumably they also have a site admin account.
  • Site admin - Users with the admin role. They can assign owners and change permissions, and have full read and write access. Most ACL checks are skipped for the admin role.
  • Entity owner - Users who "own" an entity (bag or recipe). They can change permissions for that entity, and have full read and write access. Entity admins cannot change the owner.
  • Entity admin - Users granted the admin permission on an entity. They cannot change the owner, but they can change the permissions of other users for that entity.
  • Entity write - Users granted the write permission on an enitity. They can list, read, create, update, and delete tiddlers, but cannot change permissions.
  • Entity read - Users granted the read permission on an entity. They can list and read tiddlers, but cannot do anything else.

Entities

  • Bag - A collection of tiddlers with unique titles
  • Recipe - A stack of bags in a specific order. Bags may inherit the ACL of a recipe they are included in.

Roles

  • Roles have names and descriptions
  • Multiple users can be assigned to a role

Permissions

  • READ - Read tiddlers from an entity.
  • WRITE - Write tiddlers to an entity.
  • ADMIN - Update ACL for an entity.

Admin

  • There is an admin role and an admin permission.
  • Roles are given specific permissions for specific entities (bags and recipes).
  • The admin role has automatic admin permission for everything.

ACL

  • Grants the Permission for an Entity to a Role.
  • Entities without an ACL are either public or private (configurable).

Ownership

  • Ownership of an entity grants the "admin" permission.
  • Only site admins can change the owner of an entity.
  • Users with "admin" permission cannot change ownership.

User Types

Administrator (ADMIN) Role

  • Full system access and configuration rights
  • Most access checks are skipped
  • Can create, modify, and delete user accounts
  • Manages role assignments and permissions
  • Has full permissions on all recipes and bags

Regular Users

  • Created by administrators
  • Permissions determined by assigned roles
  • Access limited to specific recipes based on role permissions
  • Access to recipes without an ACL is based on recipe config

Guest Users

  • Users not logged in.
  • No inherent permissions
  • Can access recipes which allow it
  • Read/write can be enabled by server config
  • Useful for public wikis or documentation sites

Architecture

Bags and Recipes

9th March 2024 at 2:21pm

The bags and recipes model is a reference architecture for how tiddlers can be shared between multiple wikis. It was first introduced by TiddlyWeb in 2008.

The principles of bags and recipes can be simply stated:

  1. Tiddlers are stored in named "bags"
  2. Bags have access controls that determines which users can read or write to them
  3. Recipes are named lists of bags, ordered from lowest priority to highest
  4. The tiddlers within a recipe are accumulated in turn from each bag in the recipe in order of increasing priority. Thus, if there are multiple tiddlers with the same title in different bags then the one from the highest priority bag will be used as the recipe tiddler
  5. Wikis are composed by splicing the tiddlers from the corresponding recipe into the standard TW5 HTML template

A very simple example of the recipe/bag model might be for a single user who maintains the following bags:

  • recipes - tiddlers related to cooking recipes
  • work - tiddlers related to work
  • app - common tiddlers for customising TiddlyWiki

Those bags would be used with the following recipes:

  • recipes –> recipes, app - wiki for working with recipes, with common custom components
  • work –> work, app - wiki for working with work, with common custom components
  • app –> app - wiki for maintaining custom components

All of this will work dynamically, so changes to the app bag will instantly ripple into the affected hosted wikis.

A more complex example might be for a teacher working with a group of students:

  • student-{name} bag for each students work
  • teacher-course bag for the coursework, editable by the teacher
  • teacher-tools bag for custom tools used by the teacher

Those bags would be exposed through the following hosted wikis:

  • student-{name} hosted wiki for each students work, including the coursework material
  • teacher-course hosted wiki for the coursework, editable by the teacher
  • teacher hosted wiki for the teacher, bringing together all the bags, giving them an overview of all the students work

Checklists and Features

5th May 2025 at 2:54pm

ACL

  • Verify that anonymous users only have the access defined by allowAnon.
    • No access to owned bags.
    • No access to bags with ACL defined.
  • Verify that logged in users only have the access expected
    • No access to bags owned by other users.
    • Unless they are in the ACL for the bag.
    • No more access than what is granted by the ACL.
  • Verify that all admin permissions are based on the admin role, not the first user.
  • Verify that admin's cannot remove the admin role from themselves.

Recipes

HelloThere

17th April 2025 at 5:04pm

TiddlyWiki is Growing Up

MultiWikiServer is a new development that drastically improves TiddlyWiki's capabilities when running as a server under Node.js. It brings TiddlyWiki up to par with common web-based tools like WordPress or MediaWiki by supporting multiple wikis and multiple users at the same time.

Planned features include:

  • Hosting multiple wikis at once, using the Bags and Recipes mechanism for sharing data between them
  • Full support for SQLite, as well as MariaDB/MySQL, Postgres, and Microsoft SQL Server
  • Robust built-in synchronisation handlers for syncing data to the filesystem
  • Flexible authentication and authorisation options
  • Improved handling of file uploads and attachments, allowing gigabyte video files to be uploaded and streamed
  • Instantaneous synchronisation of changes between the server and all connected clients
  • Workflow processing on the server, for example to automatically compress images, or to archive webpages

MWS is currently under development at GitHub but it is already functional and usable.

HTTP API

28th April 2025 at 3:26pm

Installation

28th April 2025 at 2:41pm

These instructions require minimal knowledge of the terminal and require NodeJS to be installed.

  1. Open a terminal window and set the current directory to the folder you want to create the project folder in.
  2. The init command creates the project folder and installs the required dependencies and config files. You can change the name to whatever you like.
    npm init @tiddlywiki/mws@latest "new_folder_name" 
  3. Set the current directory to the project folder that was just created:
    cd "new_folder_name" 
  4. Start MWS:
    npm start
  5. Visit http://localhost:8080 in a browser on the same computer.
  6. When you have finished using MWS, stop the server with ctrl-C

See Troubleshooting if you encounter any errors.

Updating MWS

To update your copy of MWS in the future with newer changes will require re-downloading the code, taking care not to lose any changes you might have made.

  1. Make a backup: Copy or zip your project folder to a safe backup folder.
    tar -cf archive.tar my_folder
  2. Get the latest version. Notice that the second word is install instead of init. This pulls the latest version from NPM and installs it.
    npm install @tiddlywiki/mws@latest --save-exact
  3. Start the server. On startup, MWS checks the database schema and updates it automatically if there are changes. Normally this works just fine, but it can fail, which is why it's important to save a backup first.
    npm start

Git repo

It is optionally recommended to save a history of your project configuration using git.

  • On Windows you can use GitHub Desktop.
  • On Linux, git is usually preinstalled or available via the default package manager for your distro.

If you decide to do this, you should move the wiki folder out of the project folder, and change the wikiPath setting accordingly.

Moduler Design

28th April 2025 at 2:33pm

Sessions and Login

  • GET /login and POST /logout
  • parse incoming requests and provide AuthUser

Users and Roles

  • the user management UI
  • provide lists for other features
    • the list of roles [name, description]
    • the list of users [id, username]

Recipes and Bags (+ACL)

  • manage recipes and bags
  • manage the acl for them
  • manage available plugins
  • provide acl assertion checks for services to use

Router and Transport

  • Maps route definitions into service handlers
  • HTTP, Websockets

Database Connector

  • Prisma handles query translation (wasm or native addon)
  • Prisma supports custom adapters, and also provides standard ones
  • Connector classes in MWS take care of database setup and schema updates

MWS and SQLite

17th April 2025 at 7:41pm

Introduction

SQLite is a very popular open source, embedded SQL database with some unusual characteristics. It has proved itself to be robust, fast and scalable, and has been widely adopted in a range of applications including web browsers, mobile devices, and embedded systems.

The "embedded" part means that developers access SQLite as a library of C functions that run as part of a larger application. This contrasts with more familiar database applications like Microsoft's SQL Server or Oracle that are accessed as network services.

MWS uses SQLite for the tiddler store and associated data. It brings many advantages:

Misconceptions

TiddlyWiki 5 has always incorporated a database. Until MWS, that database has always been a custom tiddler database written in JavaScript. Over the years it has been enhanced and optimised with indexes and other database features that have given us reasonably decent performance for a range of common operations.

One particular misconception to avoid is the idea that SQLite replaces the folders of .tid files that characterise the Node.js configuration of TiddlyWiki. Those files are generated by a separate sync operation. They are not the actual database itself. In the context of MWS, SQLite is a fast and efficient way to store tiddlers between requests. Regardless of how tiddlers are stored internally, MWS can still save .tid files to the file system, just as TW5 does today.

Database Engines

SQLite is perfect for MWS because it doesn't require any extra setup. But MWS is not restricted to SQLite. It uses Prisma for the database access layer, which supports several other database engines, including MariaDB (the MySQL fork) and Postgres.

Better-SQLite3

Currently WAL mode is not enabled. It has plenty of advantages, and a few minor disadvantages, but mostly it just takes extra thought to use correctly. It has more advantages for high-traffic servers that need serious concurrency. Better-SQLite3 defaults to synchronous=NORMAL for WAL mode. Eventually we will probably add a setting to enable it.

Better-SQLite3 supports multi-threading via Node workers. Either way we have to implement proper support for transactions, which mostly just means reserving a worker for the duration of the transaction.

Better-SQLite3 has foriegn keys enabled by default.

Better-SQLite3 uses native addons. If your platform isn't supported, or you need a wasm-only solution, feel free to open an issue on Github sharing your use-case.

MWS uses Prisma to communicate with SQLite, and in theory, MWS should work with anything Prisma supports.

MWS Banner.png

Notes

7th May 2025 at 12:28pm

Reference

28th April 2025 at 3:26pm

Request Server

22nd April 2025 at 11:30pm

This is more or less a reference on the design philosophy.

HTTP request path

The server comprises several classes which handle various parts of the request path.

The Listener classes call the Router class with incoming requests. The Router class starts an async (Promise-based) code path with a catch handler protecting it.

There are two kinds of classes, routing and state.

Routing classes hold config state for that part of the request path. They are used to reach a specific objective. Examples include the Router and Authentication classes.

State classes hold information specific to each request. It may be used by any of the routing classes. The primary goal is usually to hide implementation details, or to pass information between routing classes. The StateObject class is the primary example, but modules may have their own state classes.

Both kinds of classes should check incoming information thoroughly and throw quickly if anything is wrong. The constructors of state classes are part of the request chain and may throw as well.

The request may be completed at any point in the request chain by throwing the STREAM_ENDED symbol. The catch all handler will ignore the symbol, but will send an error 500 if headers are not marked as sent at that point.

It is important that the entire request be awaited and eventually resolve or reject. A promise should never be left hanging. The Router class takes care of making sure every request has finished with some response, but if the promise never resolves or rejects, there is no way to do this.

TableOfContents

Troublesheeting gyp/prebuild Installation Errors

17th April 2025 at 7:26pm

Installation may fail with errors related to gyp or prebuild. These errors are caused by missing dependencies or incorrect versions of dependencies.

Note that in most cases, these errors occur because of the use of the npm module better-sqlite3. This module is mostly written in C, and thus requires compilation for the target platform. MWS supports switchable database engines, and also supports the use of the node-sqlite3-wasm module which is written in JavaScript and does not require compilation and so may avoid these errors. See Database Engines for more details of how to switch between engines.

The following steps may help resolve errors involving gyp or prebuild:

  • Ensure that you have the latest version of Node.js installed. You can download the latest version from the Node.js website.
  • Update npm to the latest version by running the following command in your terminal:
    npm install -g npm@latest
  • Clear the npm cache by running the following command in your terminal:
    npm cache clean --force
  • Delete the node_modules folder in your project by running the following command in your terminal:
    rm -rf node_modules
  • Reinstall the dependencies by running the following command in your terminal:
    npm install
  • If you continue to encounter errors, try running the following command in your terminal:
    npm rebuild
  • If you are still experiencing issues, you may need to manually install the gyp and prebuild dependencies. You can do this by running the following commands in your terminal:
    npm install -g node-gyp
    npm install -g prebuild
  • Once you have installed the dependencies, try reinstalling the TiddlyWiki dependencies by running the following command in your terminal:
    npm install

Troubleshooting

Usage

28th April 2025 at 2:43pm

Once MWS is successfully installed, you can access it by visiting http://localhost:8080 in a browser on the same computer.

On first start, a default user is created with username admin and password 1234. However, also by default, the server is only accessible to browsers on the same machine.

If you intend to make an MWS installation available on the Internet, the server should first be secured with the following steps:

  • Change the administrator password
  • Install HTTPS server certificates.