This page is part of a static HTML representation of the TiddlyWiki at https://tiddlywiki.com/

Architecture

Bags and Recipes

9th March 2024 at 2:21pm

The bags and recipes model is a reference architecture for how tiddlers can be shared between multiple wikis. It was first introduced by TiddlyWeb in 2008.

The principles of bags and recipes can be simply stated:

  1. Tiddlers are stored in named "bags"
  2. Bags have access controls that determines which users can read or write to them
  3. Recipes are named lists of bags, ordered from lowest priority to highest
  4. The tiddlers within a recipe are accumulated in turn from each bag in the recipe in order of increasing priority. Thus, if there are multiple tiddlers with the same title in different bags then the one from the highest priority bag will be used as the recipe tiddler
  5. Wikis are composed by splicing the tiddlers from the corresponding recipe into the standard TW5 HTML template

A very simple example of the recipe/bag model might be for a single user who maintains the following bags:

  • recipes - tiddlers related to cooking recipes
  • work - tiddlers related to work
  • app - common tiddlers for customising TiddlyWiki

Those bags would be used with the following recipes:

  • recipes –> recipes, app - wiki for working with recipes, with common custom components
  • work –> work, app - wiki for working with work, with common custom components
  • app –> app - wiki for maintaining custom components

All of this will work dynamically, so changes to the app bag will instantly ripple into the affected hosted wikis.

A more complex example might be for a teacher working with a group of students:

  • student-{name} bag for each students work
  • teacher-course bag for the coursework, editable by the teacher
  • teacher-tools bag for custom tools used by the teacher

Those bags would be exposed through the following hosted wikis:

  • student-{name} hosted wiki for each students work, including the coursework material
  • teacher-course hosted wiki for the coursework, editable by the teacher
  • teacher hosted wiki for the teacher, bringing together all the bags, giving them an overview of all the students work

HelloThere

17th April 2025 at 5:04pm

TiddlyWiki is Growing Up

MultiWikiServer is a new development that drastically improves TiddlyWiki's capabilities when running as a server under Node.js. It brings TiddlyWiki up to par with common web-based tools like WordPress or MediaWiki by supporting multiple wikis and multiple users at the same time.

Planned features include:

  • Hosting multiple wikis at once, using the Bags and Recipes mechanism for sharing data between them
  • Full support for SQLite, as well as MariaDB/MySQL, Postgres, and Microsoft SQL Server
  • Robust built-in synchronisation handlers for syncing data to the filesystem
  • Flexible authentication and authorisation options
  • Improved handling of file uploads and attachments, allowing gigabyte video files to be uploaded and streamed
  • Instantaneous synchronisation of changes between the server and all connected clients
  • Workflow processing on the server, for example to automatically compress images, or to archive webpages

MWS is currently under development at GitHub but it is already functional and usable.

Installation

17th April 2025 at 4:58pm

These instructions require minimal knowledge of the terminal and require NodeJS to be installed.

  1. Open a terminal window and set the current directory to the folder you want to create the project folder in.
  2. The init command creates the project folder and installs the required dependencies and config files. You can change the name to whatever you like.
    npm init @tiddlywiki/mws@latest "new_folder_name" 
  3. Set the current directory to the project folder that was just created:
    cd "new_folder_name" 
  4. Start MWS:
    npm start
  5. Visit http://localhost:8080 in a browser on the same computer.
  6. When you have finished using MWS, stop the server with ctrl-C

See Troubleshooting if you encounter any errors.

Updating MWS

To update your copy of MWS in the future with newer changes will require re-downloading the code, taking care not to lose any changes you might have made.

  1. Make a backup: Copy or zip your project folder to a safe backup folder.
    tar -cf archive.tar my_folder
  2. Get the latest version. Notice that the second word is install instead of init. This pulls the latest version from NPM and installs it.
    npm install @tiddlywiki/mws@latest
  3. Start the server. On startup, MWS checks the database schema and updates it automatically if there are changes. Normally this works just fine, but it can fail, which is why it's important to save a backup first.
    npm start

Moduler Design

18th April 2025 at 6:30pm

Sessions and Login

  • GET /login and POST /logout
  • parse incoming requests and provide AuthUser

Users and Roles

  • the user management UI
  • provide lists for other features
    • the list of roles [name, description]
    • the list of users [id, username]

Recipes and Bags (+ACL)

  • manage recipes and bags
  • manage the acl for them
  • manage available plugins
  • provide acl assertion checks for services to use

Router and Transport

  • Maps route definitions into service handlers
  • HTTP, Websockets

MWS and SQLite

17th April 2025 at 7:41pm

Introduction

SQLite is a very popular open source, embedded SQL database with some unusual characteristics. It has proved itself to be robust, fast and scalable, and has been widely adopted in a range of applications including web browsers, mobile devices, and embedded systems.

The "embedded" part means that developers access SQLite as a library of C functions that run as part of a larger application. This contrasts with more familiar database applications like Microsoft's SQL Server or Oracle that are accessed as network services.

MWS uses SQLite for the tiddler store and associated data. It brings many advantages:

Misconceptions

TiddlyWiki 5 has always incorporated a database. Until MWS, that database has always been a custom tiddler database written in JavaScript. Over the years it has been enhanced and optimised with indexes and other database features that have given us reasonably decent performance for a range of common operations.

One particular misconception to avoid is the idea that SQLite replaces the folders of .tid files that characterise the Node.js configuration of TiddlyWiki. Those files are generated by a separate sync operation. They are not the actual database itself. In the context of MWS, SQLite is a fast and efficient way to store tiddlers between requests. Regardless of how tiddlers are stored internally, MWS can still save .tid files to the file system, just as TW5 does today.

Database Engines

SQLite is perfect for MWS because it doesn't require any extra setup. But MWS is not restricted to SQLite. It uses Prisma for the database access layer, which supports several other database engines, including MariaDB (the MySQL fork) and Postgres.

MWS Banner.png

Reference

Authentication & Authorization

Overview

Our application has transitioned from a conventional username/password authentication system to a more robust Authentication and Authorization implementation. This new system supports multiple user accounts, roles, permissions, and a comprehensive access control list.

Key Features

  1. Multiple User Accounts
  2. Role-based Access Control
  3. Granular Permissions
  4. Access Control List (ACL)

Application Access & Security

Access Levels

Conceptually, there are 6 access levels.

  • Site owner - has file system access to the server, and can run CLI commands. Presumably they also have a site admin account.
  • Site admin - Users with the admin role. They can assign owners and change permissions, and have full read and write access.
  • Entity owner - Users who "own" an entity (bag or recipe). They can change permissions for that entity, and have full read and write access.
  • Entity admin - Users granted the admin permission on an entity. They cannot change the owner, but they can change the permissions of other users for that entity.
  • Entity write - Users granted the write permission on an enitity. They can list, read, create, update, and delete tiddlers, but cannot change permissions.
  • Entity read - Users granted the read permission on an entity. They can list and read tiddlers, but cannot do anything else.

Initial Setup

When you first launch the Multiwiki Server, it creates a default admin user. The username is admin and the password is 1234. To secure your installation you should change this immediately.

Entities

  • Bag - A collection of tiddlers with unique titles
  • Recipe - A stack of bags in a specific order. Bags may inherit the ACL of a recipe they are included in.

Roles

  • Roles have names and descriptions
  • Multiple users can be assigned to a role

Permissions

  • READ - Read tiddlers from an entity.
  • WRITE - Write tiddlers to an entity.
  • ADMIN - Update ACL for an entity.

ACL

  • Grants the Permission for an Entity to a Role.
  • Entities without an ACL are either public or private (configurable).

Ownership

  • Ownership of an entity grants the "admin" permission.
  • Only site admins can change the owner of an entity.
  • Users with "admin" permission cannot change ownership.

User Types

Administrator (ADMIN) Role

  • Full system access and configuration rights
  • Most access checks are skipped
  • Can create, modify, and delete user accounts
  • Manages role assignments and permissions
  • Has full permissions on all recipes and bags

Regular Users

  • Created by administrators
  • Permissions determined by assigned roles
  • Access limited to specific recipes based on role permissions
  • Access to recipes without an ACL is based on recipe config

Guest Users

  • Users not logged in.
  • No inherent permissions
  • Can access recipes which allow it
  • Read/write can be enabled by server config
  • Useful for public wikis or documentation sites

Access Control

Access Control is implemented separately for both recipes and bags, but bags can in inherit the ACL of recipes they are added to.

Permission Inheritance

  • Users receive combined permissions from all assigned roles
  • When roles grant different permission levels for the same resource, the higher access level is granted. For example, if one role grants "read" and another grants "write" access to a recipe, the user receives "write" access since it includes all lower-level permissions.

If this were reversed, and users with explicit "read" access were forbidden from writing, it would significantly complicate roles.

Imagine a group of engineers working on several projects. Each project has people who are responsible for editing the documentation for everyone else. So everyone needs to be granted the read permission on all projects, but only a few people are granted the write permission on each project.

The easiest approach is to maintain one role which grants "read" access to all users, and then maintain additional roles to grant "write" access to the users responsible for each project.

If granting read access explicitly prevented a user from writing, we would need to create two roles for each project, one which which grants write access for the project, and another which grants read access to all other projects.

Every time a new project is added, all other projects would need to grant "read" access to that new "all projects but one" role on every single wiki for every single project in the entire organization.

User Management & Security Architecture

User Account Management

Users can be administered through two interfaces:

  1. Web-based Administrative Interface
    • Accessible to all users.
    • Provides graphical interface for user operations
    • Real-time validation and feedback
  2. Command-line Interface (CLI) Tools
    • Only available to server owner.
    • Suitable for automation and scripting
    • Enables batch operations
    • Useful for system initialization

Each user account contains the following essential components:

  • Username
    • Must be unique across the system
    • Case-sensitive
    • Used for authentication and audit trails
  • Password
    • Stored using secure hashing algorithms
    • Never stored in plaintext
    • Subject to complexity requirements
  • Email
    • Eventually used for the obvious password reset.
  • Role Assignments
    • Multiple roles can be assigned
    • Inherited permissions from all assigned roles
    • Dynamic permission calculation based on role combination

HTTP API

The MultiWikiServer HTTP API provides management and tiddler endpoints. It was based on the API of TiddlyWeb, first developed in 2008 by Chris Dent, but is more oriented around JSON and Typescript.

The design goals of the API are:

  • Connect the client and server as transparently as possible.
  • Follow the principles of remote procedure calls.
  • Be easy to understand and use via Javascript.
  • Have strict validation of incoming requests.

General points about the design:

  • On the client, most request paths are defined by a single adaptor function which accepts the input for the server and returns a promise which resolves to the server's parsed response.
  • The server may throw a user error, which is expected to return to the client. It should give them an actionable understanding of what went wrong.

Request Server

22nd April 2025 at 11:30pm

This is more or less a reference on the design philosophy.

HTTP request path

The server comprises several classes which handle various parts of the request path.

The Listener classes call the Router class with incoming requests. The Router class starts an async (Promise-based) code path with a catch handler protecting it.

There are two kinds of classes, routing and state.

Routing classes hold config state for that part of the request path. They are used to reach a specific objective. Examples include the Router and Authentication classes.

State classes hold information specific to each request. It may be used by any of the routing classes. The primary goal is usually to hide implementation details, or to pass information between routing classes. The StateObject class is the primary example, but modules may have their own state classes.

Both kinds of classes should check incoming information thoroughly and throw quickly if anything is wrong. The constructors of state classes are part of the request chain and may throw as well.

The request may be completed at any point in the request chain by throwing the STREAM_ENDED symbol. The catch all handler will ignore the symbol, but will send an error 500 if headers are not marked as sent at that point.

It is important that the entire request be awaited and eventually resolve or reject. A promise should never be left hanging. The Router class takes care of making sure every request has finished with some response, but if the promise never resolves or rejects, there is no way to do this.

TableOfContents

Troublesheeting gyp/prebuild Installation Errors

17th April 2025 at 7:26pm

Installation may fail with errors related to gyp or prebuild. These errors are caused by missing dependencies or incorrect versions of dependencies.

Note that in most cases, these errors occur because of the use of the npm module better-sqlite3. This module is mostly written in C, and thus requires compilation for the target platform. MWS supports switchable database engines, and also supports the use of the node-sqlite3-wasm module which is written in JavaScript and does not require compilation and so may avoid these errors. See Database Engines for more details of how to switch between engines.

The following steps may help resolve errors involving gyp or prebuild:

  • Ensure that you have the latest version of Node.js installed. You can download the latest version from the Node.js website.
  • Update npm to the latest version by running the following command in your terminal:
    npm install -g npm@latest
  • Clear the npm cache by running the following command in your terminal:
    npm cache clean --force
  • Delete the node_modules folder in your project by running the following command in your terminal:
    rm -rf node_modules
  • Reinstall the dependencies by running the following command in your terminal:
    npm install
  • If you continue to encounter errors, try running the following command in your terminal:
    npm rebuild
  • If you are still experiencing issues, you may need to manually install the gyp and prebuild dependencies. You can do this by running the following commands in your terminal:
    npm install -g node-gyp
    npm install -g prebuild
  • Once you have installed the dependencies, try reinstalling the TiddlyWiki dependencies by running the following command in your terminal:
    npm install

Troubleshooting

Usage

17th April 2025 at 5:05pm

Once MWS is successfully installed, you can access it by visiting http://localhost:8080 in a browser on the same computer.

On first start, a default user is created with username admin and password 1234. However, also by default, the server is only accessible to browsers on the same machine.

If you intend to put an MWS installation on a public network like the Internet, the server will need to be secured with the following steps:

  • Change the administrator password.
  • Install HTTPS server certificates.