This page is part of a static HTML representation of the TiddlyWiki at https://tiddlywiki.com/

Architecture

Bags and Recipes

9th March 2024 at 2:21pm

The bags and recipes model is a reference architecture for how tiddlers can be shared between multiple wikis. It was first introduced by TiddlyWeb in 2008.

The principles of bags and recipes can be simply stated:

  1. Tiddlers are stored in named "bags"
  2. Bags have access controls that determines which users can read or write to them
  3. Recipes are named lists of bags, ordered from lowest priority to highest
  4. The tiddlers within a recipe are accumulated in turn from each bag in the recipe in order of increasing priority. Thus, if there are multiple tiddlers with the same title in different bags then the one from the highest priority bag will be used as the recipe tiddler
  5. Wikis are composed by splicing the tiddlers from the corresponding recipe into the standard TW5 HTML template

A very simple example of the recipe/bag model might be for a single user who maintains the following bags:

  • recipes - tiddlers related to cooking recipes
  • work - tiddlers related to work
  • app - common tiddlers for customising TiddlyWiki

Those bags would be used with the following recipes:

  • recipes –> recipes, app - wiki for working with recipes, with common custom components
  • work –> work, app - wiki for working with work, with common custom components
  • app –> app - wiki for maintaining custom components

All of this will work dynamically, so changes to the app bag will instantly ripple into the affected hosted wikis.

A more complex example might be for a teacher working with a group of students:

  • student-{name} bag for each students work
  • teacher-course bag for the coursework, editable by the teacher
  • teacher-tools bag for custom tools used by the teacher

Those bags would be exposed through the following hosted wikis:

  • student-{name} hosted wiki for each students work, including the coursework material
  • teacher-course hosted wiki for the coursework, editable by the teacher
  • teacher hosted wiki for the teacher, bringing together all the bags, giving them an overview of all the students work

Database Engines

MWS uses SQLite for data storage. It supports choosing between two "database engines" that are based on two different npm modules:

  • better-sqlite3 is written partially in C/C++ and so requires compilation for the target platform
  • node-sqlite3-wasm is written entirely in JavaScript and does not require compilation, but does require a WebAssembly-capable Node.js host. This is not currently possible on some platforms such as iOS/iPadOS

By default npm install will install both database engines. By default it will use better-sqlite3. To switch to using node-sqlite3-wasm, set the system configuration tiddler $:/config/MultiWikiServer/Engine to wasm (the default value is better). Note that this tiddler resides in the Administration Wiki.

Avoiding Installation Errors

If you encounter errors during npm install related to gyp or prebuild, you may be able to avoid them by using node-sqlite3-wasm instead of better-sqlite3. However, it will be necessary to manually install node-sqlite3-wasm and its dependencies. This can be done by running the following commands in your terminal:

npm install node-sqlite3-wasm

HelloThere

TiddlyWiki is Growing Up

MultiWikiServer is a new development that drastically improves TiddlyWiki's capabilities when running as a server under Node.js. It brings TiddlyWiki up to par with common web-based tools like WordPress or MediaWiki by supporting multiple wikis and multiple users at the same time.

Features under development include:

  • Hosting multiple wikis at once, using the Bags and Recipes mechanism for sharing data between them
  • Based on SQLite for performance and reliability
  • Robust built-in synchronisation handlers for syncing data to the filesystem
  • Flexible authentication and authorisation options
  • Improved handling of file uploads and attachments, allowing gigabyte video files to be uploaded and streamed
  • Instantaneous synchronisation of changes between the server and all connected clients
  • Workflow processing on the server, for example to automatically compress images, or to archive webpages

MWS is currently under development at GitHub but it is already functional and usable.

Installation

5th November 2024 at 1:37pm

These instructions require minimal knowledge of the terminal. There are also alternative instructions for those using Git.

  1. Download the code direct from GitHub
  2. Open a terminal window and set the current directory to the root of the downloaded folder
  3. Install the dependencies with the command:
    npm install
  4. To verify that MWS is working correctly, start the server with the command:
    npm start
    and then visit http://localhost:8080 in a browser on the same computer
  5. When you have finished using MWS, stop the server with ctrl-C

See Troubleshooting if you encounter any errors.

To update your copy of MWS in the future with newer changes will require re-downloading the code, taking care not to lose any changes you might have made.

Installation using Git

5th November 2024 at 1:37pm

These instructions require basic knowledge both of the terminal and of Git. There are also alternative instructions without using Git.

  1. Clone the code from GitHub with:
    git clone -b multi-wiki-support --single-branch https://github.com/TiddlyWiki/TiddlyWiki5
  2. Open a terminal window and set the current directory to the root of the downloaded folder
  3. Install the dependencies with the command:
    npm install
  4. Start the server with the command:
    npm start
  5. To use MWS, visit http://localhost:8080 in a browser on the same computer
  6. When you have finished using MWS, stop the server with ctrl-C

See Troubleshooting if you encounter any errors.

To update your copy of MWS with newer changes from GitHub, run the following command:

git pull

MWS and SQLite

Introduction

SQLite is a very popular open source embedded SQL database with some unusual characteristics. It has proved itself to be robust, fast and scalable, and has been widely adopted in a range of applications including web browsers, mobile devices, and embedded systems.

The "embedded" part means that developers access SQLite as a library of C functions that run as part of a larger application. This contrasts with more familiar database applications like Microsoft's SQL Server or Oracle that are accessed as network services.

MWS uses SQLite for the tiddler store and associated data. It brings many advantages:

Misconceptions

TiddlyWiki 5 has always incorporated a database. Until MWS, that database has always been a custom tiddler database written in JavaScript. Over the years it has been enhanced and optimised with indexes and other database features that have given us reasonably decent performance for a range of common operations.

In terms of the traditional architecture of TiddlyWiki, MWS uses SQLite as the basis for an internal API that is equivalent to that of the $tw.Wiki object: basic CRUD operations on a database of tiddlers stored by their titles.

In the context of MWS, SQLite is just a fast and efficient equivalent of TiddlyWiki's existing JavaScript database engine. It gives us the option of persisting the database in file storage, but we also retain the option to run the database entirely within memory and rely on a file synchronisation process to save changes as individual .tid files in the file system, just as we do today.

One particular misconception to avoid is the idea that SQLite replaces the folders of .tid files that characterise the Node.js configuration of TiddlyWiki. These are separate components with a different purpose. The tiddler files are the result of syncing a database to the filesystem, and that database can be conceptually interchanged between our custom JavaScript database or the new SQLite implementation in MWS.

MWS Banner.png

Reference

Authentication & Authorization

Overview

Our application has transitioned from a conventional username/password authentication system to a more robust Authentication and Authorization implementation. This new system supports multiple user accounts, roles, permissions, and a comprehensive access control list.

Key Features

  1. Multiple User Accounts
  2. Role-based Access Control
  3. Granular Permissions
  4. Access Control List (ACL)

Application Access & Security

Initial Setup

When you first launch the Multiwiki Server, it operates in an unauthenticated mode to facilitate initial configuration. During this initial state, the system creates a temporary anonymous administrator account. Upon accessing the landing page, you'll receive a prominent security warning with instructions to establish a permanent ADMIN account. It's crucial to create this account immediately to secure your installation.

User Types and Permissions

Administrator (ADMIN)

  • Full system access and configuration rights
  • Can create, modify, and delete user accounts
  • Manages role assignments and permissions
  • Controls global system settings
  • Can configure guest access policies
  • Has complete control over all recipes and tiddlers

Regular Users

  • Custom accounts created by administrators
  • Permissions determined by assigned roles
  • Access limited to specific recipes based on role permissions
  • Can have READ and/or WRITE permissions

Guest Users

  • Default anonymous access level
  • No inherent permissions
  • Can only access recipes without Access Control Lists (ACLs)
  • Read/write capabilities configurable by ADMIN
  • Useful for public wikis or documentation sites

Access Control System

Recipe-Level Security

  • Access control is implemented at the recipe level
  • Each recipe can have its own Access Control List (ACL)
  • Permissions are granular:
    • READ: Allows viewing recipe contents
    • WRITE: Allows modifications to recipe contents

Role-Based Access Control (RBAC)

  • Administrators can create custom roles
  • Roles can be assigned specific READ/WRITE permissions
  • Users inherit permissions from their assigned roles
  • Multiple roles can be assigned to a single user
  • Provides flexible and scalable access management

Permission Inheritance

  • Users receive combined permissions from all assigned roles
  • When roles grant different permission levels for the same resource, the higher access level is granted. For example, if one role grants "read" and another grants "write" access to a recipe, the user receives "write" access since it includes all lower-level permissions.
  • Guest access is overridden by recipe ACLs
  • When different permission rules conflict, the system follows a "most restrictive wins" principle: if any applicable rule denies access or requires a higher security level, that restriction takes precedence over more permissive rules. This ensures security is maintained even when users have multiple overlapping role assignments or inherited permissions.

This security model allows for fine-grained control over content access while maintaining flexibility for both private and public wiki deployments.

User Management & Security Architecture

User Account Management

Users can be administered through two interfaces:

  1. Web-based Administrative Interface
    • Accessible only to ADMIN users
    • Provides graphical interface for user operations
    • Real-time validation and feedback
  2. Command-line Interface (CLI) Tools
    • Suitable for automation and scripting
    • Enables batch operations
    • Useful for system initialization

Each user account contains the following essential components:

  • Username
    • Must be unique across the system
    • Case-sensitive
    • Used for authentication and audit trails
  • Password
    • Stored using secure hashing algorithms
    • Never stored in plaintext
    • Subject to complexity requirements
  • Role Assignments
    • Multiple roles can be assigned
    • Inherited permissions from all assigned roles
    • Dynamic permission calculation based on role combination

Role & Permission Framework

Role Management

Roles serve as permission containers and provide organized access control. The system includes:

Built-in Roles:

  • ADMIN
    • Highest privilege level
    • Full system access
    • Cannot be modified or deleted
    • Can create and manage other roles
    • Controls guest access policies
  • USER
    • Basic access rights
    • Typically limited to specific recipes
    • Custom Roles (Examples):**
  • MANAGER
    • Intermediate access level
    • Can manage subset of resources
    • Custom roles as needed for specific use cases

Permission Architecture

Core Permissions:

  • READ Permission
    • View recipe contents
    • Access tiddler data
    • View metadata
    • Export capabilities
  • WRITE Permission
    • Create new tiddlers
    • Modify existing content
    • Delete resources
    • Manage recipe contents
    • Guest Access:**
  • No default permissions
  • Access limited to non-ACL recipes
  • Configurable by ADMIN users
  • Suitable for public wikis

Access Control List (ACL) Implementation

The ACL system provides granular security control through:

Entity-Level Control

  • Recipe-based access control
  • Individual resource protection
  • Hierarchical permission inheritance

Authentication Process Flow

  • Initial Authentication
    • User submits credentials
    • System validates username existence
    • Password hash comparison
    • Session token generation
  • Session Management
    • Secure session storage
    • Token-based authentication
    • Automatic session expiration
    • Re-authentication requirements

Authorization Workflow

  • Request Processing
    • Capture user action request
    • Identify target resource
    • Extract required permissions
  • Permission Validation
    • Check user roles
    • Aggregate permissions
    • Verify ACL entries
    • Apply guest policies if applicable
  • Access Decision
    • Compare required vs. available permissions
    • Apply most restrictive policy
    • Return access decision

System Extension Guidelines

Adding New Roles

  1. Access administrative interface
  2. Define role identifier
  3. Assign base permissions
  4. Configure ACL mappings
  5. Test role functionality

Permission Expansion

  1. Define new permission type
  2. Update ACL structure
  3. Implement permission checks
  4. Update validation logic
  5. Document new permission

Security Considerations

  • Regular permission audits
  • Role assignment reviews
  • Guest access monitoring
  • Security log analysis
  • Access pattern monitoring

This comprehensive security model provides flexible, maintainable, and secure access control while supporting both authenticated and guest users within the Multiwiki Server environment.

HTTP API

The MultiWikiServer HTTP API provides access to resources hosted by the MWS store. It is based on the API of TiddlyWeb, first developed in 2008 by Chris Dent.

The design goals of the API are:

  • To follow the principles of REST where practical
  • To present resources as nouns, not verbs

General points about the design:

  • In MWS there are no resources that end with / (except for the root path which is /)

TableOfContents

Troublesheeting gyp/prebuild Installation Errors

5th November 2024 at 1:37pm

Installation may fail with errors related to gyp or prebuild. These errors are caused by missing dependencies or incorrect versions of dependencies.

Note that in most cases, these errors occur because of the use of the npm module better-sqlite3. This module is mostly written in C, and thus requires compilation for the target platform. MWS supports switchable database engines, and also supports the use of the node-sqlite3-wasm module which is written in JavaScript and does not require compilation and so may avoid these errors. See Database Engines for more details of how to switch between engines.

The following steps may help resolve errors involving gyp or prebuild:

  1. Ensure that you have the latest version of Node.js installed. You can download the latest version from the Node.js website.
  2. Update npm to the latest version by running the following command in your terminal:
    npm install -g npm@latest
  3. Clear the npm cache by running the following command in your terminal:
    npm cache clean --force
  4. Delete the node_modules folder in your TiddlyWiki directory by running the following command in your terminal:
    rm -rf node_modules
  5. Reinstall the dependencies by running the following command in your terminal:
    npm install
  6. If you continue to encounter errors, try running the following command in your terminal:
    npm rebuild
  7. If you are still experiencing issues, you may need to manually install the gyp and prebuild dependencies. You can do this by running the following commands in your terminal:
    npm install -g node-gyp
    npm install -g prebuild
  8. Once you have installed the dependencies, try reinstalling the TiddlyWiki dependencies by running the following command in your terminal:
    npm install

Troubleshooting

Usage

5th November 2024 at 1:37pm

Once MWS is successfully installed, you can access it by visiting http://localhost:8080 in a browser on the same computer.

By default, MWS is installed with full anonymous access enabled, meaning that anyone with access to the server has full access to read and modify anything. However, also by default, the server is only accessible to browsers on the same machine.

If you intend to put an MWS installation on a public network like the Internet, the server will need to be secured with the following steps:

  • Create and login with a new administrator account
  • Disable anonymouse access