wasm-dbms

What is wasm-dbms?

wasm-dbms is an embeddable relational database engine written in Rust, designed to run entirely inside WebAssembly runtimes. Unlike traditional databases that run as external services, wasm-dbms compiles into your WASM module and manages data directly in linear or stable memory — no network calls, no external dependencies.

You define your schema as Rust structs with derive macros, and wasm-dbms provides full CRUD operations, ACID transactions, foreign key integrity, validation, and sanitization — all running within the sandbox of your WASM module.

wasm-dbms supports any WASM runtime (Wasmtime, Wasmer, WasmEdge) and offers first-class integration with Internet Computer canisters through the ic-dbms adapter.

Documentation

Guides

Step-by-step guides for building databases with wasm-dbms:

• Getting Started - Set up your first wasm-dbms database
• CRUD Operations - Insert, select, update, and delete records
• Querying - Filters, ordering, pagination, and field selection
• Transactions - ACID transactions with commit/rollback
• Relationships - Foreign keys, delete behaviors, and eager loading
• Custom Data Types - Define your own data types (enums, structs)
• Schema Migrations - Evolve your schema across releases without losing data
• Wasmtime Example - Using wasm-dbms with the WIT Component Model and Wasmtime

Reference

API and type reference documentation:

• Data Types - All supported column types
• Schema Definition - Table attributes and generated types
• Migrations - Schema migration lifecycle, ops, and policy
• Validation - Built-in and custom validators
• Sanitization - Built-in and custom sanitizers
• JSON - JSON data type and filtering
• Errors - Error types and handling

IC Integration

For deploying wasm-dbms as an Internet Computer canister:

• IC Getting Started - Deploy a database canister on the IC
• Access Control - Managing the ACL
• Client API - Using the IC client library

WASI Integration

For deploying wasm-dbms on WASI runtimes (Wasmer, Wasmtime, WasmEdge):

• WASI Memory Provider - File-backed persistent storage for WASI

Technical Documentation

For advanced users and contributors:

• Architecture - Three-layer system overview
• Memory Management - Stable memory internals
• Join Engine - Cross-table join query internals

Quick Example

Define your schema:

#![allow(unused)]
fn main() {
use wasm_dbms_api::prelude::*;

#[derive(Debug, Table, Clone, PartialEq, Eq)]
#[table = "users"]
pub struct User {
    #[primary_key]
    pub id: Uint32,
    #[sanitizer(TrimSanitizer)]
    #[validate(MaxStrlenValidator(100))]
    pub name: Text,
    #[validate(EmailValidator)]
    pub email: Text,
}
}

Use the Database trait for CRUD operations:

#![allow(unused)]
fn main() {
use wasm_dbms_api::prelude::*;

// Insert
let user = UserInsertRequest {
    id: 1.into(),
    name: "Alice".into(),
    email: "alice@example.com".into(),
};
database.insert::<User>(user)?;

// Query
let query = Query::builder()
    .filter(Filter::eq("name", Value::Text("Alice".into())))
    .build();
let users = database.select::<User>(query)?;
}

Features

• Schema-driven: Define tables as Rust structs with derive macros
• Runtime-agnostic: Works on any WASM runtime, not tied to a specific platform
• CRUD operations: Full insert, select, update, delete support
• ACID transactions: Commit/rollback with isolation
• Foreign keys: Referential integrity with cascade/restrict behaviors
• Validation & Sanitization: Built-in validators and sanitizers
• JSON support: Store and query semi-structured data
• IC Integration: First-class support for Internet Computer canisters via ic-dbms

Get Started

• Prerequisites
• Project Setup
  • Workspace Structure
  • Cargo Configuration
• Define Your Schema
  • Create the Schema Crate
  • Define Tables
• Define a Database Schema
• Using the Database
  • Create a DbmsContext
  • Perform CRUD Operations
• Quick Example: Complete Workflow
• Testing with HeapMemoryProvider
• Next Steps

This guide walks you through setting up a database using wasm-dbms. By the end, you’ll have a working database with CRUD operations and transactions.

Prerequisites

Before starting, ensure you have:

• Rust 1.91.1 or later
• wasm32-unknown-unknown target: rustup target add wasm32-unknown-unknown

Project Setup

Workspace Structure

We recommend organizing your project as a Cargo workspace with a schema crate:

my-dbms-project/
├── Cargo.toml          # Workspace manifest
├── schema/             # Schema definitions (reusable types)
│   ├── Cargo.toml
│   └── src/
│       └── lib.rs
└── app/                # Your application using the database
    ├── Cargo.toml
    └── src/
        └── lib.rs

Workspace Cargo.toml:

[workspace]
members = ["schema", "app"]
resolver = "2"

Cargo Configuration

Create .cargo/config.toml to configure the getrandom crate for WebAssembly:

[target.wasm32-unknown-unknown]
rustflags = ['--cfg', 'getrandom_backend="custom"']

This is required because the uuid crate depends on getrandom.

Define Your Schema

Create the Schema Crate

Create schema/Cargo.toml:

[package]
name = "my-schema"
version = "0.1.0"
edition = "2024"

[dependencies]
wasm-dbms-api = "0.6"

Define Tables

In schema/src/lib.rs, define your database tables using the Table derive macro:

#![allow(unused)]
fn main() {
use wasm_dbms_api::prelude::*;

#[derive(Debug, Table, Clone, PartialEq, Eq)]
#[table = "users"]
pub struct User {
    #[primary_key]
    pub id: Uint32,
    #[sanitizer(TrimSanitizer)]
    #[validate(MaxStrlenValidator(100))]
    pub name: Text,
    #[validate(EmailValidator)]
    pub email: Text,
    pub created_at: DateTime,
}

#[derive(Debug, Table, Clone, PartialEq, Eq)]
#[table = "posts"]
pub struct Post {
    #[primary_key]
    pub id: Uint32,
    #[validate(MaxStrlenValidator(200))]
    pub title: Text,
    pub content: Text,
    pub published: Boolean,
    #[foreign_key(entity = "User", table = "users", column = "id")]
    pub author_id: Uint32,
}
}

Required derives: Table, Clone

The Table macro generates additional types for each table:

Define a Database Schema

Once you’ve defined your tables, create a schema struct with #[derive(DatabaseSchema)] to wire them together:

#![allow(unused)]
fn main() {
use wasm_dbms::prelude::DatabaseSchema;

#[derive(DatabaseSchema)]
#[tables(User = "users", Post = "posts")]
pub struct MySchema;
}

The DatabaseSchema derive macro auto-generates the DatabaseSchema<M> trait implementation and a register_tables method. This replaces what would otherwise be ~130+ lines of manual dispatch code.

Using the Database

Create a DbmsContext

The DbmsContext holds all database state. Create one using a MemoryProvider:

#![allow(unused)]
fn main() {
use wasm_dbms::prelude::*;
use wasm_dbms_api::prelude::*;

// For testing, use HeapMemoryProvider
let ctx = DbmsContext::new(HeapMemoryProvider::default());

// Register tables from the schema
MySchema::register_tables(&ctx).expect("failed to register tables");
}

Perform CRUD Operations

Create a WasmDbmsDatabase from the context to perform operations:

#![allow(unused)]
fn main() {
use wasm_dbms::prelude::*;
use my_schema::{User, UserInsertRequest};

// Create a one-shot (non-transactional) database
let database = WasmDbmsDatabase::oneshot(&ctx, MySchema);

// Insert a record
let user = UserInsertRequest {
    id: 1.into(),
    name: "Alice".into(),
    email: "alice@example.com".into(),
    created_at: DateTime::now(),
};
database.insert::<User>(user)?;

// Query records
let query = Query::builder().all().build();
let users = database.select::<User>(query)?;
}

Quick Example: Complete Workflow

Here’s a complete example showing insert, query, update, and delete operations:

#![allow(unused)]
fn main() {
use wasm_dbms_api::prelude::*;
use my_schema::{User, UserInsertRequest, UserUpdateRequest};

fn example(database: &impl Database) -> Result<(), DbmsError> {
    // 1. INSERT a new user
    let insert_req = UserInsertRequest {
        id: 1.into(),
        name: "Alice".into(),
        email: "alice@example.com".into(),
        created_at: DateTime::now(),
    };
    database.insert::<User>(insert_req)?;

    // 2. SELECT users
    let query = Query::builder()
        .filter(Filter::eq("name", Value::Text("Alice".into())))
        .build();
    let users = database.select::<User>(query)?;
    println!("Found {} user(s)", users.len());

    // 3. UPDATE the user
    let update_req = UserUpdateRequest::builder()
        .set_email("alice.new@example.com".into())
        .filter(Filter::eq("id", Value::Uint32(1.into())))
        .build();
    let updated = database.update::<User>(update_req)?;
    println!("Updated {} record(s)", updated);

    // 4. DELETE the user
    let deleted = database.delete::<User>(
        DeleteBehavior::Restrict,
        Some(Filter::eq("id", Value::Uint32(1.into()))),
    )?;
    println!("Deleted {} record(s)", deleted);

    Ok(())
}
}

Testing with HeapMemoryProvider

For unit tests, use HeapMemoryProvider which stores data in heap memory:

#![allow(unused)]
fn main() {
use wasm_dbms::prelude::*;
use wasm_dbms_api::prelude::*;
use my_schema::{User, UserInsertRequest, MySchema};

#[test]
fn test_insert_and_select() {
    let ctx = DbmsContext::new(HeapMemoryProvider::default());
    MySchema::register_tables(&ctx).expect("register failed");
    let database = WasmDbmsDatabase::oneshot(&ctx, MySchema);

    let insert_req = UserInsertRequest {
        id: 1.into(),
        name: "Test User".into(),
        email: "test@example.com".into(),
        created_at: DateTime::now(),
    };

    database.insert::<User>(insert_req).expect("insert failed");

    let query = Query::builder().all().build();
    let users = database.select::<User>(query).expect("select failed");

    assert_eq!(users.len(), 1);
    assert_eq!(users[0].name.as_str(), "Test User");
}
}

For deploying on the Internet Computer as a canister, see the IC Getting Started Guide.

Next Steps

Now that you have a working database, explore these topics:

• CRUD Operations - Detailed guide on all database operations
• Querying - Filters, ordering, pagination, and field selection
• Transactions - ACID transactions with commit/rollback
• Relationships - Foreign keys and eager loading
• Custom Data Types - Define your own data types (enums, structs)
• Schema Definition - Complete schema reference
• Data Types - All supported field types

CRUD Operations

• CRUD Operations
  • Overview
  • Insert
    • Basic Insert
    • Handling Primary Keys
    • Nullable Fields
    • Insert with Transaction
  • Select
    • Select All Records
    • Select with Filter
    • Select Specific Columns
    • Select with Eager Loading
  • Update
    • Basic Update
    • Partial Updates
    • Update with Filter
    • Update Return Value
  • Delete
    • Delete with Filter
    • Delete Behaviors
    • Delete All Records
  • Operations with Transactions
  • Error Handling

Overview

wasm-dbms provides four fundamental database operations through the Database trait:

All operations:

• Support optional transaction IDs
• Validate and sanitize data according to schema rules
• Enforce foreign key constraints

Insert

Basic Insert

To insert a record, create an InsertRequest and call the insert method:

#![allow(unused)]
fn main() {
use wasm_dbms_api::prelude::*;
use my_schema::{User, UserInsertRequest};

let user = UserInsertRequest {
    id: 1.into(),
    name: "Alice".into(),
    email: "alice@example.com".into(),
    created_at: DateTime::now(),
};

// Insert without transaction
database.insert::<User>(user)?;
}

Handling Primary Keys

Every table must have a primary key. Insert will fail if a record with the same primary key already exists:

#![allow(unused)]
fn main() {
// First insert succeeds
database.insert::<User>(user1)?;

// Second insert with same ID fails with PrimaryKeyConflict
let result = database.insert::<User>(user2_same_id);
assert!(matches!(result, Err(DbmsError::Query(QueryError::PrimaryKeyConflict))));
}

Nullable Fields

For fields wrapped in Nullable<T>, you can insert either a value or null:

#![allow(unused)]
fn main() {
#[derive(Debug, Table, Clone, PartialEq, Eq)]
#[table = "profiles"]
pub struct Profile {
    #[primary_key]
    pub id: Uint32,
    pub bio: Nullable<Text>,      // Optional field
    pub website: Nullable<Text>,  // Optional field
}

// Insert with value
let profile = ProfileInsertRequest {
    id: 1.into(),
    bio: Nullable::Value("Hello world".into()),
    website: Nullable::Null,  // No website
};

database.insert::<Profile>(profile)?;
}

Insert with Transaction

To insert within a transaction, use a transactional database instance:

#![allow(unused)]
fn main() {
// Begin transaction
let tx_id = ctx.begin_transaction();

// Create a transactional database
let database = WasmDbmsDatabase::from_transaction(&ctx, my_schema, tx_id);

// Insert within transaction
database.insert::<User>(user)?;

// Commit or rollback
database.commit()?;
}

Select

Select All Records

Use Query::builder().all() to select all records:

#![allow(unused)]
fn main() {
use wasm_dbms_api::prelude::*;

let query = Query::builder().all().build();
let users: Vec<UserRecord> = database.select::<User>(query)?;

for user in users {
    println!("User: {} ({})", user.name, user.email);
}
}

Select with Filter

Add filters to narrow down results:

#![allow(unused)]
fn main() {
// Select users with specific name
let query = Query::builder()
    .filter(Filter::eq("name", Value::Text("Alice".into())))
    .build();

let users = database.select::<User>(query)?;
}

See the Querying Guide for comprehensive filter documentation.

Select Specific Columns

Select only the columns you need:

#![allow(unused)]
fn main() {
let query = Query::builder()
    .columns(vec!["id".to_string(), "name".to_string()])
    .build();

let users = database.select::<User>(query)?;
// Only id and name are populated; other fields have default values
}

Select with Eager Loading

Load related records in a single query using with():

#![allow(unused)]
fn main() {
// Load posts with their authors
let query = Query::builder()
    .all()
    .with("users")  // Eager load the related users table
    .build();

let posts = database.select::<Post>(query)?;
}

See the Relationships Guide for more on eager loading.

Update

Basic Update

Create an UpdateRequest to modify records:

#![allow(unused)]
fn main() {
use my_schema::UserUpdateRequest;

let update = UserUpdateRequest::builder()
    .set_name("Alice Smith".into())
    .filter(Filter::eq("id", Value::Uint32(1.into())))
    .build();

let affected_rows = database.update::<User>(update)?;

println!("Updated {} row(s)", affected_rows);
}

Partial Updates

Only specify the fields you want to change. Unspecified fields remain unchanged:

#![allow(unused)]
fn main() {
// Only update the email, keep everything else
let update = UserUpdateRequest::builder()
    .set_email("new.email@example.com".into())
    .filter(Filter::eq("id", Value::Uint32(1.into())))
    .build();

database.update::<User>(update)?;
}

Update with Filter

The filter determines which records are updated:

#![allow(unused)]
fn main() {
// Update all users with a specific domain
let update = UserUpdateRequest::builder()
    .set_verified(true.into())
    .filter(Filter::like("email", "%@company.com"))
    .build();

let affected = database.update::<User>(update)?;
println!("Verified {} company users", affected);
}

Update Return Value

Update returns the number of affected rows:

#![allow(unused)]
fn main() {
let affected = database.update::<User>(update)?;

if affected == 0 {
    println!("No records matched the filter");
} else {
    println!("Updated {} record(s)", affected);
}
}

Delete

Delete with Filter

Delete records matching a filter:

#![allow(unused)]
fn main() {
use wasm_dbms_api::prelude::DeleteBehavior;

let filter = Filter::eq("id", Value::Uint32(1.into()));

let deleted = database.delete::<User>(
    DeleteBehavior::Restrict,
    Some(filter),
)?;

println!("Deleted {} record(s)", deleted);
}

Delete Behaviors

When deleting records that are referenced by foreign keys, you must specify a behavior:

Restrict Example:

#![allow(unused)]
fn main() {
// Will fail if any posts reference this user
let result = database.delete::<User>(
    DeleteBehavior::Restrict,
    Some(Filter::eq("id", Value::Uint32(1.into()))),
);

match result {
    Ok(count) => println!("Deleted {} user(s)", count),
    Err(DbmsError::Query(QueryError::ForeignKeyConstraintViolation)) => {
        println!("Cannot delete: user has posts");
    }
    Err(e) => return Err(e),
}
}

Cascade Example:

#![allow(unused)]
fn main() {
// Deletes the user AND all their posts
database.delete::<User>(
    DeleteBehavior::Cascade,
    Some(Filter::eq("id", Value::Uint32(1.into()))),
)?;
}

Delete All Records

Pass None as the filter to delete all records (use with caution):

#![allow(unused)]
fn main() {
// Delete ALL users (respecting foreign key behavior)
let deleted = database.delete::<User>(
    DeleteBehavior::Cascade,
    None,  // No filter = all records
)?;

println!("Deleted all {} users and their related records", deleted);
}

Operations with Transactions

All CRUD operations can be performed within a transaction. When using a transactional database instance, operations won’t be visible to other callers until committed:

#![allow(unused)]
fn main() {
// Begin transaction
let tx_id = ctx.begin_transaction();
let mut database = WasmDbmsDatabase::from_transaction(&ctx, my_schema, tx_id);

// Perform operations within transaction
database.insert::<User>(user1)?;
database.insert::<User>(user2)?;

// Update within same transaction
let update = UserUpdateRequest::builder()
    .set_verified(true.into())
    .filter(Filter::all())
    .build();
database.update::<User>(update)?;

// Commit all changes atomically
database.commit()?;
}

See the Transactions Guide for comprehensive transaction documentation.

Error Handling

CRUD operations can fail for various reasons. Here are common errors:

Example error handling:

#![allow(unused)]
fn main() {
use wasm_dbms_api::prelude::{DbmsError, QueryError};

let result = database.insert::<User>(user);

match result {
    Ok(()) => println!("Insert successful"),
    Err(DbmsError::Query(QueryError::PrimaryKeyConflict)) => {
        println!("User with this ID already exists");
    }
    Err(DbmsError::Query(QueryError::BrokenForeignKeyReference)) => {
        println!("Referenced record does not exist");
    }
    Err(DbmsError::Validation(msg)) => {
        println!("Validation failed: {}", msg);
    }
    Err(e) => {
        println!("Unexpected error: {:?}", e);
    }
}
}

See the Errors Reference for complete error documentation.

For IC canister client usage with the IcDbmsCanisterClient, see the IC CRUD Guide.

Querying

• Querying
  • Overview
  • Query Builder
    • Basic Queries
    • Query Structure
  • Filters
    • Comparison Filters
    • List Membership
    • Pattern Matching
    • Null Checks
    • Combining Filters
  • JSON Filters
  • Ordering
    • Single Column Ordering
    • Multiple Column Ordering
  • Pagination
    • Limit
    • Offset
    • Pagination Pattern
  • Field Selection
    • Select All Fields
    • Select Specific Fields
  • Eager Loading
  • Distinct
    • Basic Distinct
    • Distinct by Multiple Columns
    • Distinct with Ordering and Pagination
    • Distinct Semantics
  • Aggregations
    • Defining Aggregates
    • Group By and Having
    • Aggregate Result Types
  • Joins
    • Join Types
    • Basic Join
    • Left, Right, and Full Joins
    • Chaining Multiple Joins
    • Qualified Column Names
    • Joins vs Eager Loading
  • Index-Accelerated Queries
    • How Indexes Improve Queries
    • Which Filters Use Indexes
    • Residual Filters
    • Transaction-Aware Lookups

Overview

wasm-dbms provides a powerful query API for retrieving data from your tables. Queries are built using the QueryBuilder and can include:

• Filters - Narrow down which records to return
• Ordering - Sort results by one or more columns
• Pagination - Limit results and implement pagination
• Field Selection - Choose which columns to return
• Eager Loading - Load related records in a single query
• Joins - Combine rows from multiple tables

Query Builder

Basic Queries

Use Query::builder() to construct queries:

#![allow(unused)]
fn main() {
use wasm_dbms_api::prelude::*;

// Select all records
let query = Query::builder().all().build();

// Select with filter
let query = Query::builder()
.filter(Filter::eq("status", Value::Text("active".into())))
.build();

// Complex query with multiple options
let query = Query::builder()
.filter(Filter::gt("age", Value::Int32(18.into())))
.order_by("created_at", OrderDirection::Descending)
.limit(10)
.offset(20)
.build();
}

Query Structure

A query consists of these optional components:

Filters

Filters determine which records match your query. All filters are created using the Filter struct.

Comparison Filters

Examples:

#![allow(unused)]
fn main() {
// Find users older than 21
let filter = Filter::gt("age", Value::Int32(21.into()));

// Find products under $50
let filter = Filter::lt("price", Value::Decimal(50.0.into()));

// Find orders from a specific date
let filter = Filter::ge("created_at", Value::DateTime(some_datetime));
}

List Membership

Check if a value is in a list of values:

#![allow(unused)]
fn main() {
// Find users with specific roles
let filter = Filter::in_list("role", vec![
    Value::Text("admin".into()),
    Value::Text("moderator".into()),
    Value::Text("editor".into()),
]);

// Find products in certain categories
let filter = Filter::in_list("category_id", vec![
    Value::Uint32(1.into()),
    Value::Uint32(2.into()),
    Value::Uint32(5.into()),
]);
}

Pattern Matching

Use like for pattern matching with wildcards:

#![allow(unused)]
fn main() {
// Find users whose email ends with @company.com
let filter = Filter::like("email", "%@company.com");

// Find products starting with "Pro"
let filter = Filter::like("name", "Pro%");

// Find codes with pattern XX-###
let filter = Filter::like("code", "__-___");

// Find text containing literal %
let filter = Filter::like("description", "%%25%% off");
}

Null Checks

Check for null or non-null values:

#![allow(unused)]
fn main() {
// Find users without a phone number
let filter = Filter::is_null("phone");

// Find users with a profile picture
let filter = Filter::not_null("avatar_url");
}

Combining Filters

Filters can be combined using logical operators:

AND - Both conditions must match:

#![allow(unused)]
fn main() {
// Active users over 18
let filter = Filter::eq("status", Value::Text("active".into()))
.and(Filter::gt("age", Value::Int32(18.into())));
}

OR - Either condition matches:

#![allow(unused)]
fn main() {
// Admins or moderators
let filter = Filter::eq("role", Value::Text("admin".into()))
.or(Filter::eq("role", Value::Text("moderator".into())));
}

NOT - Negate a condition:

#![allow(unused)]
fn main() {
// Users who are not banned
let filter = Filter::eq("status", Value::Text("banned".into())).not();
}

Complex combinations:

#![allow(unused)]
fn main() {
// (active AND age > 18) OR role = "admin"
let filter = Filter::eq("status", Value::Text("active".into()))
.and(Filter::gt("age", Value::Int32(18.into())))
.or(Filter::eq("role", Value::Text("admin".into())));

// NOT (deleted OR archived)
let filter = Filter::eq("status", Value::Text("deleted".into()))
.or(Filter::eq("status", Value::Text("archived".into())))
.not();
}

JSON Filters

For columns with Json type, use specialized JSON filters. See the JSON Reference for comprehensive documentation.

Quick examples:

#![allow(unused)]
fn main() {
// Check if JSON contains a pattern
let pattern = Json::from_str(r#"{"active": true}"#).unwrap();
let filter = Filter::json("metadata", JsonFilter::contains(pattern));

// Extract and compare a value
let filter = Filter::json(
"settings",
JsonFilter::extract_eq("theme", Value::Text("dark".into()))
);

// Check if a path exists
let filter = Filter::json("data", JsonFilter::has_key("user.email"));
}

Ordering

Single Column Ordering

Sort results by a single column:

#![allow(unused)]
fn main() {
// Sort by name ascending (A-Z)
let query = Query::builder()
.all()
.order_by("name", OrderDirection::Ascending)
.build();

// Sort by created_at descending (newest first)
let query = Query::builder()
.all()
.order_by("created_at", OrderDirection::Descending)
.build();
}

Multiple Column Ordering

Chain multiple order_by calls for secondary sorting:

#![allow(unused)]
fn main() {
// Sort by category, then by price within each category
let query = Query::builder()
.all()
.order_by("category", OrderDirection::Ascending)
.order_by("price", OrderDirection::Descending)
.build();

// Sort by status, then by priority, then by created_at
let query = Query::builder()
.all()
.order_by("status", OrderDirection::Ascending)
.order_by("priority", OrderDirection::Descending)
.order_by("created_at", OrderDirection::Ascending)
.build();
}

Pagination

Limit

Restrict the number of records returned:

#![allow(unused)]
fn main() {
// Get only the first 10 records
let query = Query::builder()
.all()
.limit(10)
.build();
}

Offset

Skip a number of records before returning results:

#![allow(unused)]
fn main() {
// Skip the first 20 records
let query = Query::builder()
.all()
.offset(20)
.build();
}

Pagination Pattern

Combine limit and offset for pagination:

#![allow(unused)]
fn main() {
const PAGE_SIZE: u64 = 20;

fn get_page_query(page: u64) -> Query {
    Query::builder()
        .all()
        .order_by("id", OrderDirection::Ascending)  // Consistent ordering is important
        .limit(PAGE_SIZE)
        .offset(page * PAGE_SIZE)
        .build()
}

// Page 0: records 0-19
let page_0 = get_page_query(0);

// Page 1: records 20-39
let page_1 = get_page_query(1);

// Page 2: records 40-59
let page_2 = get_page_query(2);
}

Tip: Always use order_by with pagination to ensure consistent ordering across pages.

Field Selection

Select All Fields

Use .all() to select all columns:

#![allow(unused)]
fn main() {
let query = Query::builder()
.all()
.build();

let users = database.select::<User>(query)?;
// All fields are populated
}

Select Specific Fields

Use .columns() to select only specific columns:

#![allow(unused)]
fn main() {
let query = Query::builder()
.columns(vec!["id".to_string(), "name".to_string(), "email".to_string()])
.build();

let users = database.select::<User>(query)?;
// Only id, name, and email are populated
// Other fields will have default values
}

Note: The primary key is always included, even if not specified.

Eager Loading

Load related records in a single query using .with():

#![allow(unused)]
fn main() {
// Define tables with foreign key
#[derive(Debug, Table, Clone, PartialEq, Eq)]
#[table = "posts"]
pub struct Post {
    #[primary_key]
    pub id: Uint32,
    pub title: Text,
    #[foreign_key(entity = "User", table = "users", column = "id")]
    pub author_id: Uint32,
}

// Query posts with authors eagerly loaded
let query = Query::builder()
.all()
.with("users")
.build();

let posts = database.select::<Post>(query)?;
}

See the Relationships Guide for more on eager loading.

Distinct

Use .distinct(&[...]) to remove duplicate rows from the result set based on one or more columns. Rows are deduplicated by the tuple of values across the listed columns; the first row encountered for each distinct tuple is kept.

Basic Distinct

#![allow(unused)]
fn main() {
// Get the unique set of names from the users table
let query = Query::builder()
    .all()
    .distinct(&["name"])
    .build();

let users = database.select::<User>(query)?;
}

Distinct by Multiple Columns

#![allow(unused)]
fn main() {
// Unique (category, vendor) pairs from products
let query = Query::builder()
    .all()
    .distinct(&["category", "vendor"])
    .build();

let products = database.select::<Product>(query)?;
}

Distinct with Ordering and Pagination

DISTINCT runs before ORDER BY, OFFSET, and LIMIT, so paging through distinct values works as expected:

#![allow(unused)]
fn main() {
// Page 2 (size 10) of unique names, alphabetical
let query = Query::builder()
    .all()
    .distinct(&["name"])
    .order_by_asc("name")
    .offset(10)
    .limit(10)
    .build();
}

Without DISTINCT, LIMIT 10 could yield ten copies of the same name. With DISTINCT, the limit applies to the deduplicated stream.

Distinct Semantics

• Lookup is performed against the source row’s columns. The columns named in .distinct(...) do not need to be in the field selection.
• A column not present on the row is treated as Value::Null. Listing an unknown column collapses every row into a single result.
• Calling .distinct(&[]) (or omitting it) is a no-op.
• Pipeline order: WHERE -> DISTINCT -> eager loading -> column selection -> ORDER BY -> OFFSET / LIMIT. See the Query API Reference for the full pipeline.

Tip: distinct(&[pk_column]) returns at most one row per primary key, which can be useful when joining sources that fan out the parent rows.

Aggregations

Aggregations summarise groups of rows using COUNT, SUM, AVG, MIN, and MAX. Group rows with .group_by(...), filter the resulting groups with .having(...), and describe the aggregates to compute via the AggregateFunction enum.

Defining Aggregates

Each aggregate is one variant of AggregateFunction:

#![allow(unused)]
fn main() {
use wasm_dbms_api::prelude::AggregateFunction;

let aggregates = vec![
    AggregateFunction::Count(None),               // COUNT(*)
    AggregateFunction::Count(Some("email".into())), // COUNT(email)
    AggregateFunction::Sum("amount".into()),
    AggregateFunction::Avg("amount".into()),
    AggregateFunction::Min("created_at".into()),
    AggregateFunction::Max("created_at".into()),
];
}

Count(None) counts every row in the group; Count(Some(col)) counts only rows where col is non-null. The other variants take a column name and operate over its values.

Group By and Having

Use .group_by(&[...]) to define grouping keys and .having(filter) to filter the aggregated groups:

#![allow(unused)]
fn main() {
let query = Query::builder()
    .all()
    .group_by(&["category"])
    .having(Filter::gt("count", Value::Uint64(10u64.into())))
    .order_by_desc("category")
    .build();
}

HAVING is evaluated after aggregation, against grouping keys and aggregate results. WHERE (set with .and_where() / .or_where()) still applies first to the raw rows.

Aggregate Result Types

Aggregated queries return AggregatedRow values:

#![allow(unused)]
fn main() {
pub struct AggregatedRow {
    pub group_keys: Vec<Value>,
    pub values: Vec<AggregatedValue>,
}
}

group_keys carries the grouping tuple (one Value per group_by column). values holds one AggregatedValue per requested aggregate, in the same order as the AggregateFunction list.

#![allow(unused)]
fn main() {
pub enum AggregatedValue {
    Count(u64),
    Sum(Value),
    Avg(Value),
    Min(Value),
    Max(Value),
}
}

Count is always u64; the other variants wrap a Value whose concrete variant matches the source column’s data type.

See the Query API Reference for the full type definitions and pipeline ordering.

Joins

Joins combine rows from two or more tables based on a related column, producing a single result set with columns from all joined tables. Use joins when you need to correlate data across tables in a single flat result – for example, listing posts alongside their author names.

Note: Joins require the select_join method, which returns rows with [JoinColumnDef] that include the source table name. Typed select::<T> rejects queries that contain joins with a JoinInsideTypedSelect error.

Join Types

Basic Join

Use .inner_join(table, left_column, right_column) to join two tables:

#![allow(unused)]
fn main() {
use wasm_dbms_api::prelude::*;

// Join users with their posts (INNER JOIN)
let query = Query::builder()
    .all()
    .inner_join("posts", "id", "user_id")
    .build();

// Use select_join since joins return rows with table provenance
let rows = database.select_join("users", query)?;

// Each row contains columns from both "users" and "posts"
for row in &rows {
    for (col_def, value) in row {
        // col_def.table tells you which table the column came from
        println!("{}.{} = {:?}", col_def.table.as_deref().unwrap_or("?"), col_def.name, value);
    }
}
}

Left, Right, and Full Joins

#![allow(unused)]
fn main() {
// LEFT JOIN: all users, even those without posts
let query = Query::builder()
    .all()
    .left_join("posts", "id", "user_id")
    .build();

// RIGHT JOIN: all posts, even those with missing/deleted authors
let query = Query::builder()
    .all()
    .right_join("posts", "id", "user_id")
    .build();

// FULL JOIN: all users and all posts, matched where possible
let query = Query::builder()
    .all()
    .full_join("posts", "id", "user_id")
    .build();
}

For LEFT, RIGHT, and FULL joins, columns from the unmatched side are filled with Value::Null.

Chaining Multiple Joins

Chain multiple joins to combine more than two tables:

#![allow(unused)]
fn main() {
// Users -> Posts -> Comments
let query = Query::builder()
    .all()
    .inner_join("posts", "id", "user_id")
    .left_join("comments", "posts.id", "post_id")
    .build();

let rows = database.select_join("users", query)?;
}

Joins are processed left-to-right. The second join operates on the result of the first.

Qualified Column Names

When joining tables that share column names, use table.column syntax to disambiguate:

#![allow(unused)]
fn main() {
// Both "users" and "posts" have an "id" column
let query = Query::builder()
    .field("users.id")
    .field("users.name")
    .field("posts.title")
    .inner_join("posts", "users.id", "user_id")
    .and_where(Filter::eq("users.name", Value::Text("Alice".into())))
    .order_by_asc("posts.title")
    .build();
}

Qualified names (table.column) work in:

• Field selection (.field())
• Filters (.and_where(), .or_where())
• Ordering (.order_by_asc(), .order_by_desc())
• Join ON conditions

Unqualified names default to the FROM table (the table passed to select_join).

Joins vs Eager Loading

Use eager loading when you want typed results with related records attached. Use joins when you need a flat, cross-table result set – for example, for reporting, search, or when you need columns from multiple tables in a single row.

Index-Accelerated Queries

When a table has indexes defined (via #[index] or the automatic primary key index), the query engine can use them to avoid full table scans. This happens transparently — you write the same filters as before, and the engine picks the best available index.

How Indexes Improve Queries

Without indexes, every SELECT, UPDATE, and DELETE scans all records in the table. With an index on the filtered column, the engine navigates the B-tree to locate matching records directly, then loads only those records from memory.

#![allow(unused)]
fn main() {
#[derive(Debug, Table, Clone, PartialEq, Eq)]
#[table = "users"]
pub struct User {
    #[primary_key]
    pub id: Uint32,
    #[index]
    pub email: Text,
    pub name: Text,
}

// This uses the index on `email` — no full table scan
let query = Query::builder()
    .filter(Filter::eq("email", Value::Text("alice@example.com".into())))
    .build();

let users = database.select::<User>(query)?;
}

Which Filters Use Indexes

The filter analyzer extracts an index plan from the leftmost AND-chain of conditions on indexed columns:

Filters that fall back to full scan:

• OR at the top level
• NOT wrapping an indexable condition
• Filters on non-indexed columns
• Complex nested expressions

Residual Filters

When the index narrows down the candidate set but doesn’t fully satisfy the filter, the remaining conditions are applied as a residual check on each loaded record:

#![allow(unused)]
fn main() {
// Index on `email` handles the equality check.
// `name LIKE 'A%'` is applied as a residual filter on the results.
let filter = Filter::eq("email", Value::Text("alice@example.com".into()))
    .and(Filter::like("name", "A%"));
}

Transaction-Aware Lookups

Inside a transaction, index lookups are merged with the transaction overlay. Records added in the current transaction appear in index results, and deleted records are excluded — even though the on-disk B-tree has not been modified yet. On commit, overlay changes are flushed to the persistent B-tree. On rollback, the overlay is discarded and the B-tree remains unchanged.

Relationships

• Relationships
  • Overview
  • Defining Foreign Keys
    • Foreign Key Syntax
    • Foreign Key Constraints
  • Referential Integrity
    • Insert Validation
    • Update Validation
  • Delete Behaviors
    • Restrict
    • Cascade
    • Choosing a Delete Behavior
  • Eager Loading
    • Basic Eager Loading
    • Multiple Relations
    • Eager Loading with Filters
    • Cross-Table Queries with Joins
  • Common Patterns
    • One-to-Many
    • Many-to-Many
    • Self-Referential

Overview

wasm-dbms supports foreign key relationships between tables, providing:

• Referential integrity: Ensures foreign keys point to valid records
• Delete behaviors: Control what happens when referenced records are deleted
• Eager loading: Load related records in a single query

Defining Foreign Keys

Foreign Key Syntax

Use the #[foreign_key] attribute to define relationships:

#![allow(unused)]
fn main() {
#[derive(Debug, Table, Clone, PartialEq, Eq)]
#[table = "posts"]
pub struct Post {
    #[primary_key]
    pub id: Uint32,
    pub title: Text,
    pub content: Text,

    #[foreign_key(entity = "User", table = "users", column = "id")]
    pub author_id: Uint32,
}
}

Attribute parameters:

Foreign Key Constraints

When you define a foreign key:

1. The field type must match the referenced column type
2. The referenced table must be registered in your database schema
3. Foreign key values must reference existing records (enforced on insert/update)

Referential Integrity

wasm-dbms enforces referential integrity automatically.

Insert Validation

When inserting a record with a foreign key, the referenced record must exist:

#![allow(unused)]
fn main() {
// This user exists
database.insert::<User>(UserInsertRequest {
    id: 1.into(),
    name: "Alice".into(),
    ..
})?;

// Insert post referencing existing user - OK
database.insert::<Post>(PostInsertRequest {
    id: 1.into(),
    title: "My Post".into(),
    author_id: 1.into(),  // User 1 exists
    ..
})?;

// Insert post referencing non-existent user - FAILS
let result = database.insert::<Post>(PostInsertRequest {
    id: 2.into(),
    title: "Another Post".into(),
    author_id: 999.into(),  // User 999 doesn't exist
    ..
});

assert!(matches!(
    result,
    Err(DbmsError::Query(QueryError::BrokenForeignKeyReference))
));
}

Update Validation

Updates are also validated:

#![allow(unused)]
fn main() {
// Changing author_id to non-existent user fails
let update = PostUpdateRequest::builder()
    .set_author_id(999.into())  // User 999 doesn't exist
    .filter(Filter::eq("id", Value::Uint32(1.into())))
    .build();

let result = database.update::<Post>(update);
assert!(matches!(
    result,
    Err(DbmsError::Query(QueryError::BrokenForeignKeyReference))
));
}

Delete Behaviors

When deleting a record that is referenced by other records, you must specify how to handle the references.

Restrict

Behavior: Fail if any records reference this one.

#![allow(unused)]
fn main() {
use wasm_dbms_api::prelude::DeleteBehavior;

// User has posts - delete fails
let result = database.delete::<User>(
    DeleteBehavior::Restrict,
    Some(Filter::eq("id", Value::Uint32(1.into()))),
);

match result {
    Err(DbmsError::Query(QueryError::ForeignKeyConstraintViolation)) => {
        println!("Cannot delete: user has posts");
    }
    _ => {}
}

// Delete posts first, then user
database.delete::<Post>(
    DeleteBehavior::Restrict,
    Some(Filter::eq("author_id", Value::Uint32(1.into()))),
)?;

// Now user can be deleted
database.delete::<User>(
    DeleteBehavior::Restrict,
    Some(Filter::eq("id", Value::Uint32(1.into()))),
)?;
}

Use when: You want to prevent accidental data loss. The caller must explicitly handle related records.

Cascade

Behavior: Delete all records that reference this one (recursively).

#![allow(unused)]
fn main() {
// Deletes user AND all their posts
database.delete::<User>(
    DeleteBehavior::Cascade,
    Some(Filter::eq("id", Value::Uint32(1.into()))),
)?;
}

Cascade is recursive:

#![allow(unused)]
fn main() {
// Schema:
// User -> Posts -> Comments
// Deleting a user cascades to posts, which cascades to comments

database.delete::<User>(
    DeleteBehavior::Cascade,
    Some(Filter::eq("id", Value::Uint32(1.into()))),
)?;
// User deleted
// All user's posts deleted
// All comments on those posts deleted
}

Use when: Related records have no meaning without the parent (e.g., comments on a deleted post).

Choosing a Delete Behavior

Eager Loading

Eager loading fetches related records in a single query, avoiding N+1 query problems.

Basic Eager Loading

Use .with() to eager load a related table:

#![allow(unused)]
fn main() {
// Load posts with their authors
let query = Query::builder()
    .all()
    .with("users")  // Name of the related table
    .build();

let posts = database.select::<Post>(query)?;

// Each post now has author data available
for post in posts {
    println!("Post '{}' by author_id {}", post.title, post.author_id);
}
}

Multiple Relations

Load multiple related tables:

#![allow(unused)]
fn main() {
// Schema:
// Post -> User (author)
// Post -> Category

let query = Query::builder()
    .all()
    .with("users")
    .with("categories")
    .build();

let posts = database.select::<Post>(query)?;
}

Eager Loading with Filters

Combine eager loading with filters:

#![allow(unused)]
fn main() {
// Load published posts with their authors
let query = Query::builder()
    .filter(Filter::eq("published", Value::Boolean(true)))
    .order_by("created_at", OrderDirection::Descending)
    .limit(10)
    .with("users")
    .build();

let posts = database.select::<Post>(query)?;
}

Cross-Table Queries with Joins

In addition to eager loading, wasm-dbms supports SQL-style joins (INNER, LEFT, RIGHT, FULL) for combining rows from multiple tables into a flat result set. Joins are useful when you need columns from several tables in a single row – for example, listing post titles alongside author names. Unlike eager loading, joins return untyped results via the select_raw path.

See the Querying Guide – Joins section for full details and examples.

Common Patterns

One-to-Many

A user has many posts:

#![allow(unused)]
fn main() {
#[derive(Debug, Table, Clone, PartialEq, Eq)]
#[table = "users"]
pub struct User {
    #[primary_key]
    pub id: Uint32,
    pub name: Text,
}

#[derive(Debug, Table, Clone, PartialEq, Eq)]
#[table = "posts"]
pub struct Post {
    #[primary_key]
    pub id: Uint32,
    pub title: Text,
    #[foreign_key(entity = "User", table = "users", column = "id")]
    pub author_id: Uint32,
}

// Query all posts by a user
let query = Query::builder()
    .filter(Filter::eq("author_id", Value::Uint32(user_id.into())))
    .build();
let user_posts = database.select::<Post>(query)?;
}

Many-to-Many

Use a junction table for many-to-many relationships:

#![allow(unused)]
fn main() {
// Students and Courses (many-to-many)

#[derive(Debug, Table, Clone, PartialEq, Eq)]
#[table = "students"]
pub struct Student {
    #[primary_key]
    pub id: Uint32,
    pub name: Text,
}

#[derive(Debug, Table, Clone, PartialEq, Eq)]
#[table = "courses"]
pub struct Course {
    #[primary_key]
    pub id: Uint32,
    pub title: Text,
}

#[derive(Debug, Table, Clone, PartialEq, Eq)]
#[table = "enrollments"]
pub struct Enrollment {
    #[primary_key]
    pub id: Uint32,
    #[foreign_key(entity = "Student", table = "students", column = "id")]
    pub student_id: Uint32,
    #[foreign_key(entity = "Course", table = "courses", column = "id")]
    pub course_id: Uint32,
    pub enrolled_at: DateTime,
}

// Find all courses for a student
let query = Query::builder()
    .filter(Filter::eq("student_id", Value::Uint32(student_id.into())))
    .with("courses")
    .build();
let enrollments = database.select::<Enrollment>(query)?;
}

Self-Referential

A table can reference itself (e.g., categories with parent categories, employees with managers):

#![allow(unused)]
fn main() {
#[derive(Debug, Table, Clone, PartialEq, Eq)]
#[table = "employees"]
pub struct Employee {
    #[primary_key]
    pub id: Uint32,
    pub name: Text,
    #[foreign_key(entity = "Employee", table = "employees", column = "id")]
    pub manager_id: Nullable<Uint32>,  // Nullable for top-level employees
}

// Find all employees under a manager
let query = Query::builder()
    .filter(Filter::eq("manager_id", Value::Uint32(manager_id.into())))
    .build();
let direct_reports = database.select::<Employee>(query)?;
}

#![allow(unused)]
fn main() {
#[derive(Debug, Table, Clone, PartialEq, Eq)]
#[table = "categories"]
pub struct Category {
    #[primary_key]
    pub id: Uint32,
    pub name: Text,
    #[foreign_key(entity = "Category", table = "categories", column = "id")]
    pub parent_id: Nullable<Uint32>,  // Nullable for root categories
}

// Find root categories
let query = Query::builder()
    .filter(Filter::is_null("parent_id"))
    .build();
let root_categories = database.select::<Category>(query)?;

// Find children of a category
let query = Query::builder()
    .filter(Filter::eq("parent_id", Value::Uint32(parent_id.into())))
    .build();
let children = database.select::<Category>(query)?;
}

Transactions

• Transactions
  • Overview
  • Transaction Lifecycle
    • Begin Transaction
    • Perform Operations
    • Commit
    • Rollback
  • ACID Properties
    • Atomicity
    • Consistency
    • Isolation
    • Durability
  • Error Handling
    • Handling Failures
    • Transaction Errors
  • Best Practices
    • 1. Keep transactions short
    • 2. Always handle rollback
    • 3. Use transactions for related operations
    • 4. Don’t mix transactional and non-transactional operations
  • Examples
    • Bank Transfer
    • Order Processing

Overview

wasm-dbms supports ACID transactions, allowing you to group multiple database operations into a single atomic unit. Either all operations succeed and are committed together, or none of them take effect.

Key features:

• Atomicity: All operations in a transaction succeed or fail together
• Consistency: Data integrity constraints are maintained
• Isolation: Transactions are isolated from each other
• Durability: Committed changes persist

Transaction Lifecycle

Begin Transaction

Start a new transaction using DbmsContext::begin_transaction():

#![allow(unused)]
fn main() {
use wasm_dbms::prelude::*;

// Begin a new transaction
let tx_id = ctx.begin_transaction();
println!("Started transaction: {}", tx_id);
}

The returned transaction ID is used to create a transactional database instance.

Perform Operations

Create a WasmDbmsDatabase with the transaction ID and perform operations:

#![allow(unused)]
fn main() {
let mut database = WasmDbmsDatabase::from_transaction(&ctx, my_schema, tx_id);

// Insert within transaction
database.insert::<User>(user)?;

// Update within transaction
database.update::<User>(update)?;

// Delete within transaction
database.delete::<User>(DeleteBehavior::Restrict, Some(filter))?;

// Select within transaction (sees uncommitted changes)
let users = database.select::<User>(query)?;
}

Note: Operations within a transaction are visible to subsequent operations in the same transaction, but not to other callers until committed.

Commit

Commit the transaction to make all changes permanent:

#![allow(unused)]
fn main() {
// Commit the transaction
database.commit()?;
println!("Transaction committed successfully");
}

After commit:

• All changes become visible to other callers
• The transaction ID becomes invalid
• Changes persist in storage

Rollback

Rollback the transaction to discard all changes:

#![allow(unused)]
fn main() {
// Rollback the transaction
database.rollback()?;
println!("Transaction rolled back");
}

After rollback:

• All changes within the transaction are discarded
• The transaction ID becomes invalid
• The database state is as if the transaction never happened

ACID Properties

Atomicity

All operations in a transaction are treated as a single unit. If any operation fails, the entire transaction can be rolled back:

#![allow(unused)]
fn main() {
let tx_id = ctx.begin_transaction();
let mut database = WasmDbmsDatabase::from_transaction(&ctx, my_schema, tx_id);

// First operation succeeds
database.insert::<User>(user1)?;

// Second operation fails (e.g., primary key conflict)
let result = database.insert::<User>(user2_duplicate);

if result.is_err() {
    // Rollback everything - user1 is also discarded
    database.rollback()?;
}
}

Consistency

Transactions maintain data integrity:

• Primary key uniqueness is enforced
• Foreign key constraints are checked
• Validators run on all data
• Sanitizers are applied

#![allow(unused)]
fn main() {
let tx_id = ctx.begin_transaction();
let mut database = WasmDbmsDatabase::from_transaction(&ctx, my_schema, tx_id);

// This will fail if referenced user doesn't exist
let post = PostInsertRequest {
    id: 1.into(),
    title: "My Post".into(),
    author_id: 999.into(),  // Non-existent user
};

let result = database.insert::<Post>(post);
// Returns Err(DbmsError::Query(QueryError::BrokenForeignKeyReference))
}

Isolation

Changes made within a transaction are not visible to other callers until committed:

#![allow(unused)]
fn main() {
// Database A starts a transaction
let tx_id = ctx.begin_transaction();
let mut db_a = WasmDbmsDatabase::from_transaction(&ctx, my_schema, tx_id);
db_a.insert::<User>(new_user)?;

// Database B queries - does NOT see the new user
let db_b = WasmDbmsDatabase::oneshot(&ctx, my_schema);
let users = db_b.select::<User>(query)?;
assert!(!users.iter().any(|u| u.id == new_user.id));

// Database A commits
db_a.commit()?;

// Now Database B can see the user
let users = db_b.select::<User>(query)?;
assert!(users.iter().any(|u| u.id == new_user.id));
}

Durability

Committed transactions persist in storage. When using stable memory providers (e.g., on the Internet Computer), data survives across upgrades.

Error Handling

Handling Failures

When an operation fails within a transaction, you should typically rollback:

#![allow(unused)]
fn main() {
let tx_id = ctx.begin_transaction();
let mut database = WasmDbmsDatabase::from_transaction(&ctx, my_schema, tx_id);

fn process_order(database: &impl Database) -> Result<(), DbmsError> {
    // Multiple operations that should succeed together
    database.insert::<Order>(order)?;
    database.update::<Inventory>(update)?;
    database.insert::<OrderItem>(item)?;
    Ok(())
}

match process_order(&database) {
    Ok(()) => {
        database.commit()?;
        println!("Order processed successfully");
    }
    Err(e) => {
        database.rollback()?;
        println!("Order failed, rolled back: {:?}", e);
    }
}
}

Transaction Errors

#![allow(unused)]
fn main() {
use wasm_dbms_api::prelude::{DbmsError, TransactionError};

match database.commit() {
    Ok(()) => println!("Committed"),
    Err(DbmsError::Transaction(TransactionError::NoActiveTransaction)) => {
        println!("No active transaction to commit");
    }
    Err(e) => println!("Other error: {:?}", e),
}
}

Best Practices

1. Keep transactions short

Long-running transactions hold resources and block other operations:

#![allow(unused)]
fn main() {
// GOOD: Prepare data outside transaction
let users_to_insert = prepare_users();

let tx_id = ctx.begin_transaction();
let mut database = WasmDbmsDatabase::from_transaction(&ctx, my_schema, tx_id);
for user in users_to_insert {
    database.insert::<User>(user)?;
}
database.commit()?;

// BAD: Doing expensive work inside transaction
let tx_id = ctx.begin_transaction();
let mut database = WasmDbmsDatabase::from_transaction(&ctx, my_schema, tx_id);
for raw_data in large_dataset {
    let user = expensive_parsing(raw_data);  // Don't do this in transaction
    database.insert::<User>(user)?;
}
database.commit()?;
}

2. Always handle rollback

Ensure transactions are either committed or rolled back:

#![allow(unused)]
fn main() {
let tx_id = ctx.begin_transaction();
let mut database = WasmDbmsDatabase::from_transaction(&ctx, my_schema, tx_id);

let result = (|| -> Result<(), DbmsError> {
    database.insert::<User>(user1)?;
    database.insert::<User>(user2)?;
    Ok(())
})();

match result {
    Ok(()) => database.commit()?,
    Err(_) => database.rollback()?,
}
}

3. Use transactions for related operations

Group operations that should succeed or fail together:

#![allow(unused)]
fn main() {
// GOOD: Related operations in transaction
let tx_id = ctx.begin_transaction();
let mut database = WasmDbmsDatabase::from_transaction(&ctx, my_schema, tx_id);
database.insert::<Order>(order)?;
database.insert::<Payment>(payment)?;
database.update::<Inventory>(inv_update)?;
database.commit()?;

// BAD: Unrelated operations in transaction (unnecessary)
let tx_id = ctx.begin_transaction();
let mut database = WasmDbmsDatabase::from_transaction(&ctx, my_schema, tx_id);
database.insert::<UserPreferences>(prefs)?;
database.insert::<AuditLog>(log)?;  // Unrelated
database.commit()?;
}

4. Don’t mix transactional and non-transactional operations

#![allow(unused)]
fn main() {
let tx_id = ctx.begin_transaction();
let mut database = WasmDbmsDatabase::from_transaction(&ctx, my_schema, tx_id);

// GOOD: All operations use the transaction
database.insert::<Order>(order)?;
database.insert::<OrderItem>(item)?;

// BAD: Mixing transaction and non-transaction
let oneshot = WasmDbmsDatabase::oneshot(&ctx, my_schema);
database.insert::<Order>(order)?;
oneshot.insert::<AuditLog>(log)?;  // Not in transaction!
}

Examples

Bank Transfer

Transfer money between accounts atomically:

#![allow(unused)]
fn main() {
fn transfer(
    ctx: &DbmsContext<impl MemoryProvider>,
    from_account: u32,
    to_account: u32,
    amount: Decimal,
) -> Result<(), DbmsError> {
    let tx_id = ctx.begin_transaction();
    let mut database = WasmDbmsDatabase::from_transaction(ctx, my_schema, tx_id);

    // Deduct from source account
    let deduct = AccountUpdateRequest::builder()
        .decrease_balance(amount)
        .filter(Filter::eq("id", Value::Uint32(from_account.into())))
        .build();
    database.update::<Account>(deduct)?;

    // Add to destination account
    let add = AccountUpdateRequest::builder()
        .increase_balance(amount)
        .filter(Filter::eq("id", Value::Uint32(to_account.into())))
        .build();
    database.update::<Account>(add)?;

    // Record the transfer
    let transfer_record = TransferInsertRequest {
        id: Uuid::new_v4().into(),
        from_account: from_account.into(),
        to_account: to_account.into(),
        amount,
        timestamp: DateTime::now(),
    };
    database.insert::<Transfer>(transfer_record)?;

    // Commit atomically
    database.commit()?;
    Ok(())
}
}

Order Processing

Process an order with inventory update:

#![allow(unused)]
fn main() {
fn process_order(
    ctx: &DbmsContext<impl MemoryProvider>,
    order: OrderInsertRequest,
    items: Vec<OrderItemInsertRequest>,
) -> Result<u32, Box<dyn std::error::Error>> {
    let tx_id = ctx.begin_transaction();
    let mut database = WasmDbmsDatabase::from_transaction(ctx, my_schema, tx_id);

    // Insert the order
    database.insert::<Order>(order.clone())?;

    // Insert order items and update inventory
    for item in items {
        // Insert order item
        database.insert::<OrderItem>(item.clone())?;

        // Decrease inventory
        let inv_update = InventoryUpdateRequest::builder()
            .decrease_quantity(item.quantity)
            .filter(Filter::eq("product_id", Value::Uint32(item.product_id)))
            .build();

        let updated = database.update::<Inventory>(inv_update)?;

        if updated == 0 {
            // Product not in inventory, rollback
            database.rollback()?;
            return Err("Product not found in inventory".into());
        }
    }

    // All successful, commit
    database.commit()?;
    Ok(order.id.into())
}
}

Custom Data Types

• Custom Data Types
  • Overview
  • Defining a Custom Type
    • Step 1: Define the Type
    • Step 2: Implement Display
    • Step 3: Implement Encode
    • Step 4: Implement DataType and Derive CustomDataType
  • Using Custom Types in Tables
    • The custom_type Attribute
    • Nullable Custom Types
  • Filtering and Querying
  • Ordering Contract
  • Examples
    • Enum: Priority
    • Struct: Address

Overview

wasm-dbms ships with a set of built-in data types that cover the most common use cases. When your domain requires types that go beyond those built-ins, you can define custom data types.

Custom data types let you store any Rust type – enums, newtypes, structs – inside your tables. The DBMS engine stores them as opaque bytes internally and uses a type tag string to identify each custom type.

When to use custom types:

• Domain-specific enums (e.g., Priority, Status, Role)
• Composite value objects (e.g., Address, Coordinates)
• Newtypes that wrap primitives with domain meaning (e.g., Email(String))

Defining a Custom Type

Creating a custom type requires four steps:

1. Define the type with the required derives
2. Implement Display
3. Implement Encode (binary serialization)
4. Implement DataType and derive CustomDataType

Step 1: Define the Type

Your type must derive or implement several traits. For enums, all must be implemented manually or derived:

#![allow(unused)]
fn main() {
use serde::{Deserialize, Serialize};

#[derive(
    Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord,
    Hash, Default, Serialize, Deserialize,
)]
pub enum Priority {
    #[default]
    Low,
    Medium,
    High,
}
}

For structs, the same traits are required:

#![allow(unused)]
fn main() {
#[derive(
    Debug, Clone, PartialEq, Eq, PartialOrd, Ord,
    Hash, Default, Serialize, Deserialize,
)]
pub struct Address {
    pub street: String,
    pub city: String,
    pub zip: String,
}
}

Required traits:

Note: For IC canister usage, also derive CandidType and Deserialize from the candid crate.

Step 2: Implement Display

The Display implementation provides a human-readable representation used for logging and diagnostics:

#![allow(unused)]
fn main() {
use std::fmt;

impl fmt::Display for Priority {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        match self {
            Priority::Low => write!(f, "low"),
            Priority::Medium => write!(f, "medium"),
            Priority::High => write!(f, "high"),
        }
    }
}

impl fmt::Display for Address {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        write!(f, "{}, {} {}", self.street, self.city, self.zip)
    }
}
}

Step 3: Implement Encode

The Encode trait defines how your type is serialized to and from bytes for memory storage. Enums require a manual implementation; for structs, you can use #[derive(Encode)].

Enum (manual implementation):

#![allow(unused)]
fn main() {
use std::borrow::Cow;
use wasm_dbms_api::prelude::*;

impl Encode for Priority {
    const SIZE: DataSize = DataSize::Fixed(1);
    const ALIGNMENT: PageOffset = DEFAULT_ALIGNMENT;

    fn encode(&self) -> Cow<'_, [u8]> {
        Cow::Owned(vec![match self {
            Priority::Low => 0,
            Priority::Medium => 1,
            Priority::High => 2,
        }])
    }

    fn decode(data: Cow<[u8]>) -> MemoryResult<Self> {
        match data[0] {
            0 => Ok(Priority::Low),
            1 => Ok(Priority::Medium),
            2 => Ok(Priority::High),
            other => Err(MemoryError::DecodeError(
                DecodeError::TryFromSliceError(
                    format!("invalid Priority byte: {other}"),
                ),
            )),
        }
    }

    fn size(&self) -> MSize {
        1
    }
}
}

Struct (derive macro):

The #[derive(Encode)] macro works for structs whose fields all implement Encode. Since String does not implement Encode but Text does, use wasm-dbms types for the struct fields:

#![allow(unused)]
fn main() {
use wasm_dbms_api::prelude::*;

#[derive(
    Debug, Clone, PartialEq, Eq, PartialOrd, Ord,
    Hash, Default, Serialize, Deserialize,
    Encode,
)]
pub struct Address {
    pub street: Text,
    pub city: Text,
    pub zip: Text,
}
}

Key Encode concepts:

Step 4: Implement DataType and Derive CustomDataType

Finally, implement the DataType marker trait and derive CustomDataType with a unique type tag:

#![allow(unused)]
fn main() {
use wasm_dbms_api::prelude::*;

impl DataType for Priority {}

// Manual CustomDataType implementation for enums
impl CustomDataType for Priority {
    const TYPE_TAG: &'static str = "priority";
}

// Manual From<Priority> for Value implementation
impl From<Priority> for Value {
    fn from(val: Priority) -> Value {
        Value::Custom(CustomValue {
            type_tag: <Priority as CustomDataType>::TYPE_TAG.to_string(),
            encoded: Encode::encode(&val).into_owned(),
            display: val.to_string(),
        })
    }
}
}

For structs, you can use the CustomDataType derive macro instead of the manual implementation above:

#![allow(unused)]
fn main() {
use wasm_dbms_api::prelude::*;

impl DataType for Address {}

#[derive(CustomDataType)]
#[type_tag = "address"]
pub struct Address {
    // ...
}
}

The #[derive(CustomDataType)] macro generates both the CustomDataType trait implementation and the From<T> for Value conversion. For enums, you must write these implementations manually.

Type tag rules:

• Must be unique across all custom types in your database
• Must be stable across upgrades (changing it makes existing data unreadable)
• Use lowercase, descriptive names (e.g., "priority", "address", "role")

Using Custom Types in Tables

The custom_type Attribute

To use a custom type in a table, annotate the field with #[custom_type]:

#![allow(unused)]
fn main() {
use wasm_dbms_api::prelude::*;

#[derive(Debug, Table, Clone, PartialEq, Eq)]
#[table = "tasks"]
pub struct Task {
    #[primary_key]
    pub id: Uint32,
    pub title: Text,
    #[custom_type]
    pub priority: Priority,
    #[custom_type]
    pub address: Address,
}
}

Without the #[custom_type] attribute, the Table macro won’t know how to handle your type and compilation will fail.

Nullable Custom Types

Custom types can be wrapped in Nullable<T> for optional fields:

#![allow(unused)]
fn main() {
#[derive(Debug, Table, Clone, PartialEq, Eq)]
#[table = "tasks"]
pub struct Task {
    #[primary_key]
    pub id: Uint32,
    pub title: Text,
    #[custom_type]
    pub priority: Nullable<Priority>,
}
}

When Nullable::Null, the value is stored as Value::Null. When Nullable::Value(v), it is stored as Value::Custom(...).

Filtering and Querying

To filter on custom type fields, construct a Value::Custom with the appropriate CustomValue:

#![allow(unused)]
fn main() {
use wasm_dbms_api::prelude::*;

// Create a filter for Priority::High
let high_priority = Priority::High;
let filter = Filter::eq("priority", high_priority.into());

// You can also construct the Value manually
let filter = Filter::eq("priority", Value::Custom(CustomValue {
    type_tag: "priority".to_string(),
    encoded: Encode::encode(&Priority::High).into_owned(),
    display: "high".to_string(),
}));
}

To extract a custom type from a Value:

#![allow(unused)]
fn main() {
let value: Value = Priority::High.into();

// Get the raw CustomValue
if let Some(cv) = value.as_custom() {
    println!("type: {}, display: {}", cv.type_tag, cv.display);
}

// Decode into the concrete type
if let Some(priority) = value.as_custom_type::<Priority>() {
    println!("Priority: {priority}");
}
}

Ordering Contract

Custom types support all filter operations: Eq, Ne, In, Gt, Lt, Ge, Le.

For equality filters (Eq, Ne, In), the only requirement is that the Encode implementation produces canonical output – the same value always encodes to the same bytes.

For range filters (Gt, Lt, Ge, Le) and ORDER BY, the encoding must be order-preserving: if a < b according to Ord, then a.encode() < b.encode() lexicographically. This is because the DBMS compares custom values by their encoded bytes.

Example of order-preserving encoding:

The Priority enum above encodes Low = 0, Medium = 1, High = 2. Since Low < Medium < High in the Ord implementation and [0] < [1] < [2] lexicographically, range filters work correctly.

Warning: If your encoding is not order-preserving, equality filters will still work, but range filters and sorting will produce incorrect results.

Examples

Enum: Priority

A complete example of a custom enum type used in a table:

#![allow(unused)]
fn main() {
use std::borrow::Cow;
use std::fmt;

use serde::{Deserialize, Serialize};
use wasm_dbms_api::prelude::*;

// 1. Define the type
#[derive(
    Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord,
    Hash, Default, Serialize, Deserialize,
)]
pub enum Priority {
    #[default]
    Low,
    Medium,
    High,
}

// 2. Implement Display
impl fmt::Display for Priority {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        match self {
            Priority::Low => write!(f, "low"),
            Priority::Medium => write!(f, "medium"),
            Priority::High => write!(f, "high"),
        }
    }
}

// 3. Implement Encode (manual for enums)
impl Encode for Priority {
    const SIZE: DataSize = DataSize::Fixed(1);
    const ALIGNMENT: PageOffset = DEFAULT_ALIGNMENT;

    fn encode(&self) -> Cow<'_, [u8]> {
        Cow::Owned(vec![match self {
            Priority::Low => 0,
            Priority::Medium => 1,
            Priority::High => 2,
        }])
    }

    fn decode(data: Cow<[u8]>) -> MemoryResult<Self> {
        match data[0] {
            0 => Ok(Priority::Low),
            1 => Ok(Priority::Medium),
            2 => Ok(Priority::High),
            other => Err(MemoryError::DecodeError(
                DecodeError::TryFromSliceError(
                    format!("invalid Priority byte: {other}"),
                ),
            )),
        }
    }

    fn size(&self) -> MSize {
        1
    }
}

// 4. Implement DataType + CustomDataType + From<Priority> for Value
impl DataType for Priority {}

impl CustomDataType for Priority {
    const TYPE_TAG: &'static str = "priority";
}

impl From<Priority> for Value {
    fn from(val: Priority) -> Value {
        Value::Custom(CustomValue {
            type_tag: <Priority as CustomDataType>::TYPE_TAG.to_string(),
            encoded: Encode::encode(&val).into_owned(),
            display: val.to_string(),
        })
    }
}

// Use in a table
#[derive(Debug, Table, Clone, PartialEq, Eq)]
#[table = "tasks"]
pub struct Task {
    #[primary_key]
    pub id: Uint32,
    pub title: Text,
    #[custom_type]
    pub priority: Priority,
}
}

Struct: Address

A complete example of a custom struct type. Structs can use #[derive(Encode)] and #[derive(CustomDataType)]:

#![allow(unused)]
fn main() {
use std::fmt;

use serde::{Deserialize, Serialize};
use wasm_dbms_api::prelude::*;

// 1. Define the type with Encode and CustomDataType derives
#[derive(
    Debug, Clone, PartialEq, Eq, PartialOrd, Ord,
    Hash, Default, Serialize, Deserialize,
    Encode, CustomDataType,
)]
#[type_tag = "address"]
pub struct Address {
    pub street: Text,
    pub city: Text,
    pub zip: Text,
}

// 2. Implement Display
impl fmt::Display for Address {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        write!(
            f, "{}, {} {}",
            self.street.as_str(),
            self.city.as_str(),
            self.zip.as_str(),
        )
    }
}

// 3. Implement DataType
impl DataType for Address {}

// Use in a table
#[derive(Debug, Table, Clone, PartialEq, Eq)]
#[table = "customers"]
pub struct Customer {
    #[primary_key]
    pub id: Uint32,
    pub name: Text,
    #[custom_type]
    pub address: Address,
}
}

Schema Migrations

• Schema Migrations
  • Overview
  • When You Need a Migration
  • The Workflow
  • Adding a Column
    • Nullable Columns
    • Non-Nullable Columns with a Static Default
    • Non-Nullable Columns with a Dynamic Default
  • Renaming a Column
  • Changing a Column Type
    • Compatible Widening
    • Custom Transform
  • Dropping a Column or Table
  • Tightening Constraints
  • Adding and Dropping Indexes
  • Running Migrations
    • Generic Backend
    • IC Canister
  • Inspecting Drift Without Migrating
  • Recovering from a Failed Migration
  • Testing Migrations
  • Common Pitfalls

For the full type and API reference (snapshot format, op enum, error variants, IC endpoints), see the Migrations Reference.

Overview

A migration in wasm-dbms is the process of bringing the on-disk data layout into agreement with the schema your binary was compiled against. The framework persists a TableSchemaSnapshot for every table on disk and hashes them into a single schema_hash. On boot, the DBMS recomputes the hash from the compiled schema and compares it. If they differ, the database enters drift state and refuses CRUD until you call migrate(policy).

Migrations are:

• Forward-only. Failed migrations roll back to the pre-migration state, but the framework provides no path from a newer snapshot to an older compiled schema.
• Explicit. The DBMS never auto-migrates on init. The operator decides when (and whether) to run them.
• Atomic. Every op runs inside a single journaled session — either every byte change commits, or none does.
• Pre-flighted. Each plan is validated against the current data before any page is touched. Errors here cost nothing.

When You Need a Migration

Drift fires whenever the encoded snapshot of any compiled table differs from the snapshot stored on disk. In practice, that means any of:

• Adding, removing, or renaming a struct that derives Table.
• Adding, removing, or renaming a field on such a struct.
• Changing a field’s type (e.g. Uint32 → Uint64, or Text → custom enum).
• Toggling #[primary_key], #[unique], #[autoincrement], Nullable<T>, or #[foreign_key(...)].
• Adding or removing an #[index] (single-column or grouped).
• Bumping #[alignment = N].

Never trigger drift:

• Adding #[validate(...)], #[sanitizer(...)], or #[default = ...] on its own (sanitizer/validator are runtime-only; #[default] is migration metadata that lives in the snapshot but is consulted by the planner, not by the drift hash for unrelated changes).
• Reordering doc comments or Debug derives.
• Changing the table’s Rust struct name without changing #[table = "..."].

The Workflow

For most schema changes, the loop is:

1. Edit the schema in your #[derive(Table)] structs.
2. Build and deploy the new binary. On the IC, this is a canister upgrade.
3. Inspect drift. Call dbms.has_drift() (or the has_schema_drift Candid query). Skip if false.
4. Plan. Call dbms.plan_migration() and review the Vec<MigrationOp>.
5. Apply. Call dbms.migrate(policy) once the plan looks right.

The remaining sections walk through the common shapes of step 1 and the policy choices for step 5.

Adding a Column

Nullable Columns

Easiest case. The new column is implicitly NULL for every existing row.

#![allow(unused)]
fn main() {
#[derive(Debug, Table, Clone, PartialEq, Eq)]
#[table = "users"]
pub struct User {
    #[primary_key]
    pub id: Uint32,
    pub name: Text,

    pub bio: Nullable<Text>,   // NEW — no further work needed
}
}

Plan output:

AddColumn { table: "users", column: ColumnSnapshot { name: "bio", nullable: true, default: None, ... } }

migrate(MigrationPolicy::default()) applies it cleanly.

Non-Nullable Columns with a Static Default

If the new column is NOT NULL, the planner needs a default value to backfill existing rows. The cheapest way is the #[default = ...] attribute:

#![allow(unused)]
fn main() {
pub struct User {
    #[primary_key]
    pub id: Uint32,
    pub name: Text,

    #[default = 0]
    pub login_count: Uint32,
}
}

The expression must convert into the column’s Value variant via From/Into. Examples:

#![allow(unused)]
fn main() {
#[default = 0]                                pub login_count: Uint32,
#[default = false]                            pub is_admin: Boolean,
#[default = ""]                               pub locale: Text,
#[default = MyCustomEnum::Default]            pub status: MyCustomEnum,  // requires #[custom_type]
}

Non-Nullable Columns with a Dynamic Default

Sometimes the default depends on runtime context (e.g. derived from another column, or generated by a hash). Mark the table #[migrate] and override Migrate::default_value:

#![allow(unused)]
fn main() {
#[derive(Debug, Table, Clone, PartialEq, Eq)]
#[table = "events"]
#[migrate]
pub struct Event {
    #[primary_key]
    pub id: Uint32,
    pub kind: Text,

    pub severity: Uint8,   // NEW
}

impl Migrate for Event {
    fn default_value(column: &str) -> Option<Value> {
        match column {
            "severity" => Some(Value::Uint8(Uint8(1))),  // medium severity by default
            _ => None,
        }
    }
}
}

Returning None here falls back to the #[default] attribute. Returning None from both produces MigrationError::DefaultMissing.

Note: without #[migrate], the Table macro emits an empty impl Migrate for T {} for you. Adding a hand-written impl on top of it would be a duplicate.

Renaming a Column

A naive rename — change the field name and ship — looks to the planner like a DropColumn followed by an AddColumn. That destroys the data. Use #[renamed_from(...)] to tell the planner the rename history:

#![allow(unused)]
fn main() {
pub struct User {
    #[primary_key]
    pub id: Uint32,

    #[renamed_from("name", "username")]
    pub full_name: Text,
}
}

The planner walks the slice in order: it first looks for a stored column named name; if that misses, it tries username. The first hit emits RenameColumn { old, new: "full_name" } and the column’s data carries over intact.

Multiple renames across releases: keep older entries at the tail. If you renamed username → name in v2 and name → full_name in v3, list ["name", "username"] so a v1-installed canister upgrading directly to v3 still finds its column.

Changing a Column Type

Compatible Widening

The framework auto-widens these without user code:

Just edit the field type and migrate. Plan output is WidenColumn { ... }.

Custom Transform

Anything else — narrowing, sign flip, int↔float, int↔text, custom enum reshape — needs a transform_column impl:

#![allow(unused)]
fn main() {
#[derive(Debug, Table, Clone, PartialEq, Eq)]
#[table = "events"]
#[migrate]
pub struct Event {
    #[primary_key]
    pub id: Uint32,

    pub severity: Uint8,   // was: Text("low" | "medium" | "high")
}

impl Migrate for Event {
    fn default_value(_column: &str) -> Option<Value> { None }

    fn transform_column(column: &str, old: Value) -> DbmsResult<Option<Value>> {
        match column {
            "severity" => match old {
                Value::Text(Text(s)) => match s.as_str() {
                    "low" => Ok(Some(Value::Uint8(Uint8(1)))),
                    "medium" => Ok(Some(Value::Uint8(Uint8(5)))),
                    "high" => Ok(Some(Value::Uint8(Uint8(9)))),
                    other => Err(DbmsError::Migration(MigrationError::TransformAborted {
                        table: "events".into(),
                        column: column.into(),
                        reason: format!("unknown severity `{other}`"),
                    })),
                },
                _ => Ok(None),
            },
            _ => Ok(None),
        }
    }
}
}

Return values:

• Ok(Some(v)) → store v. The planner emits TransformColumn { old_type: Text, new_type: Uint8 }.
• Ok(None) → no transform. The framework errors with MigrationError::IncompatibleType unless a widening already applies.
• Err(_) → abort the migration. The journal rolls back.

Dropping a Column or Table

DropColumn and DropTable are destructive. The default MigrationPolicy::default() refuses them:

#![allow(unused)]
fn main() {
let plan = dbms.plan_migration()?;   // shows DropTable / DropColumn ops
let result = dbms.migrate(MigrationPolicy::default());
// → Err(DbmsError::Migration(MigrationError::DestructiveOpDenied { op: "DropColumn" }))
}

Opt in explicitly:

#![allow(unused)]
fn main() {
dbms.migrate(MigrationPolicy { allow_destructive: true })?;
}

Tip: keep allow_destructive: false in the standard upgrade path and set it to true only when the operator has manually inspected plan_migration() output. A typo in #[table = "..."] looks identical to a deliberate drop in the diff.

Tightening Constraints

A tightening is any AlterColumn change in the restrictive direction:

• nullable: true → nullable: false
• unique: false → unique: true
• adding a #[foreign_key(...)]

Tightenings run after all data rewrites (relaxations, widenings, transforms, adds). The planner validates existing rows against the new constraint at this step. Any violation produces MigrationError::ConstraintViolation { table, column, reason } and rolls back the entire session.

Recommended pattern (split across two releases):

1. Release N — relax + backfill:

   #![allow(unused)]
   fn main() {
   pub email: Nullable<Text>,   // still nullable
   }

   Backfill NULL rows manually or via a one-off update before shipping the next release.

2. Release N+1 — tighten:

   #![allow(unused)]
   fn main() {
   #[unique]
   pub email: Text,             // now NOT NULL + unique
   }

This isolates ConstraintViolation to a release whose cause is obvious.

Adding and Dropping Indexes

Add an #[index] and the planner emits AddIndex. Remove it and you get DropIndex. Composite indexes match by (sorted column list, unique), so changing the group name on a composite index is equivalent to dropping the old one and adding a new one with the same shape.

#![allow(unused)]
fn main() {
pub struct User {
    #[primary_key]
    pub id: Uint32,

    #[index]                                  // NEW
    #[unique]
    pub email: Text,
}
}

Index migrations rebuild the B+ tree from scratch, so they scale O(n log n) with row count.

Running Migrations

Generic Backend

#![allow(unused)]
fn main() {
use wasm_dbms::prelude::*;
use wasm_dbms_api::prelude::MigrationPolicy;

fn boot(mut dbms: Dbms<...>) -> DbmsResult<()> {
    if dbms.has_drift() {
        let plan = dbms.plan_migration()?;
        eprintln!("schema drift detected, applying {} ops", plan.len());
        for op in &plan {
            eprintln!("  {op:?}");
        }
        dbms.migrate(MigrationPolicy::default())?;
    }
    Ok(())
}
}

migrate is idempotent: when there is no drift, it is a no-op.

IC Canister

The #[derive(DbmsCanister)] macro emits three admin-gated endpoints:

has_schema_drift : () -> (bool) query;
plan_migration  : () -> (Result_Vec_MigrationOp);
migrate         : (MigrationPolicy) -> (Result);

Wire them into your post_upgrade hook so that an upgrade automatically heals drift, gated on operator confirmation:

#![allow(unused)]
fn main() {
#[ic_cdk::post_upgrade]
fn post_upgrade() {
    DBMS_CONTEXT.with(|ctx| {
        // Inspect drift and decide whether to auto-migrate. For
        // safety the framework refuses destructive ops by default.
        let mut db = WasmDbmsDatabase::oneshot(ctx, MyDbmsCanister);
        if db.has_drift() {
            db.migrate(MigrationPolicy::default())
                .expect("migration failed");
        }
    });
}
}

Or, for stricter control, leave the canister in drift state after upgrade and run migrate from a tooling script after operator review.

Inspecting Drift Without Migrating

plan_migration() is safe to call regardless of drift state and never touches stable memory. Use it to:

• Diff a development branch against production data.
• Generate a changelog entry from MigrationOp Debug output.
• Catch unintended drops in CI before the binary ships.

#![allow(unused)]
fn main() {
let plan = dbms.plan_migration()?;
for op in plan {
    println!("{op:?}");
}
}

Recovering from a Failed Migration

A failed migrate() call rolls back every page touched in the journal session. Stored snapshots, schema_hash, and the in-memory drift flag are not mutated on failure. So after an error:

• The DBMS stays in drift state.
• Stored data is byte-identical to its pre-migration state.
• ACL methods still work.

Recovery is iterative:

1. Read the error variant. IncompatibleType, DefaultMissing, ConstraintViolation, DestructiveOpDenied, and TransformAborted each call out the offending table/column/reason.
2. Fix the cause: add #[default], write a transform_column arm, clean offending rows via ACL-allowed admin endpoints, or relax the policy.
3. Redeploy the binary (or just retry migrate if the fix is data-side, not schema-side).

There is no partial-success state to clean up. Either the plan applied in full or it didn’t apply at all.

Testing Migrations

The migration pipeline is testable end-to-end on the heap memory provider:

1. Register the old schema with a fresh DbmsContext.
2. Insert representative fixtures.
3. Drop the context and reopen it with the new schema (no rebuild, since this is just Rust code).
4. Assert has_drift() == true, inspect plan_migration(), call migrate(policy).
5. Read the rows back and assert the expected post-migration state.

#![allow(unused)]
fn main() {
#[test]
fn renames_preserve_data() {
    // v1 schema: column "name"
    let ctx = DbmsContext::new(HeapMemoryProvider::default());
    SchemaV1::register_tables(&ctx).unwrap();
    let mut db = WasmDbmsDatabase::oneshot(&ctx, SchemaV1);
    db.insert::<UserV1>(/* ... */).unwrap();
    drop(db);

    // v2 schema: column renamed to "full_name"
    let mut db = WasmDbmsDatabase::oneshot(&ctx, SchemaV2);
    assert!(db.has_drift());
    db.migrate(MigrationPolicy::default()).unwrap();

    let users: Vec<UserV2Record> = db.select::<UserV2>(Query::builder().build()).unwrap();
    assert_eq!(users[0].full_name, Some(/* ... */));
}
}

Round-trip the snapshots through Encode::encode / Encode::decode to confirm the wire format hasn’t shifted.

Common Pitfalls

• Renaming without #[renamed_from]. The planner has no way to know your intent; it will emit DropColumn + AddColumn and silently lose data the moment allow_destructive: true is set.
• Adding a non-nullable column without a default. Pre-flight will reject the plan with DefaultMissing. Either provide #[default], override Migrate::default_value, or make the column Nullable<T>.
• Tightening on dirty data. A nullable: false flip after a release that allowed nulls will fail unless every row already satisfies the constraint. Backfill in a prior release.
• Reordering DataTypeSnapshot discriminants. The on-disk format depends on the exact tag bytes. Treat the enum as frozen — new variants take fresh tags, removed ones leave a reserved hole.
• Bumping #[alignment = N]. This changes the on-disk record layout for the table. Until WidenColumn is generalised to handle alignment changes, this requires a manual rewrite. Avoid unless absolutely necessary.
• Calling migrate before register_tables. The drift hash is computed from the registered set. Always register every table that backs a #[derive(Table)] struct in the compiled binary, even if you don’t expect to write to it this release.

Wasmtime Example

• Overview
• How It Works
  • WIT Interface
  • Guest Component
  • Host Binary
• FileMemoryProvider
• Building and Running
  • Prerequisites
  • Build
  • Run
• Extending with Custom Tables
• Key Concepts

This guide explains how to use wasm-dbms with the WebAssembly Component Model (WIT) and Wasmtime. It walks through the example in crates/wasm-dbms/example/.

Overview

The WebAssembly Component Model defines a standard way for WASM modules to expose typed interfaces using WIT (WebAssembly Interface Types). This example shows how to:

1. Define a WIT interface for the wasm-dbms CRUD and transaction API
2. Build a guest WASM component that wraps wasm-dbms behind the WIT interface
3. Run the guest inside a native Wasmtime host

This approach makes wasm-dbms usable from any Component Model host, not just Rust. The WIT contract at /wit/dbms.wit can be consumed by hosts written in Go, Python, JavaScript, or any language with Component Model tooling.

How It Works

WIT Interface

The WIT definition (/wit/dbms.wit) exposes a database interface with these operations:

• select — query rows from a table with optional filter, ordering, limit, and offset
• insert — insert a row into a table, optionally within a transaction
• update — update rows matching a filter, optionally within a transaction
• delete — delete rows matching a filter, optionally within a transaction
• begin-transaction — start a new ACID transaction
• commit / rollback — finalize or abort a transaction

Values are passed as a value variant type that covers booleans, integers, floats, strings, blobs, and null. Filters are JSON-serialized strings matching the wasm_dbms_api::Filter type.

This raw/dynamic API is intentional: WIT cannot express Rust generics or user-defined table schemas, so type safety is enforced inside the guest by the wasm-dbms engine.

Guest Component

The guest (crates/wasm-dbms/example/guest/) compiles to wasm32-wasip2 and exports the WIT database interface. Internally it:

1. Initializes a DbmsContext<FileMemoryProvider> lazily on first call
2. Registers example tables (users, posts) using #[derive(Table)]
3. Converts between WIT variant values and wasm-dbms Value types
4. Dispatches operations through a DatabaseSchema implementation

The bridge layer in lib.rs handles all the type conversions between the WIT boundary and the typed wasm-dbms internals.

Host Binary

The host (crates/wasm-dbms/example/host/) is a native Rust binary using Wasmtime. It:

1. Creates a Wasmtime engine with Component Model enabled
2. Sets up a WASI context with a preopened directory for the database file
3. Loads the guest .wasm component and instantiates it
4. Calls the exported database functions to demonstrate all operations

FileMemoryProvider

The FileMemoryProvider implements the MemoryProvider trait using std::fs file I/O. It provides persistent, file-backed storage so that data survives across invocations.

#![allow(unused)]
fn main() {
use wasm_dbms_memory::prelude::MemoryProvider;

pub struct FileMemoryProvider {
    file: File,       // open file handle
    size: u64,        // current size in bytes
    pages: u64,       // allocated pages (size / PAGE_SIZE)
}
}

Operations:

• grow(n) — extends the file by n × 65536 bytes
• read(offset, buf) — seeks to offset and reads into buffer
• write(offset, buf) — seeks to offset, writes buffer, and flushes

The provider is initialized with a file path relative to the WASI preopened directory (defaults to wasm-dbms.db).

Note: FileMemoryProvider does not handle concurrent access. It assumes single-writer usage.

Building and Running

Prerequisites

• Rust 1.91.1+

• wasm32-wasip2 target:

  rustup target add wasm32-wasip2
  

• just command runner

Build

# Build guest + host
just build_wasm_dbms_example

This compiles the guest to wasm32-wasip2 (producing a WASM component at .artifact/wasm-dbms-example-guest.wasm) and builds the native host binary.

Run

just test_wasm_dbms_example

Or run manually:

cargo run --release -p wasm-dbms-example-host -- .artifact/wasm-dbms-example-guest.wasm

The demo inserts users and posts, queries them with filters and ordering, demonstrates transaction commit (data persists) and rollback (data discarded), then cleans up.

Extending with Custom Tables

To add your own tables to the example:

1. Define the table in guest/src/schema.rs:

   #![allow(unused)]
   fn main() {
   #[derive(Debug, Table, Clone, PartialEq, Eq)]
   #[table = "comments"]
   pub struct Comment {
       #[primary_key]
       pub id: Uint32,
       pub body: Text,
       #[foreign_key(entity = "Post", table = "posts", column = "id")]
       pub post_id: Uint32,
   }
   }

2. Add dispatch arms for "comments" in every method of ExampleDatabaseSchema in guest/src/schema.rs (select, insert, update, delete, validate_insert, validate_update, referenced_tables).

3. Register the table in register_tables():

   #![allow(unused)]
   fn main() {
   ctx.register_table::<Comment>()?;
   }

4. Update the column lookup in table_columns() (guest/src/lib.rs):

   #![allow(unused)]
   fn main() {
   "comments" => Ok(schema::Comment::columns()),
   }

5. Rebuild with just build_wasm_dbms_example.

Key Concepts

Next Steps

• Getting Started — Set up wasm-dbms from scratch with the Database trait
• CRUD Operations — Detailed guide on all database operations
• Transactions — ACID transactions with commit/rollback
• Schema Definition — Complete schema reference

Schema Definition

• Schema Definition
  • Overview
  • Table Definition
    • Required Derives
    • Table Attribute
  • Column Attributes
    • Primary Key
    • Autoincrement
    • Unique
    • Index
    • Foreign Key
    • Custom Type
    • Sanitizer
    • Validate
    • Candid
    • Alignment
  • Migration Attributes
    • Default Value
    • Renamed From
    • Migrate Override
  • Generated Types
    • Record Type
    • InsertRequest Type
    • UpdateRequest Type
    • ForeignFetcher Type
  • Complete Example
  • Best Practices

Overview

wasm-dbms schemas are defined entirely in Rust using derive macros and attributes. Each struct represents a database table, and each field represents a column.

Key concepts:

• Structs with #[derive(Table)] become database tables
• Fields become columns with their types
• Attributes configure primary keys, foreign keys, validation, and more

Warning

The schema snapshot format used for migration detection imposes hard limits on identifier lengths and table shape. Exceeding any of these will cause the snapshot encoder to truncate or panic at runtime:

• Table name: at most 255 bytes (UTF-8).
• Column name: at most 255 bytes (UTF-8). Applies to every column, including the primary key and any column referenced by an index or foreign key.
• Custom data type name: at most 255 bytes (UTF-8).
• Foreign key target (table name and column name): each at most 255 bytes.
• Columns per index: at most 255.
• Columns per table: at most 65,535.
• Indexes per table: at most 65,535.

Pick short, snake_case identifiers. The 255-byte cap is well above any sensible name length, but binary identifiers or non-ASCII text can blow past it faster than expected because the limit is in bytes, not characters.

Table Definition

Required Derives

Every table struct must have these derives:

#![allow(unused)]
fn main() {
use wasm_dbms_api::prelude::*;

#[derive(Debug, Table, Clone, PartialEq, Eq)]
#[table = "users"]
pub struct User {
    #[primary_key]
    pub id: Uint32,
    pub name: Text,
}
}

Note: For IC canister usage, also add CandidType and Deserialize derives plus the #[candid] attribute. See the IC Schema Reference.

Table Attribute

The #[table = "name"] attribute specifies the table name in the database:

#![allow(unused)]
fn main() {
#[derive(Table, ...)]
#[table = "user_accounts"]  // Table name in database
pub struct UserAccount {    // Rust struct name (can differ)
    // ...
}
}

Naming conventions:

• Use snake_case for table names
• Table names should be plural (e.g., users, posts, order_items)
• Keep names short but descriptive

Column Attributes

Primary Key

Every table must have exactly one primary key:

#![allow(unused)]
fn main() {
#[derive(Table, ...)]
#[table = "users"]
pub struct User {
    #[primary_key]
    pub id: Uint32,  // Primary key
    pub name: Text,
}
}

Primary key rules:

• Exactly one field must be marked with #[primary_key]
• Primary keys must be unique across all records
• Primary keys cannot be null
• Common types: Uint32, Uint64, Uuid, Text

UUID as primary key:

#![allow(unused)]
fn main() {
#[derive(Table, ...)]
#[table = "orders"]
pub struct Order {
    #[primary_key]
    pub id: Uuid,  // UUID primary key
    pub total: Decimal,
}
}

Autoincrement

Automatically generate sequential values for a column on insert:

#![allow(unused)]
fn main() {
#[derive(Table, ...)]
#[table = "users"]
pub struct User {
    #[primary_key]
    #[autoincrement]
    pub id: Uint32,  // Automatically assigned 1, 2, 3, ...
    pub name: Text,
}
}

Autoincrement rules:

• Only integer types are supported: Int8, Int16, Int32, Int64, Uint8, Uint16, Uint32, Uint64
• The counter starts at zero and increments by one on each insert
• Each autoincrement column has an independent counter
• Counters persist across canister upgrades (stored in stable memory)
• When the counter reaches the type’s maximum value, inserts return an AutoincrementOverflow error
• Deleted records do not recycle their autoincrement values
• A table can have multiple #[autoincrement] columns

Choosing the right type:

Tip: Uint64 is recommended for most use cases. Only use smaller types when storage space is critical and you are certain the record count will stay within bounds.

Combining with other attributes:

#![allow(unused)]
fn main() {
#[primary_key]
#[autoincrement]
pub id: Uint64,  // Auto-generated unique primary key
}

Unique

Enforce uniqueness on a column:

#![allow(unused)]
fn main() {
#[derive(Table, ...)]
#[table = "users"]
pub struct User {
    #[primary_key]
    pub id: Uint32,

    #[unique]
    pub email: Text,  // Must be unique across all rows

    pub name: Text,
}
}

Unique constraint rules:

• Insert and update operations that would create a duplicate value return a UniqueConstraintViolation error
• Multiple fields in the same table can each be marked #[unique] independently
• A #[unique] field automatically gets a B+ tree index – no separate #[index] annotation is needed
• Primary keys are always unique by definition; you don’t need #[unique] on a #[primary_key] field

Combining with other attributes:

#![allow(unused)]
fn main() {
#[unique]
#[sanitizer(TrimSanitizer)]
#[sanitizer(LowerCaseSanitizer)]
#[validate(EmailValidator)]
pub email: Text,  // Sanitized, validated, then checked for uniqueness
}

Note: Sanitization and validation run before the uniqueness check, so the sanitized value is what gets compared.

Index

Define indexes on columns for faster lookups:

#![allow(unused)]
fn main() {
#[derive(Table, ...)]
#[table = "users"]
pub struct User {
    #[primary_key]
    pub id: Uint32,

    #[index]
    pub email: Text,  // Single-column index

    pub name: Text,
}
}

The primary key is always an implicit index – you don’t need to add #[index] to it.

Composite indexes:

Use group to group multiple fields into a single composite index:

#![allow(unused)]
fn main() {
#[derive(Table, ...)]
#[table = "products"]
pub struct Product {
    #[primary_key]
    pub id: Uint32,

    #[index(group = "category_brand")]
    pub category: Text,

    #[index(group = "category_brand")]
    pub brand: Text,

    pub name: Text,
}
}

Fields sharing the same group name form a composite index, with columns ordered by field declaration order. In the example above, the composite index covers (category, brand).

Syntax variants:

#![allow(unused)]
fn main() {
// Single-column index
#[index]

// Composite index (group multiple fields by name)
#[index(group = "group_name")]
}

Foreign Key

Define relationships between tables:

#![allow(unused)]
fn main() {
#[derive(Table, ...)]
#[table = "posts"]
pub struct Post {
    #[primary_key]
    pub id: Uint32,
    pub title: Text,

    #[foreign_key(entity = "User", table = "users", column = "id")]
    pub author_id: Uint32,
}
}

Attribute parameters:

Nullable foreign key:

#![allow(unused)]
fn main() {
#[foreign_key(entity = "User", table = "users", column = "id")]
pub manager_id: Nullable<Uint32>,  // Can be null
}

Self-referential foreign key:

#![allow(unused)]
fn main() {
#[derive(Table, ...)]
#[table = "categories"]
pub struct Category {
    #[primary_key]
    pub id: Uint32,
    pub name: Text,

    #[foreign_key(entity = "Category", table = "categories", column = "id")]
    pub parent_id: Nullable<Uint32>,
}
}

Custom Type

Mark a field as a user-defined custom data type:

#![allow(unused)]
fn main() {
#[derive(Table, ...)]
#[table = "tasks"]
pub struct Task {
    #[primary_key]
    pub id: Uint32,
    #[custom_type]
    pub priority: Priority,  // User-defined type
}
}

The #[custom_type] attribute tells the Table macro that this field implements the CustomDataType trait. Without it, the macro won’t know how to serialize and deserialize the field.

Nullable custom types:

#![allow(unused)]
fn main() {
#[custom_type]
pub priority: Nullable<Priority>,  // Optional custom type
}

See the Custom Data Types Guide for how to define custom types.

Sanitizer

Apply data transformations before storage:

#![allow(unused)]
fn main() {
#[derive(Table, ...)]
#[table = "users"]
pub struct User {
    #[primary_key]
    pub id: Uint32,

    #[sanitizer(TrimSanitizer)]
    pub name: Text,

    #[sanitizer(LowerCaseSanitizer)]
    #[sanitizer(TrimSanitizer)]
    pub email: Text,

    #[sanitizer(RoundToScaleSanitizer(2))]
    pub balance: Decimal,

    #[sanitizer(ClampSanitizer, min = 0, max = 120)]
    pub age: Uint8,
}
}

Syntax variants:

#![allow(unused)]
fn main() {
// Unit struct (no parameters)
#[sanitizer(TrimSanitizer)]

// Tuple struct (positional parameter)
#[sanitizer(RoundToScaleSanitizer(2))]

// Named fields struct
#[sanitizer(ClampSanitizer, min = 0, max = 100)]
}

See Sanitization Reference for all available sanitizers.

Validate

Add validation rules:

#![allow(unused)]
fn main() {
#[derive(Table, ...)]
#[table = "users"]
pub struct User {
    #[primary_key]
    pub id: Uint32,

    #[validate(MaxStrlenValidator(100))]
    pub name: Text,

    #[validate(EmailValidator)]
    pub email: Text,

    #[validate(UrlValidator)]
    pub website: Nullable<Text>,
}
}

Validation happens after sanitization:

#![allow(unused)]
fn main() {
#[sanitizer(TrimSanitizer)]           // 1. First: trim whitespace
#[validate(MaxStrlenValidator(100))]  // 2. Then: check length
pub name: Text,
}

See Validation Reference for all available validators.

Candid

Enable CandidType and Deserialize derives on generated types:

#![allow(unused)]
fn main() {
#[derive(Debug, Table, CandidType, Deserialize, Clone, PartialEq, Eq)]
#[candid]
#[table = "users"]
pub struct User {
    #[primary_key]
    pub id: Uint32,
    pub name: Text,
}
}

When the #[candid] attribute is present, the Table macro adds candid::CandidType, serde::Serialize, and serde::Deserialize derives to the generated Record, InsertRequest, and UpdateRequest types.

When to use:

• Required for IC canister deployment where types must cross canister boundaries via Candid
• Any context where generated types need Candid serialization

Note: The #[candid] attribute only affects the types generated by the Table macro. You still need to derive CandidType and Deserialize on the table struct itself.

See the IC Schema Reference for full IC integration details.

Alignment

Advanced: Configure memory alignment for dynamic-size tables:

#![allow(unused)]
fn main() {
#[derive(Table, ...)]
#[table = "large_records"]
#[alignment = 64]  // 64-byte alignment
pub struct LargeRecord {
    #[primary_key]
    pub id: Uint32,
    pub data: Text,  // Variable-size field
}
}

When to use:

• Performance tuning for specific access patterns
• Optimizing memory layout for large records

Rules:

• Minimum alignment is 8 bytes for dynamic types
• Default alignment is 32 bytes
• Fixed-size tables ignore this attribute (alignment equals record size)

Caution: Only change alignment if you understand the performance implications.

Migration Attributes

These attributes feed the schema migration subsystem. They produce no runtime behaviour for normal CRUD; the planner only consults them when the compiled schema diverges from the snapshot stored in stable memory.

See the Schema Migrations Guide for the end-to-end flow (drift detection, plan_migration, migrate(policy)).

Default Value

Attach a per-column default that the migration planner uses when adding a non-nullable column to an existing table:

#![allow(unused)]
fn main() {
#[derive(Table, ...)]
#[table = "users"]
pub struct User {
    #[primary_key]
    pub id: Uint32,

    pub name: Text,

    #[default = 0]
    pub login_count: Uint32,
}
}

How it is used:

• When migrate() plans an AddColumn op for a non-nullable column, it pulls the value from #[default = ...] (after first checking Migrate::default_value).
• Without a resolvable default, planning aborts with MigrationError::MissingDefault.

Rules:

• The expression must convert into the column’s Value variant via From/Into. Examples: #[default = 0] on Uint32, #[default = ""] on Text, #[default = false] on Boolean.
• The expression is evaluated at migration time, not at insert time, so it has no effect on regular INSERT calls — those still need an explicit value (or omit the field if nullable).
• Custom data types must implement From<MyType> for Value; the #[derive(CustomDataType)] macro emits this automatically.
• Defaults are persisted into the table’s snapshot (ColumnSnapshot::default), so the planner can compare them across releases.

Combining with nullable:

#![allow(unused)]
fn main() {
// Redundant — nullable columns default to NULL implicitly. Don't write
// #[default] on a Nullable<T> field.
pub bio: Nullable<Text>,
}

Renamed From

Tell the migration planner that a column used to be known by one or more previous names:

#![allow(unused)]
fn main() {
#[derive(Table, ...)]
#[table = "users"]
pub struct User {
    #[primary_key]
    pub id: Uint32,

    #[renamed_from("username", "user_name")]
    pub name: Text,
}
}

How it is used:

When planning a migration, the planner first matches stored columns against compiled columns by name. For each compiled column with no direct match, it walks renamed_from in order and looks for a stored column with one of those names. The first hit is emitted as a RenameColumn op, preserving the column’s data.

Rules:

• Entries are string literals.
• Order matters: list newer renames first, older renames last (mirroring the chronological order of releases).
• A stored column matched by renamed_from is not matched by another compiled column. If two compiled columns claim the same previous name, the earlier-declared field wins.
• Without #[renamed_from], a column rename is indistinguishable from a DropColumn + AddColumn pair, which loses data.

Migrate Override

By default, #[derive(Table)] emits an empty impl Migrate for T {} for every table, giving you trait defaults for default_value and transform_column. Add #[migrate] at the struct level to suppress that emission and provide a hand-written impl:

#![allow(unused)]
fn main() {
#[derive(Table, ...)]
#[table = "events"]
#[migrate]
pub struct Event {
    #[primary_key]
    pub id: Uint32,
    pub kind: Text,
    pub severity: Uint8,
}

impl Migrate for Event {
    fn default_value(column: &str) -> Option<Value> {
        match column {
            "severity" => Some(Value::Uint8(Uint8(1))),
            _ => None,
        }
    }

    fn transform_column(column: &str, old: Value) -> DbmsResult<Option<Value>> {
        match column {
            // Example: convert legacy text severities into the new Uint8 column.
            "severity" => match old {
                Value::Text(Text(s)) => match s.as_str() {
                    "low" => Ok(Some(Value::Uint8(Uint8(1)))),
                    "medium" => Ok(Some(Value::Uint8(Uint8(5)))),
                    "high" => Ok(Some(Value::Uint8(Uint8(9)))),
                    other => Err(DbmsError::Migration(MigrationError::TransformAborted {
                        table: "events".into(),
                        column: column.into(),
                        reason: format!("unknown severity `{other}`"),
                    })),
                },
                _ => Ok(None),
            },
            _ => Ok(None),
        }
    }
}
}

When to use #[migrate]:

• The new column is non-nullable and the default cannot be a constant literal (e.g. requires hashing the row, or pulls from another column).
• A column changed to an incompatible type that is not in the widening whitelist, and you can derive the new value from the old one.

Trait contract:

Note: Without #[migrate], do not write impl Migrate for T {} yourself — the macro already emitted one and you would get a duplicate-impl error.

Generated Types

The Table macro generates several types for each table.

Record Type

{StructName}Record - The full record type returned from queries:

#![allow(unused)]
fn main() {
// Generated from User struct
pub struct UserRecord {
    pub id: Uint32,
    pub name: Text,
    pub email: Text,
}

// Usage
let users: Vec<UserRecord> = database.select::<User>(query)?;

for user in users {
    println!("{}: {}", user.id, user.name);
}
}

InsertRequest Type

{StructName}InsertRequest - Request type for inserting records:

#![allow(unused)]
fn main() {
// Generated from User struct
pub struct UserInsertRequest {
    pub id: Uint32,
    pub name: Text,
    pub email: Text,
}

// Usage
let user = UserInsertRequest {
    id: 1.into(),
    name: "Alice".into(),
    email: "alice@example.com".into(),
};

database.insert::<User>(user)?;
}

UpdateRequest Type

{StructName}UpdateRequest - Request type for updating records:

#![allow(unused)]
fn main() {
// Generated from User struct (with builder pattern)
let update = UserUpdateRequest::builder()
    .set_name("New Name".into())
    .set_email("new@example.com".into())
    .filter(Filter::eq("id", Value::Uint32(1.into())))
    .build();

// Usage
database.update::<User>(update)?;
}

Builder methods:

• set_{field_name}(value) - Set a field value
• filter(Filter) - WHERE clause (required)
• build() - Build the update request

ForeignFetcher Type

{StructName}ForeignFetcher - Internal type for eager loading:

#![allow(unused)]
fn main() {
// Generated automatically, used internally
// You typically don't interact with this directly
}

Complete Example

#![allow(unused)]
fn main() {
// schema/src/lib.rs
use wasm_dbms_api::prelude::*;

#[derive(Debug, Table, Clone, PartialEq, Eq)]
#[table = "users"]
pub struct User {
    #[primary_key]
    pub id: Uint32,

    #[sanitizer(TrimSanitizer)]
    #[validate(MaxStrlenValidator(100))]
    pub name: Text,

    #[unique]
    #[sanitizer(TrimSanitizer)]
    #[sanitizer(LowerCaseSanitizer)]
    #[validate(EmailValidator)]
    pub email: Text,

    pub created_at: DateTime,

    pub is_active: Boolean,
}

#[derive(Debug, Table, Clone, PartialEq, Eq)]
#[table = "posts"]
pub struct Post {
    #[primary_key]
    pub id: Uuid,

    #[validate(MaxStrlenValidator(200))]
    pub title: Text,

    pub content: Text,

    pub published: Boolean,

    #[index(group = "author_date")]
    #[foreign_key(entity = "User", table = "users", column = "id")]
    pub author_id: Uint32,

    pub metadata: Nullable<Json>,

    #[index(group = "author_date")]
    pub created_at: DateTime,
}

#[derive(Debug, Table, Clone, PartialEq, Eq)]
#[table = "comments"]
pub struct Comment {
    #[primary_key]
    pub id: Uuid,

    #[validate(MaxStrlenValidator(1000))]
    pub content: Text,

    #[foreign_key(entity = "User", table = "users", column = "id")]
    pub author_id: Uint32,

    #[foreign_key(entity = "Post", table = "posts", column = "id")]
    pub post_id: Uuid,

    pub created_at: DateTime,
}
}

For generating a complete IC canister API from this schema, see the IC Schema Reference.

Best Practices

1. Keep schema in a separate crate

my-project/
├── schema/           # Reusable types
│   ├── Cargo.toml
│   └── src/lib.rs
└── app/              # Application using the database
    ├── Cargo.toml
    └── src/lib.rs

2. Use appropriate primary key types

#![allow(unused)]
fn main() {
// Sequential IDs - simple, good for internal use
pub id: Uint32,

// UUIDs - better for distributed systems, no guessing
pub id: Uuid,
}

3. Always validate user input

#![allow(unused)]
fn main() {
#[validate(MaxStrlenValidator(1000))]  // Prevent huge strings
pub content: Text,

#[validate(EmailValidator)]  // Validate format
pub email: Text,
}

4. Use nullable for optional fields

#![allow(unused)]
fn main() {
pub phone: Nullable<Text>,  // Clearly optional
pub bio: Nullable<Text>,
}

5. Consider sanitization for consistency

#![allow(unused)]
fn main() {
#[sanitizer(TrimSanitizer)]
#[sanitizer(LowerCaseSanitizer)]
pub email: Text,  // Always lowercase, no whitespace
}

6. Document your schema

#![allow(unused)]
fn main() {
/// User account information
#[derive(Table, ...)]
#[table = "users"]
pub struct User {
    /// Unique user identifier
    #[primary_key]
    pub id: Uint32,

    /// User's display name (max 100 chars)
    #[validate(MaxStrlenValidator(100))]
    pub name: Text,
}
}

Data Types

• Data Types
  • Overview
  • Integer Types
    • Unsigned Integers
    • Signed Integers
  • Decimal
  • Text
  • Boolean
  • Date and Time
    • Date
    • DateTime
  • Binary Data
    • Blob
  • Identifiers
    • Uuid
  • Semi-Structured Data
    • Json
  • Nullable
  • Custom Types
  • Type Conversion Reference

Overview

wasm-dbms provides a rich set of data types for defining table schemas. Each type maps to standard Rust types for seamless integration.

Type categories:

Note: The Principal type is available in ic-dbms-api for Internet Computer integration. See the IC Data Types reference for details.

Integer Types

Unsigned Integers

Uint8 - 8-bit unsigned integer (0 to 255)

#![allow(unused)]
fn main() {
use wasm_dbms_api::prelude::Uint8;

#[derive(Table, ...)]
#[table = "settings"]
pub struct Setting {
    #[primary_key]
    pub id: Uint32,
    pub priority: Uint8,  // 0-255
}

// Usage
let setting = SettingInsertRequest {
    id: 1.into(),
    priority: 10.into(),  // or Uint8::from(10)
};
}

Uint16 - 16-bit unsigned integer (0 to 65,535)

#![allow(unused)]
fn main() {
use wasm_dbms_api::prelude::Uint16;

pub struct Product {
    pub stock_count: Uint16,  // 0-65,535
}

let count: Uint16 = 1000.into();
}

Uint32 - 32-bit unsigned integer (0 to 4,294,967,295)

#![allow(unused)]
fn main() {
use wasm_dbms_api::prelude::Uint32;

pub struct User {
    #[primary_key]
    pub id: Uint32,  // Common for primary keys
}

let id: Uint32 = 12345.into();
}

Uint64 - 64-bit unsigned integer (0 to 18,446,744,073,709,551,615)

#![allow(unused)]
fn main() {
use wasm_dbms_api::prelude::Uint64;

pub struct Transaction {
    pub amount_e8s: Uint64,  // For large numbers like token amounts
}

let amount: Uint64 = 1_000_000_000u64.into();
}

Signed Integers

Int8 - 8-bit signed integer (-128 to 127)

#![allow(unused)]
fn main() {
use wasm_dbms_api::prelude::Int8;

pub struct Temperature {
    pub celsius: Int8,  // -128 to 127
}

let temp: Int8 = (-10).into();
}

Int16 - 16-bit signed integer (-32,768 to 32,767)

#![allow(unused)]
fn main() {
use wasm_dbms_api::prelude::Int16;

pub struct Altitude {
    pub meters: Int16,  // Can be negative (below sea level)
}

let altitude: Int16 = (-100).into();
}

Int32 - 32-bit signed integer (-2,147,483,648 to 2,147,483,647)

#![allow(unused)]
fn main() {
use wasm_dbms_api::prelude::Int32;

pub struct Account {
    pub balance_cents: Int32,  // Can be negative (debt)
}

let balance: Int32 = (-5000).into();
}

Int64 - 64-bit signed integer

#![allow(unused)]
fn main() {
use wasm_dbms_api::prelude::Int64;

pub struct Statistics {
    pub total_change: Int64,  // Large signed values
}

let change: Int64 = (-1_000_000_000i64).into();
}

Decimal

Decimal - Arbitrary-precision decimal number

#![allow(unused)]
fn main() {
use wasm_dbms_api::prelude::Decimal;

pub struct Product {
    pub price: Decimal,      // $19.99
    pub weight_kg: Decimal,  // 2.5
}

// From f64
let price: Decimal = 19.99.into();

// From string (more precise)
let price: Decimal = "19.99".parse().unwrap();

// With sanitizer for rounding
#[derive(Table, ...)]
#[table = "products"]
pub struct Product {
    #[primary_key]
    pub id: Uint32,
    #[sanitizer(RoundToScaleSanitizer(2))]  // Round to 2 decimal places
    pub price: Decimal,
}
}

Note: Use RoundToScaleSanitizer to ensure consistent decimal precision.

Text

Text - UTF-8 string

#![allow(unused)]
fn main() {
use wasm_dbms_api::prelude::Text;

pub struct User {
    pub name: Text,
    pub email: Text,
    pub bio: Text,
}

// From &str
let name: Text = "Alice".into();

// From String
let email: Text = String::from("alice@example.com").into();

// Access the string
let text: Text = "Hello".into();
assert_eq!(text.as_str(), "Hello");
}

With validation:

#![allow(unused)]
fn main() {
#[derive(Table, ...)]
#[table = "users"]
pub struct User {
    #[primary_key]
    pub id: Uint32,
    #[validate(MaxStrlenValidator(100))]
    pub name: Text,
    #[validate(EmailValidator)]
    pub email: Text,
}
}

With sanitization:

#![allow(unused)]
fn main() {
#[derive(Table, ...)]
#[table = "users"]
pub struct User {
    #[primary_key]
    pub id: Uint32,
    #[sanitizer(TrimSanitizer)]
    #[sanitizer(CollapseWhitespaceSanitizer)]
    pub name: Text,
    #[sanitizer(LowerCaseSanitizer)]
    pub email: Text,
}
}

Boolean

Boolean - True or false value

#![allow(unused)]
fn main() {
use wasm_dbms_api::prelude::Boolean;

pub struct User {
    pub is_active: Boolean,
    pub email_verified: Boolean,
}

let active: Boolean = true.into();
let verified: Boolean = false.into();

// Convert back
let value: bool = active.into();
}

Filtering by boolean:

#![allow(unused)]
fn main() {
// Find active users
let filter = Filter::eq("is_active", Value::Boolean(true));

// Find unverified users
let filter = Filter::eq("email_verified", Value::Boolean(false));
}

Date and Time

Date

Date - Calendar date (year, month, day)

#![allow(unused)]
fn main() {
use wasm_dbms_api::prelude::Date;

pub struct Event {
    pub event_date: Date,
}

// Create from components
let date = Date::new(2024, 6, 15);  // June 15, 2024

// From chrono NaiveDate (if using chrono)
use chrono::NaiveDate;
let naive = NaiveDate::from_ymd_opt(2024, 6, 15).unwrap();
let date: Date = naive.into();
}

DateTime

DateTime - Date and time with timezone

#![allow(unused)]
fn main() {
use wasm_dbms_api::prelude::DateTime;

pub struct User {
    pub created_at: DateTime,
    pub last_login: DateTime,
}

// Current time
let now = DateTime::now();

// From chrono DateTime<Utc>
use chrono::{DateTime as ChronoDateTime, Utc};
let chrono_dt: ChronoDateTime<Utc> = Utc::now();
let dt: DateTime = chrono_dt.into();
}

With sanitization:

#![allow(unused)]
fn main() {
#[derive(Table, ...)]
#[table = "events"]
pub struct Event {
    #[primary_key]
    pub id: Uint32,
    #[sanitizer(UtcSanitizer)]  // Convert to UTC
    pub scheduled_at: DateTime,
}
}

Binary Data

Blob

Blob - Binary large object (byte array)

#![allow(unused)]
fn main() {
use wasm_dbms_api::prelude::Blob;

pub struct Document {
    pub content: Blob,      // File content
    pub thumbnail: Blob,    // Image data
}

// From Vec<u8>
let data: Vec<u8> = vec![0x89, 0x50, 0x4E, 0x47];  // PNG header
let blob: Blob = data.into();

// From slice
let blob: Blob = Blob::from(&[1, 2, 3, 4][..]);

// Access bytes
let bytes: &[u8] = blob.as_slice();
}

Note: Be mindful of storage costs when storing large blobs. Consider storing only references (hashes, URLs) for very large files.

Identifiers

Uuid

Uuid - Universally unique identifier (128-bit)

#![allow(unused)]
fn main() {
use wasm_dbms_api::prelude::Uuid;

pub struct Order {
    #[primary_key]
    pub id: Uuid,  // UUID as primary key
}

// Generate new UUID
let id = Uuid::new_v4();

// From string
let id: Uuid = "550e8400-e29b-41d4-a716-446655440000".parse().unwrap();

// From bytes
let bytes: [u8; 16] = [/* 16 bytes */];
let id = Uuid::from_bytes(bytes);
}

Benefits over sequential IDs:

• Globally unique without coordination
• No sequential guessing
• Safe for distributed systems

Semi-Structured Data

Json

Json - JSON object or array

#![allow(unused)]
fn main() {
use wasm_dbms_api::prelude::Json;
use std::str::FromStr;

pub struct User {
    pub metadata: Json,    // Flexible schema
    pub preferences: Json, // User settings
}

// From string
let json = Json::from_str(r#"{"theme": "dark", "language": "en"}"#).unwrap();

// From serde_json::Value
use serde_json::json;
let json: Json = json!({
    "notifications": true,
    "timezone": "UTC"
}).into();
}

Querying JSON:

#![allow(unused)]
fn main() {
// Check if JSON contains pattern
let filter = Filter::json("metadata", JsonFilter::contains(
    Json::from_str(r#"{"active": true}"#).unwrap()
));

// Extract and compare
let filter = Filter::json("preferences",
    JsonFilter::extract_eq("theme", Value::Text("dark".into()))
);

// Check path exists
let filter = Filter::json("metadata", JsonFilter::has_key("email"));
}

See the JSON Reference for comprehensive JSON documentation.

Nullable

Nullable<T> - Optional value wrapper

#![allow(unused)]
fn main() {
use wasm_dbms_api::prelude::{Nullable, Text, Uint32};

pub struct User {
    #[primary_key]
    pub id: Uint32,
    pub name: Text,
    pub phone: Nullable<Text>,      // Optional phone number
    pub age: Nullable<Uint32>,      // Optional age
    pub bio: Nullable<Text>,        // Optional biography
}

// Insert with value
let user = UserInsertRequest {
    id: 1.into(),
    name: "Alice".into(),
    phone: Nullable::Value("555-1234".into()),
    age: Nullable::Null,
    bio: Nullable::Null,
};

// Check if null
let phone = user.phone;
match phone {
    Nullable::Value(p) => println!("Phone: {}", p.as_str()),
    Nullable::Null => println!("No phone number"),
}
}

Filtering nullable fields:

#![allow(unused)]
fn main() {
// Find users with phone numbers
let filter = Filter::not_null("phone");

// Find users without phone numbers
let filter = Filter::is_null("phone");

// Find users with specific phone
let filter = Filter::eq("phone", Value::Text("555-1234".into()));
}

Nullable foreign keys:

#![allow(unused)]
fn main() {
pub struct Employee {
    #[primary_key]
    pub id: Uint32,
    pub name: Text,
    #[foreign_key(entity = "Employee", table = "employees", column = "id")]
    pub manager_id: Nullable<Uint32>,  // Top-level employees have no manager
}
}

Custom Types

Beyond the built-in types listed above, wasm-dbms supports user-defined custom data types. Custom types let you store enums, structs, and newtypes in your tables by implementing the CustomDataType trait.

#![allow(unused)]
fn main() {
use wasm_dbms_api::prelude::*;

#[derive(Debug, Table, Clone, PartialEq, Eq)]
#[table = "tasks"]
pub struct Task {
    #[primary_key]
    pub id: Uint32,
    #[custom_type]
    pub priority: Priority,  // User-defined custom type
}
}

See the Custom Data Types Guide for step-by-step instructions on defining and using custom types.

Type Conversion Reference

Note: For IC canister usage, these types also map to Candid types. See the IC Data Types reference for the Candid mapping.

Conversion examples:

#![allow(unused)]
fn main() {
// Rust primitive to wasm-dbms type
let uint: Uint32 = 42u32.into();
let text: Text = "hello".into();
let boolean: Boolean = true.into();

// wasm-dbms type to Rust primitive
let num: u32 = uint.into();
let s: String = text.into();
let b: bool = boolean.into();
}

Query API Reference

• Query API Reference
  • Overview
  • Query Struct
  • QueryBuilder
    • Field Selection
    • Filters
    • Joins
    • Eager Loading
    • Distinct
    • Aggregations
    • Ordering
    • Pagination
  • Aggregate Types
    • AggregateFunction
    • AggregatedRow
    • AggregatedValue
  • Execution Order
  • Errors
  • Related Types

Overview

A Query describes what to retrieve from the database: which rows match, which columns to return, how to order and paginate them, and how to combine data across tables. Queries are constructed with QueryBuilder and consumed by Database::select, Database::select_raw, and Database::select_join.

For an introductory walkthrough, see the Querying Guide.

Query Struct

#![allow(unused)]
fn main() {
pub struct Query {
    columns: Select,
    pub distinct_by: Vec<String>,
    pub eager_relations: Vec<String>,
    pub filter: Option<Filter>,
    pub group_by: Vec<String>,
    pub having: Option<Filter>,
    pub joins: Vec<Join>,
    pub limit: Option<usize>,
    pub offset: Option<usize>,
    pub order_by: Vec<(String, OrderDirection)>,
}
}

Use Query::builder() to obtain a QueryBuilder.

QueryBuilder

Field Selection

The primary key is always included by Database::select::<T> even when not explicitly listed.

Filters

See Filters in the Querying Guide and JSON Filters for the full filter API.

Joins

Queries containing joins must be executed via Database::select_join. Calling Database::select::<T> with a joined query returns QueryError::JoinInsideTypedSelect.

Eager Loading

#![allow(unused)]
fn main() {
.with("posts")
}

Adds a foreign-key relation to load eagerly. Each relation is loaded once via a batch fetch keyed by the foreign-key column.

Distinct

#![allow(unused)]
fn main() {
.distinct(&["name"])
.distinct(&["category", "vendor"])
}

Sets distinct_by to the supplied list of column names. Rows are deduplicated by the tuple of values across those columns; the first row encountered for each distinct tuple is retained. Passing an empty slice is a no-op.

Semantics:

• Columns are looked up on the source record (ValuesSource::This).
• Missing columns are treated as Value::Null, so listing an unknown column collapses every row into a single result.
• Deduplication runs before ordering, offset, and limit.
• The selected fields (Select::Columns) do not need to include the distinct_by columns.

Aggregations

#![allow(unused)]
fn main() {
.group_by(&["category"])
.having(Filter::gt("count", Value::Uint64(10u64.into())))
}

Aggregations operate over the rows that survive WHERE and DISTINCT. Each group of rows sharing the same group_by tuple produces one AggregatedRow. The aggregate functions to compute are described by AggregateFunction; their results are returned as AggregatedValue entries inside the row.

The HAVING filter is evaluated after aggregation, against the grouping keys and aggregate results.

Ordering

Multiple order_by_* calls produce stable multi-key sorts; later keys break ties from earlier keys.

Pagination

Aggregate Types

Types used to describe and return aggregated query results. All three are re-exported from the wasm-dbms-api, ic-dbms-api, and ic-dbms-client preludes.

AggregateFunction

#![allow(unused)]
fn main() {
pub enum AggregateFunction {
    Count(Option<String>),
    Sum(String),
    Avg(String),
    Min(String),
    Max(String),
}
}

Describes one aggregate function to compute over a group of rows.

AggregatedRow

#![allow(unused)]
fn main() {
pub struct AggregatedRow {
    pub group_keys: Vec<Value>,
    pub values: Vec<AggregatedValue>,
}
}

A single row of aggregated output. group_keys holds the values of the group_by columns that identify the group; values holds the aggregate results in the same order as the AggregateFunction list supplied with the query.

AggregatedValue

#![allow(unused)]
fn main() {
pub enum AggregatedValue {
    Count(u64),
    Sum(Value),
    Avg(Value),
    Min(Value),
    Max(Value),
}
}

Carries the result of one aggregate function. Count is always a u64; the remaining variants wrap a Value whose concrete variant depends on the source column’s data type.

Execution Order

The select pipeline applies the query elements in this order — matching standard SQL semantics:

1. WHERE — filter is applied while scanning records (or via an index plan).
2. DISTINCT — distinct_by deduplicates the surviving rows.
3. GROUP BY / aggregates — when group_by is set, surviving rows are bucketed by the grouping tuple and the requested AggregateFunctions are computed per bucket, producing AggregatedRows.
4. HAVING — having filters the aggregated groups.
5. Eager loading — relations declared by with(...) are batch-fetched (non-aggregate selects only).
6. Column selection — non-selected columns are dropped from each row.
7. ORDER BY — order_by keys are applied in declared order.
8. OFFSET / LIMIT — applied last when order_by or distinct_by is set; otherwise applied during the scan for early termination.

When neither order_by nor distinct_by is present, the engine applies offset/limit during iteration to avoid materialising the entire result set.

Errors

All variants come from QueryError. Most are surfaced at planning time (before any rows are scanned) so callers fail fast.

Aggregate-specific (Database::aggregate)

Non-aggregate select paths

Related Types

• AggregateFunction — aggregate to compute per group
• AggregatedRow — single row of aggregated output
• AggregatedValue — single aggregate result value
• Filter — predicates for WHERE
• JsonFilter — JSON-specific predicates
• Join, JoinType — join clauses
• OrderDirection — ascending/descending
• Select — All or Columns(...)
• QueryError — query-time error variants

Validation Reference

• Validation Reference
  • Overview
  • Syntax
  • Built-in Validators
    • String Length Validators
    • Format Validators
    • Case Validators
    • Locale Validators
  • Implementing Custom Validators
  • Validation Errors
  • Examples

Overview

Validators enforce constraints on data being inserted or updated. If validation fails, the operation is rejected with a Validation error.

Key points:

• Validators run after sanitizers
• Validation failure rejects the entire operation
• Multiple validators can be applied to a single field
• Validators are applied on both insert and update

Syntax

The #[validate(...)] attribute adds validation rules to fields:

#![allow(unused)]
fn main() {
use wasm_dbms_api::prelude::*;

#[derive(Table, ...)]
#[table = "users"]
pub struct User {
    #[primary_key]
    pub id: Uint32,

    // Unit struct validator (no parameters)
    #[validate(EmailValidator)]
    pub email: Text,

    // Tuple struct validator (positional parameter)
    #[validate(MaxStrlenValidator(100))]
    pub name: Text,
}
}

Built-in Validators

All validators are available in wasm_dbms_api::prelude.

String Length Validators

MaxStrlenValidator - Maximum string length

#![allow(unused)]
fn main() {
#[validate(MaxStrlenValidator(255))]
pub description: Text,  // Max 255 characters
}

MinStrlenValidator - Minimum string length

#![allow(unused)]
fn main() {
#[validate(MinStrlenValidator(8))]
pub password: Text,  // At least 8 characters
}

RangeStrlenValidator - String length within range

#![allow(unused)]
fn main() {
#[validate(RangeStrlenValidator(3, 50))]
pub username: Text,  // Between 3 and 50 characters
}

Format Validators

EmailValidator - Valid email format

#![allow(unused)]
fn main() {
#[validate(EmailValidator)]
pub email: Text,  // Must be valid email
}

UrlValidator - Valid URL format

#![allow(unused)]
fn main() {
#[validate(UrlValidator)]
pub website: Text,  // Must be valid URL
}

PhoneNumberValidator - Valid phone number format

#![allow(unused)]
fn main() {
#[validate(PhoneNumberValidator)]
pub phone: Text,  // Must be valid phone number
}

MimeTypeValidator - Valid MIME type format

#![allow(unused)]
fn main() {
#[validate(MimeTypeValidator)]
pub content_type: Text,  // e.g., "application/json", "image/png"
}

RgbColorValidator - Valid RGB color format

#![allow(unused)]
fn main() {
#[validate(RgbColorValidator)]
pub color: Text,  // e.g., "#FF5733", "rgb(255, 87, 51)"
}

Case Validators

CamelCaseValidator - Must be camelCase

#![allow(unused)]
fn main() {
#[validate(CamelCaseValidator)]
pub identifier: Text,  // e.g., "myVariableName"
}

KebabCaseValidator - Must be kebab-case

#![allow(unused)]
fn main() {
#[validate(KebabCaseValidator)]
pub slug: Text,  // e.g., "my-page-slug"
}

SnakeCaseValidator - Must be snake_case

#![allow(unused)]
fn main() {
#[validate(SnakeCaseValidator)]
pub code: Text,  // e.g., "my_constant_name"
}

Locale Validators

CountryIso639Validator - ISO 639 language code

#![allow(unused)]
fn main() {
#[validate(CountryIso639Validator)]
pub language: Text,  // e.g., "en", "es", "fr"
}

CountryIso3166Validator - ISO 3166 country code

#![allow(unused)]
fn main() {
#[validate(CountryIso3166Validator)]
pub country: Text,  // e.g., "US", "GB", "DE"
}

Implementing Custom Validators

Create a struct implementing the Validate trait:

#![allow(unused)]
fn main() {
use wasm_dbms_api::prelude::{Validate, Value, DbmsResult, DbmsError};

/// Validates that a number is positive
pub struct PositiveValidator;

impl Validate for PositiveValidator {
    fn validate(&self, value: &Value) -> DbmsResult<()> {
        match value {
            Value::Int32(n) if n.0 > 0 => Ok(()),
            Value::Int64(n) if n.0 > 0 => Ok(()),
            Value::Decimal(d) if d.0 > rust_decimal::Decimal::ZERO => Ok(()),
            Value::Int32(_) | Value::Int64(_) | Value::Decimal(_) => {
                Err(DbmsError::Validation("Value must be positive".to_string()))
            }
            _ => Err(DbmsError::Validation("PositiveValidator only applies to numeric types".to_string()))
        }
    }
}

// Usage
#[derive(Table, ...)]
#[table = "products"]
pub struct Product {
    #[primary_key]
    pub id: Uint32,
    #[validate(PositiveValidator)]
    pub price: Decimal,
}
}

Custom validator with parameters (tuple struct):

#![allow(unused)]
fn main() {
/// Validates that a string matches a regex pattern
pub struct RegexValidator(pub &'static str);

impl Validate for RegexValidator {
    fn validate(&self, value: &Value) -> DbmsResult<()> {
        if let Value::Text(text) = value {
            let re = regex::Regex::new(self.0).unwrap();
            if re.is_match(text.as_str()) {
                return Ok(());
            }
        }
        Err(DbmsError::Validation(
            format!("Value does not match pattern: {}", self.0)
        ))
    }
}

// Usage
#[validate(RegexValidator(r"^[A-Z]{2}-\d{4}$"))]
pub product_code: Text,  // Must match "XX-1234" format
}

Custom validator with named parameters:

#![allow(unused)]
fn main() {
/// Validates a number is within a range
pub struct RangeValidator {
    pub min: i64,
    pub max: i64,
}

impl Validate for RangeValidator {
    fn validate(&self, value: &Value) -> DbmsResult<()> {
        let num = match value {
            Value::Int32(n) => n.0 as i64,
            Value::Int64(n) => n.0,
            _ => return Err(DbmsError::Validation("RangeValidator requires integer".to_string())),
        };

        if num >= self.min && num <= self.max {
            Ok(())
        } else {
            Err(DbmsError::Validation(
                format!("Value must be between {} and {}", self.min, self.max)
            ))
        }
    }
}

// Usage
#[validate(RangeValidator, min = 1, max = 100)]
pub percentage: Int32,
}

Validation Errors

When validation fails, a DbmsError::Validation(String) is returned:

#![allow(unused)]
fn main() {
let result = database.insert::<User>(user);

match result {
    Ok(()) => println!("Insert successful"),
    Err(DbmsError::Validation(msg)) => {
        println!("Validation failed: {}", msg);
        // e.g., "Invalid email format"
        // e.g., "String length exceeds maximum of 100"
    }
    Err(e) => println!("Other error: {:?}", e),
}
}

Examples

Comprehensive user validation:

#![allow(unused)]
fn main() {
#[derive(Debug, Table, Clone, PartialEq, Eq)]
#[table = "users"]
pub struct User {
    #[primary_key]
    pub id: Uint32,

    #[validate(RangeStrlenValidator(2, 50))]
    pub name: Text,

    #[validate(EmailValidator)]
    pub email: Text,

    #[validate(MinStrlenValidator(8))]
    pub password_hash: Text,

    #[validate(PhoneNumberValidator)]
    pub phone: Nullable<Text>,

    #[validate(UrlValidator)]
    pub website: Nullable<Text>,

    #[validate(CountryIso3166Validator)]
    pub country: Nullable<Text>,

    #[validate(CountryIso639Validator)]
    pub language: Text,
}
}

Product validation:

#![allow(unused)]
fn main() {
#[derive(Debug, Table, Clone, PartialEq, Eq)]
#[table = "products"]
pub struct Product {
    #[primary_key]
    pub id: Uuid,

    #[validate(RangeStrlenValidator(1, 200))]
    pub name: Text,

    #[validate(MaxStrlenValidator(2000))]
    pub description: Text,

    #[validate(KebabCaseValidator)]
    pub slug: Text,

    #[validate(MimeTypeValidator)]
    pub image_type: Nullable<Text>,

    #[validate(RgbColorValidator)]
    pub accent_color: Nullable<Text>,
}
}

Combined with sanitizers:

#![allow(unused)]
fn main() {
#[derive(Debug, Table, Clone, PartialEq, Eq)]
#[table = "articles"]
pub struct Article {
    #[primary_key]
    pub id: Uuid,

    // Sanitize first, then validate
    #[sanitizer(TrimSanitizer)]
    #[validate(MaxStrlenValidator(200))]
    pub title: Text,

    // Convert to slug format, then validate
    #[sanitizer(SlugSanitizer)]
    #[validate(KebabCaseValidator)]
    pub slug: Text,

    #[sanitizer(TrimSanitizer)]
    pub content: Text,
}
}

Sanitization Reference

• Overview
• Syntax
• Built-in Sanitizers
  • String Sanitizers
  • Numeric Sanitizers
  • DateTime Sanitizers
  • Null Sanitizers
• Implementing Custom Sanitizers
• Sanitization Order
• Examples

Overview

Sanitizers automatically transform data before it’s stored in the database. Unlike validators (which reject invalid data), sanitizers modify data to conform to expected formats.

Key points:

• Sanitizers run before validators
• Data is transformed, not rejected
• Multiple sanitizers can be chained
• Sanitizers apply on both insert and update

Syntax

The #[sanitizer(...)] attribute adds sanitization rules to fields:

#![allow(unused)]
fn main() {
use wasm_dbms_api::prelude::*;

#[derive(Table, ...)]
#[table = "users"]
pub struct User {
    #[primary_key]
    pub id: Uint32,

    // Unit struct sanitizer (no parameters)
    #[sanitizer(TrimSanitizer)]
    pub name: Text,

    // Tuple struct sanitizer (positional parameter)
    #[sanitizer(RoundToScaleSanitizer(2))]
    pub balance: Decimal,

    // Named fields sanitizer
    #[sanitizer(ClampSanitizer, min = 0, max = 120)]
    pub age: Uint8,
}
}

Built-in Sanitizers

All sanitizers are available in wasm_dbms_api::prelude.

String Sanitizers

TrimSanitizer - Remove leading/trailing whitespace

#![allow(unused)]
fn main() {
#[sanitizer(TrimSanitizer)]
pub name: Text,
// "  Alice  " → "Alice"
}

CollapseWhitespaceSanitizer - Collapse multiple spaces into one

#![allow(unused)]
fn main() {
#[sanitizer(CollapseWhitespaceSanitizer)]
pub description: Text,
// "Hello    World" → "Hello World"
}

LowerCaseSanitizer - Convert to lowercase

#![allow(unused)]
fn main() {
#[sanitizer(LowerCaseSanitizer)]
pub email: Text,
// "Alice@Example.COM" → "alice@example.com"
}

UpperCaseSanitizer - Convert to uppercase

#![allow(unused)]
fn main() {
#[sanitizer(UpperCaseSanitizer)]
pub country_code: Text,
// "us" → "US"
}

SlugSanitizer - Convert to URL-safe slug

#![allow(unused)]
fn main() {
#[sanitizer(SlugSanitizer)]
pub slug: Text,
// "Hello World! This is a Test" → "hello-world-this-is-a-test"
}

UrlEncodingSanitizer - URL encode special characters

#![allow(unused)]
fn main() {
#[sanitizer(UrlEncodingSanitizer)]
pub path: Text,
// "hello world" → "hello%20world"
}

Numeric Sanitizers

RoundToScaleSanitizer - Round decimal to specific precision

#![allow(unused)]
fn main() {
#[sanitizer(RoundToScaleSanitizer(2))]
pub price: Decimal,
// 19.999 → 20.00
// 19.994 → 19.99
}

ClampSanitizer - Clamp value to range (signed)

#![allow(unused)]
fn main() {
#[sanitizer(ClampSanitizer, min = -100, max = 100)]
pub temperature: Int32,
// 150 → 100
// -150 → -100
}

ClampUnsignedSanitizer - Clamp value to range (unsigned)

#![allow(unused)]
fn main() {
#[sanitizer(ClampUnsignedSanitizer, min = 0, max = 100)]
pub percentage: Uint8,
// 150 → 100
// 0 → 0
}

DateTime Sanitizers

TimezoneSanitizer - Convert to specific timezone

#![allow(unused)]
fn main() {
#[sanitizer(TimezoneSanitizer("America/New_York"))]
pub local_time: DateTime,
}

UtcSanitizer - Convert to UTC

#![allow(unused)]
fn main() {
#[sanitizer(UtcSanitizer)]
pub timestamp: DateTime,
// Any timezone → UTC
}

Null Sanitizers

NullIfEmptySanitizer - Convert empty strings to null

#![allow(unused)]
fn main() {
#[sanitizer(NullIfEmptySanitizer)]
pub bio: Nullable<Text>,
// "" → Null
// "Hello" → "Hello"
}

Implementing Custom Sanitizers

Create a struct implementing the Sanitize trait:

#![allow(unused)]
fn main() {
use wasm_dbms_api::prelude::{Sanitize, Value, DbmsResult};

/// Capitalizes the first letter of each word
pub struct TitleCaseSanitizer;

impl Sanitize for TitleCaseSanitizer {
    fn sanitize(&self, value: Value) -> DbmsResult<Value> {
        match value {
            Value::Text(text) => {
                let title_case = text
                    .as_str()
                    .split_whitespace()
                    .map(|word| {
                        let mut chars = word.chars();
                        match chars.next() {
                            None => String::new(),
                            Some(first) => {
                                first.to_uppercase().to_string() +
                                chars.as_str().to_lowercase().as_str()
                            }
                        }
                    })
                    .collect::<Vec<_>>()
                    .join(" ");
                Ok(Value::Text(title_case.into()))
            }
            other => Ok(other),  // Pass through non-text values
        }
    }
}

// Usage
#[sanitizer(TitleCaseSanitizer)]
pub title: Text,
// "hello world" → "Hello World"
}

Custom sanitizer with parameters:

#![allow(unused)]
fn main() {
/// Truncates string to max length
pub struct TruncateSanitizer(pub usize);

impl Sanitize for TruncateSanitizer {
    fn sanitize(&self, value: Value) -> DbmsResult<Value> {
        match value {
            Value::Text(text) => {
                let truncated: String = text.as_str().chars().take(self.0).collect();
                Ok(Value::Text(truncated.into()))
            }
            other => Ok(other),
        }
    }
}

// Usage
#[sanitizer(TruncateSanitizer(100))]
pub summary: Text,
// "very long text..." → truncated to 100 chars
}

Custom sanitizer with named parameters:

#![allow(unused)]
fn main() {
/// Replaces a pattern with replacement
pub struct ReplaceSanitizer {
    pub pattern: &'static str,
    pub replacement: &'static str,
}

impl Sanitize for ReplaceSanitizer {
    fn sanitize(&self, value: Value) -> DbmsResult<Value> {
        match value {
            Value::Text(text) => {
                let replaced = text.as_str().replace(self.pattern, self.replacement);
                Ok(Value::Text(replaced.into()))
            }
            other => Ok(other),
        }
    }
}

// Usage
#[sanitizer(ReplaceSanitizer, pattern = "\n", replacement = " ")]
pub single_line: Text,
}

Sanitization Order

When multiple sanitizers are applied, they run in declaration order:

#![allow(unused)]
fn main() {
#[derive(Table, ...)]
#[table = "users"]
pub struct User {
    #[primary_key]
    pub id: Uint32,

    // Order matters!
    #[sanitizer(TrimSanitizer)]              // 1. Trim whitespace
    #[sanitizer(CollapseWhitespaceSanitizer)] // 2. Collapse spaces
    #[sanitizer(LowerCaseSanitizer)]         // 3. Lowercase
    pub email: Text,
}

// Input: "  Alice@Example.COM  "
// After TrimSanitizer: "Alice@Example.COM"
// After CollapseWhitespaceSanitizer: "Alice@Example.COM" (no change)
// After LowerCaseSanitizer: "alice@example.com"
}

Sanitizers run before validators:

#![allow(unused)]
fn main() {
#[sanitizer(TrimSanitizer)]           // 1. Trim
#[validate(MaxStrlenValidator(100))]  // 2. Validate length (after trim)
pub name: Text,
}

Examples

User profile sanitization:

#![allow(unused)]
fn main() {
#[derive(Debug, Table, Clone, PartialEq, Eq)]
#[table = "users"]
pub struct User {
    #[primary_key]
    pub id: Uint32,

    // Clean up name
    #[sanitizer(TrimSanitizer)]
    #[sanitizer(CollapseWhitespaceSanitizer)]
    pub name: Text,

    // Normalize email
    #[sanitizer(TrimSanitizer)]
    #[sanitizer(LowerCaseSanitizer)]
    pub email: Text,

    // Convert empty to null
    #[sanitizer(NullIfEmptySanitizer)]
    pub bio: Nullable<Text>,

    // Uppercase country code
    #[sanitizer(UpperCaseSanitizer)]
    pub country: Nullable<Text>,
}
}

Financial data sanitization:

#![allow(unused)]
fn main() {
#[derive(Debug, Table, Clone, PartialEq, Eq)]
#[table = "transactions"]
pub struct Transaction {
    #[primary_key]
    pub id: Uuid,

    // Round to cents
    #[sanitizer(RoundToScaleSanitizer(2))]
    pub amount: Decimal,

    // Ensure positive (clamp negatives to 0)
    #[sanitizer(ClampUnsignedSanitizer, min = 0, max = 1000000)]
    pub fee: Uint32,

    // Always store in UTC
    #[sanitizer(UtcSanitizer)]
    pub timestamp: DateTime,
}
}

Content sanitization:

#![allow(unused)]
fn main() {
#[derive(Debug, Table, Clone, PartialEq, Eq)]
#[table = "articles"]
pub struct Article {
    #[primary_key]
    pub id: Uuid,

    // Clean title
    #[sanitizer(TrimSanitizer)]
    #[sanitizer(CollapseWhitespaceSanitizer)]
    pub title: Text,

    // Generate URL-safe slug
    #[sanitizer(SlugSanitizer)]
    pub slug: Text,

    // Clean up content
    #[sanitizer(TrimSanitizer)]
    pub content: Text,

    // Optional summary
    #[sanitizer(TrimSanitizer)]
    #[sanitizer(NullIfEmptySanitizer)]
    pub summary: Nullable<Text>,
}
}

Combined sanitization and validation:

#![allow(unused)]
fn main() {
#[derive(Debug, Table, Clone, PartialEq, Eq)]
#[table = "products"]
pub struct Product {
    #[primary_key]
    pub id: Uuid,

    // Sanitize then validate
    #[sanitizer(TrimSanitizer)]
    #[sanitizer(CollapseWhitespaceSanitizer)]
    #[validate(RangeStrlenValidator(1, 200))]
    pub name: Text,

    // Sanitize price to 2 decimals, no validation needed
    #[sanitizer(RoundToScaleSanitizer(2))]
    pub price: Decimal,

    // Create slug and validate format
    #[sanitizer(SlugSanitizer)]
    #[validate(KebabCaseValidator)]
    #[validate(MaxStrlenValidator(100))]
    pub slug: Text,

    // Clean URL and validate format
    #[sanitizer(TrimSanitizer)]
    #[validate(UrlValidator)]
    pub image_url: Nullable<Text>,
}
}

JSON Reference

• Overview
• Defining JSON Columns
• Creating JSON Values
• JSON Filtering
  • Path Syntax
• Filter Operations
  • Contains (Structural Containment)
  • Extract (Path Extraction + Comparison)
  • HasKey (Path Existence)
• Combining JSON Filters
• Type Conversion
• Complete Example
• Error Handling

Overview

The Json data type allows you to store and query semi-structured JSON data within your database tables. This is useful for:

• Flexible schemas where structure varies between records
• Metadata storage
• User preferences and settings
• Any scenario where data structure may evolve

Defining JSON Columns

To use JSON in your schema, use the Json type:

#![allow(unused)]
fn main() {
use wasm_dbms_api::prelude::*;

#[derive(Debug, Table, Clone, PartialEq, Eq)]
#[table = "users"]
pub struct User {
    #[primary_key]
    pub id: Uint32,
    pub name: Text,
    pub metadata: Json,          // Required JSON field
    pub settings: Nullable<Json>, // Optional JSON field
}
}

Creating JSON Values

From string:

#![allow(unused)]
fn main() {
use std::str::FromStr;
use wasm_dbms_api::prelude::Json;

let json = Json::from_str(r#"{"name": "Alice", "age": 30}"#).unwrap();
}

From serde_json::Value:

#![allow(unused)]
fn main() {
use serde_json::json;
use wasm_dbms_api::prelude::Json;

let json: Json = json!({
    "name": "Alice",
    "age": 30,
    "tags": ["developer", "rust"],
    "address": {
        "city": "New York",
        "country": "US"
    }
}).into();
}

In insert requests:

#![allow(unused)]
fn main() {
let user = UserInsertRequest {
    id: 1.into(),
    name: "Alice".into(),
    metadata: json!({
        "role": "admin",
        "permissions": ["read", "write", "delete"]
    }).into(),
    settings: Nullable::Value(json!({
        "theme": "dark",
        "notifications": true
    }).into()),
};
}

JSON Filtering

wasm-dbms provides powerful JSON filtering through the JsonFilter enum:

• Contains: Check if JSON contains a pattern (structural containment)
• Extract: Extract value at path and compare
• HasKey: Check if a path exists

Path Syntax

Paths use dot notation with bracket array indices:

Path examples:

{
  "name": "Alice",           // Path: "name"
  "user": {
    "email": "a@b.com"       // Path: "user.email"
  },
  "tags": ["a", "b", "c"],   // Path: "tags[0]" = "a"
  "matrix": [[1,2], [3,4]]   // Path: "matrix[1][0]" = 3
}

Filter Operations

Contains (Structural Containment)

Checks if the JSON column contains a specified pattern. Implements PostgreSQL @> style containment:

• Objects: All key-value pairs in pattern must exist in target (recursive)
• Arrays: All elements in pattern must exist in target (order-independent)
• Primitives: Must be equal

#![allow(unused)]
fn main() {
use wasm_dbms_api::prelude::*;
use std::str::FromStr;

// Filter where metadata contains {"active": true}
let pattern = Json::from_str(r#"{"active": true}"#).unwrap();
let filter = Filter::json("metadata", JsonFilter::contains(pattern));
}

Containment behavior:

Use cases:

• Check if user has specific role: contains({"role": "admin"})
• Check if array contains value: contains({"tags": ["important"]})
• Check nested properties: contains({"settings": {"theme": "dark"}})

Extract (Path Extraction + Comparison)

Extract a value at path and apply comparison:

#![allow(unused)]
fn main() {
// Equal
let filter = Filter::json("metadata",
    JsonFilter::extract_eq("user.name", Value::Text("Alice".into()))
);

// Greater than
let filter = Filter::json("metadata",
    JsonFilter::extract_gt("user.age", Value::Int64(18.into()))
);

// In list
let filter = Filter::json("metadata",
    JsonFilter::extract_in("status", vec![
        Value::Text("active".into()),
        Value::Text("pending".into()),
    ])
);

// Is null (path doesn't exist or value is null)
let filter = Filter::json("metadata",
    JsonFilter::extract_is_null("deleted_at")
);

// Not null (path exists and value is not null)
let filter = Filter::json("metadata",
    JsonFilter::extract_not_null("email")
);
}

Available comparison methods:

HasKey (Path Existence)

Check if a path exists in the JSON:

#![allow(unused)]
fn main() {
// Check for root-level key
let filter = Filter::json("metadata", JsonFilter::has_key("email"));

// Check for nested path
let filter = Filter::json("metadata", JsonFilter::has_key("user.address.city"));

// Check for array element
let filter = Filter::json("metadata", JsonFilter::has_key("items[0]"));
}

Note: HasKey returns true even if the value at path is null. It only checks for path existence.

Combining JSON Filters

JSON filters combine with other filters using and(), or(), not():

#![allow(unused)]
fn main() {
// has email AND age > 18
let filter = Filter::json("metadata", JsonFilter::has_key("email"))
    .and(Filter::json("metadata", JsonFilter::extract_gt("age", Value::Int64(18.into()))));

// role = "admin" OR role = "moderator"
let filter = Filter::json("metadata", JsonFilter::extract_eq("role", Value::Text("admin".into())))
    .or(Filter::json("metadata", JsonFilter::extract_eq("role", Value::Text("moderator".into()))));

// Combine with regular filters
let pattern = Json::from_str(r#"{"active": true}"#).unwrap();
let filter = Filter::eq("id", Value::Int32(1.into()))
    .and(Filter::json("metadata", JsonFilter::contains(pattern)));

// NOT has deleted_at
let filter = Filter::json("metadata", JsonFilter::has_key("deleted_at")).not();
}

Type Conversion

When extracting JSON values, they’re converted to DBMS types:

Comparison examples:

#![allow(unused)]
fn main() {
// JSON: {"count": 42}
// Extracted as Int64, compare with Int64
JsonFilter::extract_eq("count", Value::Int64(42.into()))

// JSON: {"price": 19.99}
// Extracted as Decimal, compare with Decimal
JsonFilter::extract_gt("price", Value::Decimal(10.0.into()))

// JSON: {"active": true}
// Extracted as Boolean
JsonFilter::extract_eq("active", Value::Boolean(true))

// JSON: {"name": "Alice"}
// Extracted as Text
JsonFilter::extract_eq("name", Value::Text("Alice".into()))
}

Complete Example

#![allow(unused)]
fn main() {
use wasm_dbms_api::prelude::*;
use std::str::FromStr;

#[derive(Debug, Table, Clone, PartialEq, Eq)]
#[table = "products"]
pub struct Product {
    #[primary_key]
    pub id: Uint32,
    pub name: Text,
    pub attributes: Json,  // {"color": "red", "size": "M", "tags": ["sale", "new"], "price": 29.99}
}

fn example_queries(database: &impl Database) -> Result<(), Box<dyn std::error::Error>> {
    // Find all red products
    let filter = Filter::json("attributes",
        JsonFilter::extract_eq("color", Value::Text("red".into()))
    );
    let query = Query::builder().filter(filter).build();
    let red_products = database.select::<Product>(query)?;

    // Find products with "sale" tag
    let pattern = Json::from_str(r#"{"tags": ["sale"]}"#)?;
    let filter = Filter::json("attributes", JsonFilter::contains(pattern));
    let query = Query::builder().filter(filter).build();
    let sale_products = database.select::<Product>(query)?;

    // Find products with size attribute
    let filter = Filter::json("attributes", JsonFilter::has_key("size"));
    let query = Query::builder().filter(filter).build();
    let sized_products = database.select::<Product>(query)?;

    // Find red products with price > 20
    let filter = Filter::json("attributes", JsonFilter::extract_eq("color", Value::Text("red".into())))
        .and(Filter::json("attributes", JsonFilter::extract_gt("price", Value::Decimal(20.0.into()))));
    let query = Query::builder().filter(filter).build();
    let expensive_red = database.select::<Product>(query)?;

    // Find products in specific sizes
    let filter = Filter::json("attributes",
        JsonFilter::extract_in("size", vec![
            Value::Text("S".into()),
            Value::Text("M".into()),
            Value::Text("L".into()),
        ])
    );
    let query = Query::builder().filter(filter).build();
    let standard_sizes = database.select::<Product>(query)?;

    Ok(())
}
}

Error Handling

JSON filter operations return errors for:

Invalid path syntax:

• Empty paths
• Trailing dots ("user.")
• Unclosed brackets ("items[0")
• Negative indices ("items[-1]")
• Non-numeric array indices ("items[abc]")

Non-JSON column:

• Applying JSON filter to a non-JSON column

#![allow(unused)]
fn main() {
// Invalid path - will error
let filter = Filter::json("metadata", JsonFilter::has_key("user."));  // Trailing dot

// Non-JSON column - will error
let filter = Filter::json("name", JsonFilter::has_key("field"));  // "name" is Text, not Json

let result = database.select::<User>(
    Query::builder().filter(filter).build(),
);

match result {
    Err(DbmsError::Query(QueryError::InvalidQuery)) => {
        println!("Invalid JSON filter");
    }
    _ => {}
}
}

Schema Migrations

• Schema Migrations
  • Overview
  • Lifecycle
  • Drift Detection
  • Schema Snapshots
  • Migration Plan
    • MigrationOp
    • Apply Order
  • Compatible Widening Whitelist
  • Per-Table Hooks
    • #[default] Attribute
    • #[renamed_from] Attribute
    • Migrate Trait
  • Migration Policy
  • Errors
  • API Surface
    • Generic (wasm-dbms)
    • DatabaseSchema Dispatch
    • IC Endpoints
  • Non-Goals
  • Worked Example
  • Best Practices

Overview

Schema migrations let #[derive(Table)] schemas evolve across releases without losing data or requiring manual stable-memory surgery. The framework:

1. Stores a TableSchemaSnapshot for every registered table on disk.
2. Hashes those snapshots into a single schema_hash cached on Page 0 of the schema registry.
3. On every boot, recomputes the hash from the compiled schema and compares it against the stored hash to detect drift.
4. Refuses CRUD while in drift state and waits for an explicit dbms.migrate(policy) call.
5. Applies the diff between stored and compiled snapshots transactionally via the journaled writer.

Migrations are forward-only and explicit. The DBMS never auto-migrates on init — the caller decides when (and whether) to run them.

Lifecycle

┌─────────────────────────────────────────────────────────┐
│  boot / post_upgrade                                    │
│    ├─ load SchemaRegistry (Page 0) → read schema_hash   │
│    ├─ compute current_hash from compiled schemas        │
│    └─ drift = (stored_hash != current_hash)             │
├─────────────────────────────────────────────────────────┤
│  drift == false                                         │
│    ├─ CRUD allowed                                      │
│    └─ migrate() is a no-op                              │
├─────────────────────────────────────────────────────────┤
│  drift == true                                          │
│    ├─ CRUD returns DbmsError::Migration(SchemaDrift)    │
│    ├─ ACL methods bypass the check                      │
│    ├─ plan_migration() → Vec<MigrationOp>               │
│    └─ migrate(policy) applies ops, clears drift         │
└─────────────────────────────────────────────────────────┘

Performance contract:

• Boot: one u64 read from Page 0 plus one xxh3 hash of the encoded compiled snapshots. O(tables × columns).
• Hot path (CRUD): a single bool load (drift flag on the DBMS context) plus a branch. No snapshot decode, no hash recompute.
• Snapshot decode: only on plan_migration() or migrate(). Never during CRUD.

Drift Detection

Drift is the only signal the DBMS uses to decide whether migration is required. It is computed once on boot:

1. Load SchemaRegistry from Page 0.
2. For each table in DatabaseSchema::compiled_snapshots(), encode the snapshot.
3. Compute current_hash = xxh3(sorted-by-name concatenation of encoded bytes).
4. drift = (schema_registry.schema_hash != current_hash).
5. Cache drift: bool on the DBMS context.

Every CRUD entry point early-returns Err(DbmsError::Migration(MigrationError::SchemaDrift)) while drift == true. ACL methods (acl_add_principal, acl_remove_principal, acl_allowed_principals) bypass the check so the operator can recover even if the drift state is stuck.

Schema Snapshots

A snapshot is a self-describing, versioned view of a table’s compile-time shape. It captures only what is meaningful for migration; transient or derivable fields are intentionally omitted.

#![allow(unused)]
fn main() {
pub struct TableSchemaSnapshot {
    pub version: u8,                    // bumped on any breaking layout change
    pub name: String,
    pub primary_key: String,
    pub alignment: u32,
    pub columns: Vec<ColumnSnapshot>,   // declaration order preserved
    pub indexes: Vec<IndexSnapshot>,
}

pub struct ColumnSnapshot {
    pub name: String,
    pub data_type: DataTypeSnapshot,
    pub nullable: bool,
    pub auto_increment: bool,
    pub unique: bool,
    pub primary_key: bool,
    pub foreign_key: Option<ForeignKeySnapshot>,
    pub default: Option<Value>,
}

#[repr(u8)]
pub enum DataTypeSnapshot {
    Int8 = 0x01, Int16 = 0x02, Int32 = 0x03, Int64 = 0x04,
    Uint8 = 0x10, Uint16 = 0x11, Uint32 = 0x12, Uint64 = 0x13,
    Float32 = 0x20, Float64 = 0x21, Decimal = 0x22,
    Boolean = 0x30,
    Date = 0x40, Datetime = 0x41,
    Blob = 0x50, Text = 0x51, Uuid = 0x52,
    Json = 0x60,
    Custom { tag: String, wire_size: WireSize } = 0xF0,
}

pub enum WireSize {
    Fixed(u32),       // column occupies exactly N bytes
    LengthPrefixed,   // body preceded by 2-byte LE length prefix
}
}

WireSize is derived at compile time from the custom type’s Encode::SIZE: DataSize::Fixed(n) → WireSize::Fixed(n), DataSize::Dynamic → WireSize::LengthPrefixed. The migration codec uses it to slice column bytes during a snapshot-driven rewrite without invoking the user’s Encode::decode impl.

Stability rules:

1. DataTypeSnapshot discriminants are frozen. Never reorder, never reuse a removed slot.
2. Adding a field appends at the tail and bumps the container version. Old readers stop at the previous length prefix.
3. Removing a field leaves the slot reserved. Do not shift later fields.
4. Wire format per struct: length-prefix + field-by-field little-endian. String = u16 length + UTF-8 bytes. Option = u8 flag + body. Vec = u32 length + entries.

The snapshot encoder enforces hard caps on identifier lengths and table shape — see the Schema Definition warning for the full list. Names exceeding 255 bytes will truncate or panic at runtime.

Migration Plan

The planner takes two inputs:

• stored: Vec<TableSchemaSnapshot> — read from each table’s snapshot page.
• compiled: Vec<TableSchemaSnapshot> — built from compile-time TableSchema::schema_snapshot().

Tables match by exact, case-sensitive name. The diff produces three buckets:

• compiled \ stored → CreateTable.
• stored ∩ compiled → per-table column + index diff (see below).
• stored \ compiled → DropTable.

Column diff (per matched table):

For each compiled column:

1. Look up the stored column by name. Match → step 3.
2. On miss, walk the compiled column’s renamed_from slice. The first stored column hit emits RenameColumn; continue at step 3 with the renamed stored column.
3. Compare (data_type, nullable, auto_increment, unique, primary_key, foreign_key):
   • Types differ and the change is in the widening whitelist → WidenColumn.
   • Types differ and Migrate::transform_column returns a non-trivial override → TransformColumn.
   • Types differ and neither applies → MigrationError::IncompatibleType.
   • Any constraint flag changed → AlterColumn { changes }.

Stored columns not matched by any compiled column (directly or via renamed_from) → DropColumn. Compiled columns not matched → AddColumn. If non-nullable, the planner requires either #[default = ...] or Migrate::default_value returning Some, otherwise MigrationError::DefaultMissing.

Index diff:

Indexes are matched by (sorted column list, unique) tuple. Differences emit AddIndex / DropIndex.

MigrationOp

#![allow(unused)]
fn main() {
pub enum MigrationOp {
    CreateTable { name: String, schema: TableSchemaSnapshot },
    DropTable { name: String },                            // destructive
    AddColumn { table: String, column: ColumnSnapshot },
    DropColumn { table: String, column: String },          // destructive
    RenameColumn { table: String, old: String, new: String },
    AlterColumn {
        table: String,
        column: String,
        changes: ColumnChanges,
    },
    WidenColumn {
        table: String,
        column: String,
        old_type: DataTypeSnapshot,
        new_type: DataTypeSnapshot,
    },
    TransformColumn {
        table: String,
        column: String,
        old_type: DataTypeSnapshot,
        new_type: DataTypeSnapshot,
    },
    AddIndex { table: String, index: IndexSnapshot },
    DropIndex { table: String, index: IndexSnapshot },
}

pub struct ColumnChanges {
    pub nullable: Option<bool>,
    pub unique: Option<bool>,
    pub auto_increment: Option<bool>,
    pub primary_key: Option<bool>,
    pub foreign_key: Option<Option<ForeignKeySnapshot>>, // Some(None) = drop FK
}
}

Apply Order

Ops are sorted into a deterministic order so an AddColumn referencing a new FK target finds its target table already created, and so tightenings run only after data is in place:

1. CreateTable — new FK targets must exist first.
2. DropIndex.
3. DropColumn.
4. RenameColumn.
5. AlterColumn — relaxations only (nullable: true, unique: false, drop FK).
6. WidenColumn.
7. TransformColumn.
8. AddColumn.
9. AlterColumn — tightenings (nullable: false, unique: true, add FK). The planner validates existing data; offending rows trigger MigrationError::ConstraintViolation.
10. AddIndex.
11. DropTable.

All ops execute inside a single JournaledWriter session. Any failure rolls back every page touched; stored snapshots, schema_hash, and the drift flag are not mutated on failure.

Commit step (on success):

1. Write each updated TableSchemaSnapshot to its schema_snapshot_page.
2. Recompute schema_hash and write to SchemaRegistry on Page 0.
3. Clear the in-memory drift flag.

All three writes live in the same journal session as the data rewrites, so partial migrations are impossible.

Pre-flight validation: before opening the journal session, the planner runs plan_migration(), checks MigrationPolicy, and verifies each op is applicable (AddColumn has a default or is nullable, type changes are widenings or have a transform, etc.). Errors in this phase do not touch memory.

Compatible Widening Whitelist

Auto-applied without user code. The framework rewrites records in place.

Everything else (narrowing, sign flips, int↔float, int↔text, etc.) falls through to TransformColumn or errors with MigrationError::IncompatibleType.

Per-Table Hooks

Three macro features feed the planner. They produce no runtime cost on CRUD.

#[default] Attribute

Static per-column default for AddColumn ops on non-nullable columns.

#![allow(unused)]
fn main() {
#[derive(Table, ...)]
#[table = "users"]
pub struct User {
    #[primary_key]
    pub id: Uint32,
    pub name: Text,

    #[default = 0]
    pub login_count: Uint32,
}
}

The expression must convert into the column’s Value variant via From/Into. See the Default Value section in the schema reference for the full rules.

#[renamed_from] Attribute

Lists previous names for a column so the planner can emit RenameColumn instead of DropColumn + AddColumn:

#![allow(unused)]
fn main() {
#[derive(Table, ...)]
#[table = "users"]
pub struct User {
    #[primary_key]
    pub id: Uint32,

    #[renamed_from("username", "user_name")]
    pub name: Text,
}
}

Multiple entries support recovery from skipped releases. See the Renamed From section in the schema reference.

Migrate Trait

#[derive(Table)] emits an empty impl Migrate for T {} for every table by default. Override it by adding #[migrate] at the struct level and writing the impl yourself:

#![allow(unused)]
fn main() {
pub trait Migrate
where
    Self: TableSchema,
{
    /// Dynamic default for AddColumn on a non-nullable column.
    /// `None` falls back to the static `#[default]` attribute, else
    /// DefaultMissing.
    fn default_value(_column: &str) -> Option<Value> { None }

    /// Transform a stored value during an incompatible type change.
    /// `Ok(None)` → no transform (errors unless widening applies).
    /// `Ok(Some(v))` → use `v`.
    /// `Err(_)` → abort migration; journal rolls back.
    fn transform_column(
        _column: &str,
        _old: Value,
    ) -> DbmsResult<Option<Value>> {
        Ok(None)
    }
}
}

See the Migrate Override section in the schema reference for usage examples.

Migration Policy

#![allow(unused)]
fn main() {
pub struct MigrationPolicy {
    pub allow_destructive: bool,   // DropTable, DropColumn
}

impl Default for MigrationPolicy {
    fn default() -> Self {
        Self { allow_destructive: false }
    }
}
}

The default policy refuses destructive ops. Pre-flight planning emits MigrationError::DestructiveOpDenied { op } if any DropTable or DropColumn op is present and allow_destructive is false.

#![allow(unused)]
fn main() {
// Allow drops
dbms.migrate(MigrationPolicy { allow_destructive: true })?;
}

Errors

DbmsError::Migration(MigrationError) covers the full migration pipeline:

See the Migration Errors section in the errors reference for matching examples and remediation.

API Surface

Generic (wasm-dbms)

#![allow(unused)]
fn main() {
impl<M, A, S> Dbms<M, A, S>
where
    M: MemoryProvider,
    A: AccessControl,
    S: DatabaseSchema<M, A>,
{
    /// O(1). True iff compiled schema differs from stored.
    pub fn has_drift(&self) -> bool;

    /// Compute the diff without applying. Safe to call during drift.
    pub fn plan_migration(&self) -> DbmsResult<Vec<MigrationOp>>;

    /// Apply the diff. Transactional. Errors leave the database unchanged.
    pub fn migrate(&mut self, policy: MigrationPolicy) -> DbmsResult<()>;
}
}

DatabaseSchema Dispatch

#[derive(DatabaseSchema)] emits three migration dispatch methods alongside the CRUD dispatch methods:

#![allow(unused)]
fn main() {
pub trait DatabaseSchema<M, A>
where
    M: MemoryProvider,
    A: AccessControl,
{
    // ... existing CRUD dispatch ...

    fn migrate_default(table: &str, column: &str) -> Option<Value>
    where
        Self: Sized;

    fn migrate_transform(
        table: &str,
        column: &str,
        old: Value,
    ) -> DbmsResult<Option<Value>>
    where
        Self: Sized;

    fn compiled_snapshots() -> Vec<TableSchemaSnapshot>
    where
        Self: Sized;
}
}

The macro generates match arms keyed by table name. migrate_default chains Migrate::default_value → ColumnDef::default; migrate_transform dispatches to Migrate::transform_column; compiled_snapshots calls T::schema_snapshot() for every table in the #[tables(...)] list.

IC Endpoints

#[derive(DbmsCanister)] emits three additional admin-gated endpoints:

service : (IcDbmsCanisterArgs) -> {
  // ...
  has_schema_drift : () -> (bool) query;
  plan_migration  : () -> (Result_Vec_MigrationOp);
  migrate         : (MigrationPolicy) -> (Result);
}

All three honour the existing ACL check. MigrationOp, MigrationPolicy, TableSchemaSnapshot, ColumnSnapshot, IndexSnapshot, ForeignKeySnapshot, DataTypeSnapshot, and ColumnChanges derive CandidType + Deserialize behind the candid feature in wasm-dbms-api, so they appear in the generated .did automatically.

Non-Goals

The following are intentionally out of scope:

• Table rename. Detect via renamed_from on columns; full table rename requires manual migration.
• Custom data type binary evolution. User-defined types are keyed by name; binary layout stability remains the user’s responsibility.
• Downgrade / rollback to an older schema. Migrations are forward-only. Failed migrations roll back to the pre-migration state, but there is no path from a newer snapshot to an older compiled schema.
• Automatic migration on DB init. Migration is explicit, triggered by the operator.

Worked Example

#![allow(unused)]
fn main() {
use wasm_dbms_api::prelude::*;

// Release v1
#[derive(Debug, Table, Clone, PartialEq, Eq)]
#[table = "users"]
pub struct UserV1 {
    #[primary_key]
    pub id: Uint32,
    pub name: Text,
}

// Release v2: rename `name` → `full_name`, add a non-nullable
// `login_count` column with a default of 0, and keep an index on
// `full_name`.
#[derive(Debug, Table, Clone, PartialEq, Eq)]
#[table = "users"]
pub struct UserV2 {
    #[primary_key]
    pub id: Uint32,

    #[renamed_from("name")]
    #[index]
    pub full_name: Text,

    #[default = 0]
    pub login_count: Uint32,
}
}

After upgrading the canister, dbms.has_drift() returns true. Calling dbms.migrate(MigrationPolicy::default()) produces the following ops (in apply order):

1. RenameColumn { table: "users", old: "name", new: "full_name" }
2. AddColumn { table: "users", column: ColumnSnapshot { name: "login_count", default: Some(Value::Uint32(Uint32(0))), ... } }
3. AddIndex { table: "users", index: IndexSnapshot { columns: vec!["full_name".into()], unique: false } }

The session commits atomically; existing rows now carry login_count = 0 and the rename preserves their stored values.

Best Practices

1. Land schema changes one release at a time.

Combining a rename, a tightening, and a non-nullable add in one release multiplies the chance of ConstraintViolation mid-apply. Stage each kind in its own release where feasible.

2. Tighten only after backfilling.

Plan a nullable: false flip in two steps: first add the column nullable + backfill, then tighten in the next release. This isolates MigrationError::ConstraintViolation to a release where the cause is obvious.

3. Always start with allow_destructive: false.

Run plan_migration() and inspect the ops before flipping the policy. A surprise DropTable because of a typo in #[table = "..."] is much cheaper to catch in pre-flight than after the journal commits.

4. Test drift with the real binary format.

Hand-rolled snapshots in tests are risky because the encoder is the source of truth for the wire format. Roundtrip via Encode::encode / Encode::decode and assert equality.

5. Treat DataTypeSnapshot discriminants as frozen.

Adding a new variant takes a fresh tag. Renaming or reordering existing tags breaks every snapshot in production.

6. Persist migration logs externally.

The DBMS does not retain a history of applied migrations beyond the new schema_hash. If you need an audit trail, log plan_migration() output before calling migrate().

Errors Reference

• Errors Reference
  • Overview
  • Error Hierarchy
  • DbmsError
  • Migration Errors
    • SchemaDrift
    • IncompatibleType
    • DefaultMissing
    • ConstraintViolation
    • DestructiveOpDenied
    • TransformAborted
    • WideningIncompatible
    • TransformReturnedNone
    • ForeignKeyViolation
  • Query Errors
    • PrimaryKeyConflict
    • UniqueConstraintViolation
    • BrokenForeignKeyReference
    • ForeignKeyConstraintViolation
    • UnknownColumn
    • MissingNonNullableField
    • RecordNotFound
    • InvalidQuery
  • Transaction Errors
    • TransactionNotFound
  • Validation Errors
  • Sanitization Errors
  • Memory Errors
  • Error Handling Examples

Overview

wasm-dbms uses a structured error system to provide clear information about what went wrong. Errors are categorized by their source:

Error Hierarchy

DbmsError
├── Query(QueryError)
│   ├── PrimaryKeyConflict
│   ├── UniqueConstraintViolation
│   ├── BrokenForeignKeyReference
│   ├── ForeignKeyConstraintViolation
│   ├── UnknownColumn
│   ├── MissingNonNullableField
│   ├── RecordNotFound
│   └── InvalidQuery
├── Transaction(TransactionError)
│   └── NotFound
├── Validation(String)
├── Sanitize(String)
├── Memory(MemoryError)
├── Migration(MigrationError)
│   ├── SchemaDrift
│   ├── IncompatibleType { table, column, old, new }
│   ├── DefaultMissing { table, column }
│   ├── ConstraintViolation { table, column, reason }
│   ├── DestructiveOpDenied { op }
│   ├── TransformAborted { table, column, reason }
│   ├── WideningIncompatible { table, column, old_type, new_type }
│   ├── TransformReturnedNone { table, column }
│   └── ForeignKeyViolation { table, column, target_table, value }
└── Table(TableError)

DbmsError

The top-level error enum:

#![allow(unused)]
fn main() {
use wasm_dbms_api::prelude::DbmsError;

pub enum DbmsError {
    Memory(MemoryError),
    Migration(MigrationError),
    Query(QueryError),
    Table(TableError),
    Transaction(TransactionError),
    Sanitize(String),
    Validation(String),
}
}

Matching on error types:

#![allow(unused)]
fn main() {
match error {
    DbmsError::Query(query_err) => {
        // Handle query errors
    }
    DbmsError::Transaction(tx_err) => {
        // Handle transaction errors
    }
    DbmsError::Validation(msg) => {
        // Handle validation errors
        println!("Validation failed: {}", msg);
    }
    DbmsError::Sanitize(msg) => {
        // Handle sanitization errors
        println!("Sanitization failed: {}", msg);
    }
    DbmsError::Memory(mem_err) => {
        // Handle memory errors (rare)
    }
    DbmsError::Migration(mig_err) => {
        // Handle schema migration errors
    }
    DbmsError::Table(table_err) => {
        // Handle table errors (rare)
    }
}
}

Migration Errors

MigrationError covers the schema migration pipeline: drift detection on boot, plan validation, and journaled apply. See the Migrations Reference for the full lifecycle.

#![allow(unused)]
fn main() {
use wasm_dbms_api::prelude::{DataTypeSnapshot, MigrationError};

pub enum MigrationError {
    SchemaDrift,
    IncompatibleType {
        table: String,
        column: String,
        old: DataTypeSnapshot,
        new: DataTypeSnapshot,
    },
    DefaultMissing { table: String, column: String },
    ConstraintViolation { table: String, column: String, reason: String },
    DestructiveOpDenied { op: String },
    TransformAborted { table: String, column: String, reason: String },
    WideningIncompatible {
        table: String,
        column: String,
        old_type: DataTypeSnapshot,
        new_type: DataTypeSnapshot,
    },
    TransformReturnedNone { table: String, column: String },
    ForeignKeyViolation {
        table: String,
        column: String,
        target_table: String,
        value: String,
    },
}
}

SchemaDrift

Cause: A CRUD operation was attempted while the DBMS is in drift state — the compiled schema’s hash differs from the hash stored in the schema registry.

#![allow(unused)]
fn main() {
match database.insert::<User>(req) {
    Err(DbmsError::Migration(MigrationError::SchemaDrift)) => {
        // Stop accepting writes; call dbms.migrate(policy) first.
    }
    _ => {}
}
}

Solutions:

• Call dbms.migrate(MigrationPolicy::default()) from post_upgrade (IC) or your boot path to clear the drift flag.
• Inspect the diff first via dbms.plan_migration() to confirm the ops are safe.

IncompatibleType

Cause: A column changed to a type that is neither in the widening whitelist (e.g. Int32 → Int64) nor handled by Migrate::transform_column.

Solutions:

• If the change is conceptually a widen, double-check the from/to types match the whitelist.
• Otherwise mark the table with #[migrate] and provide a transform_column impl that maps the old Value to the new type.

DefaultMissing

Cause: Planning an AddColumn op for a non-nullable column that has neither a #[default = ...] attribute nor a Migrate::default_value override.

Solutions:

• Add #[default = <expr>] to the field, or
• Implement Migrate::default_value for the table (after marking it #[migrate]), or
• Make the column Nullable<T> so NULL is the implicit default.

ConstraintViolation

Cause: Tightening an existing column (nullable: false, unique: true, add foreign key) on data that violates the new constraint.

Solutions:

• Clean the data before bumping the schema (e.g. backfill NULLs, deduplicate).
• Stage the change across two releases: relaxation + cleanup, then tightening.

DestructiveOpDenied

Cause: The planner emitted a DropTable or DropColumn op while MigrationPolicy::allow_destructive is false.

Solutions:

• Confirm the destruction is intentional and pass MigrationPolicy { allow_destructive: true }.
• Otherwise re-introduce the missing struct/field in the compiled schema.

TransformAborted

Cause: A user-supplied Migrate::transform_column impl returned Err. The journaled migration session rolls back; stored data and schema_hash are unchanged.

Solutions:

• Inspect the embedded reason string to see which row failed.
• Fix the offending data manually (or via a helper canister method) before retrying migrate.

WideningIncompatible

Cause: A WidenColumn op named a (old_type, new_type) pair that is not in the widening whitelist, and the table did not provide a Migrate::transform_column impl that handled it. The journaled session rolls back; stored data and schema_hash are unchanged.

Solutions:

• Pick a target type that fits the whitelist (e.g. Uint32 → Uint64 rather than Uint32 → Uint8).
• Mark the table #[migrate] and provide a transform_column arm that maps the old Value into the new type.
• Stage the change across two releases: convert via a transform first, then narrow as a separate widening with valid bounds.

TransformReturnedNone

Cause: Migrate::transform_column returned Ok(None) for a column that needed a transform (no widening rule applied). The migration aborts and rolls back.

Solutions:

• Implement a concrete Ok(Some(_)) arm for the column in the table’s Migrate impl.
• Or pick a target type that fits the widening whitelist so the framework converts automatically.

ForeignKeyViolation

Cause: An add-FK tightening (AlterColumn with foreign_key: Some(Some(_))) found a row whose value is absent from the target table’s referenced column. The journaled session rolls back; stored data and schema_hash are unchanged.

Solutions:

• Clean up the orphan rows in a prior release before adding the FK.
• Inspect value in the error to identify the offending record(s).

Query Errors

Query errors occur during database operations.

PrimaryKeyConflict

Cause: Attempting to insert a record with a primary key that already exists.

#![allow(unused)]
fn main() {
// Insert first user
database.insert::<User>(UserInsertRequest {
    id: 1.into(),
    name: "Alice".into(),
    ..
})?;

// Insert second user with same ID - FAILS
let result = database.insert::<User>(UserInsertRequest {
    id: 1.into(),  // Same ID!
    name: "Bob".into(),
    ..
});

match result {
    Err(DbmsError::Query(QueryError::PrimaryKeyConflict)) => {
        println!("A user with this ID already exists");
    }
    _ => {}
}
}

Solutions:

• Use a unique primary key (e.g., UUID)
• Check if record exists before inserting
• Use upsert pattern (check, then insert or update)

UniqueConstraintViolation

Cause: Attempting to insert or update a record with a value that violates a #[unique] constraint.

#![allow(unused)]
fn main() {
// Insert first user
database.insert::<User>(UserInsertRequest {
    id: 1.into(),
    email: "alice@example.com".into(),
    ..
})?;

// Insert second user with same email - FAILS
let result = database.insert::<User>(UserInsertRequest {
    id: 2.into(),
    email: "alice@example.com".into(),  // Duplicate!
    ..
});

match result {
    Err(DbmsError::Query(QueryError::UniqueConstraintViolation { field })) => {
        println!("Duplicate value on field: {}", field);
        // field == "email"
    }
    _ => {}
}
}

Also triggered on update:

#![allow(unused)]
fn main() {
// Update user 2's email to match user 1's email - FAILS
let result = database.update::<User>(
    UserUpdateRequest::from_values(
        &[(email_col, Value::Text("alice@example.com".into()))],
        Some(Filter::eq("id", Value::Uint32(2.into()))),
    ),
);
}

Solutions:

• Check if a record with the same value exists before inserting
• Use a different value

BrokenForeignKeyReference

Cause: Foreign key references a record that doesn’t exist.

#![allow(unused)]
fn main() {
// Insert post with non-existent author
let result = database.insert::<Post>(PostInsertRequest {
    id: 1.into(),
    title: "My Post".into(),
    author_id: 999.into(),  // User 999 doesn't exist!
    ..
});

match result {
    Err(DbmsError::Query(QueryError::BrokenForeignKeyReference)) => {
        println!("Referenced user does not exist");
    }
    _ => {}
}
}

Solutions:

• Ensure referenced record exists before inserting
• Create referenced record first in a transaction

ForeignKeyConstraintViolation

Cause: Attempting to delete a record that is referenced by other records (with Restrict behavior).

#![allow(unused)]
fn main() {
// User has posts - cannot delete with Restrict
let result = database.delete::<User>(
    DeleteBehavior::Restrict,
    Some(Filter::eq("id", Value::Uint32(1.into()))),
);

match result {
    Err(DbmsError::Query(QueryError::ForeignKeyConstraintViolation)) => {
        println!("Cannot delete: user has related records");
    }
    _ => {}
}
}

Solutions:

• Delete related records first
• Use DeleteBehavior::Cascade to delete related records automatically

UnknownColumn

Cause: Referencing a column that doesn’t exist in the table.

#![allow(unused)]
fn main() {
// Filter with wrong column name
let filter = Filter::eq("username", Value::Text("alice".into()));  // Column is "name", not "username"

let result = database.select::<User>(
    Query::builder().filter(filter).build(),
);

match result {
    Err(DbmsError::Query(QueryError::UnknownColumn)) => {
        println!("Column does not exist in table");
    }
    _ => {}
}
}

Solutions:

• Check column names in your schema
• Use IDE autocompletion with typed column names

MissingNonNullableField

Cause: Required field not provided in insert/update.

#![allow(unused)]
fn main() {
// This typically happens at compile time with the generated types,
// but can occur if manually constructing requests or using dynamic queries
}

Solutions:

• Provide all required fields
• Use Nullable<T> for optional fields

RecordNotFound

Cause: Operation targets a record that doesn’t exist.

#![allow(unused)]
fn main() {
// Update non-existent record
let update = UserUpdateRequest::builder()
    .set_name("New Name".into())
    .filter(Filter::eq("id", Value::Uint32(999.into())))  // Doesn't exist
    .build();

let affected = database.update::<User>(update)?;

// affected == 0 indicates no records matched
if affected == 0 {
    println!("No records found to update");
}
}

Note: Update and delete operations return the count of affected rows. A count of 0 isn’t necessarily an error but indicates no matches.

InvalidQuery

Cause: Malformed query (invalid JSON path, bad filter syntax, etc.).

#![allow(unused)]
fn main() {
// Invalid JSON path
let filter = Filter::json("metadata", JsonFilter::has_key("user."));  // Trailing dot

let result = database.select::<User>(
    Query::builder().filter(filter).build(),
);

match result {
    Err(DbmsError::Query(QueryError::InvalidQuery)) => {
        println!("Query is malformed");
    }
    _ => {}
}
}

Common causes:

• Invalid JSON paths (trailing dots, unclosed brackets)
• Applying JSON filter to non-JSON column
• Type mismatches in comparisons
• Aggregate-specific:
  • SUM or AVG on non-numeric column ("aggregate requires numeric column: '<col>'")
  • HAVING references unknown column or agg{N} ("HAVING references unknown column or aggregate: '<col>'")
  • ORDER BY references unknown agg{N} ("ORDER BY references unknown aggregate output: '<col>'")
  • LIKE or JSON filter inside HAVING
  • Joins or eager relations on Database::aggregate

JoinInsideTypedSelect

Cause: A typed Database::select::<T> was called with a query that contains joins. Joins must go through select_join.

AggregateClauseInSelect

Cause: group_by or having was set on a non-aggregate select path (select, select_raw, or select_join). Use Database::aggregate instead — those clauses have no meaning outside aggregation and are rejected to prevent silent data loss.

#![allow(unused)]
fn main() {
let result = database.select::<User>(
    Query::builder().group_by(&["role"]).build(),
);

match result {
    Err(DbmsError::Query(QueryError::AggregateClauseInSelect)) => {
        // call database.aggregate::<User>(query, &aggregates) instead
    }
    _ => {}
}
}

Transaction Errors

TransactionNotFound

Cause: Invalid transaction ID or transaction already completed.

#![allow(unused)]
fn main() {
use wasm_dbms_api::prelude::{DbmsError, TransactionError};

match database.commit() {
    Err(DbmsError::Transaction(TransactionError::NoActiveTransaction)) => {
        println!("No active transaction to commit");
    }
    _ => {}
}
}

Causes:

• Transaction ID never existed
• Transaction was already committed
• Transaction was already rolled back

Validation Errors

Cause: Data fails validation rules.

#![allow(unused)]
fn main() {
#[derive(Table, ...)]
#[table = "users"]
pub struct User {
    #[validate(EmailValidator)]
    pub email: Text,
}

// Insert with invalid email
let result = database.insert::<User>(UserInsertRequest {
    id: 1.into(),
    email: "not-an-email".into(),  // Invalid!
    ..
});

match result {
    Err(DbmsError::Validation(msg)) => {
        println!("Validation failed: {}", msg);
        // msg might be: "Invalid email format"
    }
    _ => {}
}
}

Common validation errors:

• String too long (MaxStrlenValidator)
• String too short (MinStrlenValidator)
• Invalid email format (EmailValidator)
• Invalid URL format (UrlValidator)
• Invalid phone format (PhoneNumberValidator)

Sanitization Errors

Cause: Sanitizer fails to process the data.

#![allow(unused)]
fn main() {
// Sanitization errors are rare but can occur with malformed data
match result {
    Err(DbmsError::Sanitize(msg)) => {
        println!("Sanitization failed: {}", msg);
    }
    _ => {}
}
}

Sanitization errors are less common than validation errors since sanitizers typically transform data rather than reject it.

Memory Errors

Cause: Low-level memory errors.

#![allow(unused)]
fn main() {
pub enum MemoryError {
    OutOfBounds,           // Read/write outside allocated memory
    ProviderError(String),      // Memory provider error
    InsufficientSpace,     // Not enough space to allocate
}
}

Memory errors are rare and usually indicate:

• Running out of available memory
• Corrupted memory state
• Bug in wasm-dbms (please report!)

Error Handling Examples

Basic error handling:

#![allow(unused)]
fn main() {
let result = database.insert::<User>(user);

match result {
    Ok(()) => println!("Insert successful"),
    Err(DbmsError::Query(QueryError::PrimaryKeyConflict)) => {
        println!("User already exists");
    }
    Err(DbmsError::Query(QueryError::UniqueConstraintViolation { field })) => {
        println!("Duplicate value on field: {}", field);
    }
    Err(DbmsError::Query(QueryError::BrokenForeignKeyReference)) => {
        println!("Referenced record doesn't exist");
    }
    Err(DbmsError::Validation(msg)) => {
        println!("Validation error: {}", msg);
    }
    Err(e) => {
        println!("Database error: {:?}", e);
    }
}
}

Helper function pattern:

#![allow(unused)]
fn main() {
fn handle_db_error(error: DbmsError) -> String {
    match error {
        DbmsError::Query(QueryError::PrimaryKeyConflict) =>
            "Record with this ID already exists".to_string(),
        DbmsError::Query(QueryError::UniqueConstraintViolation { field }) =>
            format!("Duplicate value on unique field: {}", field),
        DbmsError::Query(QueryError::BrokenForeignKeyReference) =>
            "Referenced record not found".to_string(),
        DbmsError::Query(QueryError::ForeignKeyConstraintViolation) =>
            "Cannot delete: record has dependencies".to_string(),
        DbmsError::Validation(msg) =>
            format!("Invalid data: {}", msg),
        _ =>
            format!("Unexpected error: {:?}", error),
    }
}
}

For IC client-specific error handling (double result pattern with CallError), see the IC Errors Reference.

Architecture

• Architecture
  • Overview
  • Three-Layer Architecture
    • Layer 1: Memory Layer
    • Layer 2: DBMS Layer
    • Layer 3: API Layer
  • Crate Organization
    • Dependency Graph
    • Generic Layer (wasm-dbms)
      • wasm-dbms-api
      • wasm-dbms-memory
      • wasm-dbms
      • wasm-dbms-macros
    • IC Layer (ic-dbms)
      • ic-dbms-api
      • ic-dbms-canister
      • ic-dbms-macros
      • ic-dbms-client
  • Data Flow
    • Insert Operation
    • Select Operation
    • Select with Join
    • Transaction Flow
  • Extension Points
    • Custom Sanitizers
    • Custom Validators
    • Custom Data Types
    • Memory Provider

Overview

wasm-dbms is built as a layered architecture where each layer has specific responsibilities and builds upon the layer below. The core DBMS engine is runtime-agnostic (wasm-dbms-* crates), while the IC-specific adapter layer (ic-dbms-* crates) provides Internet Computer integration.

This design provides:

• Separation of concerns: Each layer focuses on one aspect
• Testability: Layers can be tested independently
• Portability: The generic layer runs on any WASM runtime (Wasmtime, Wasmer, WasmEdge)
• Flexibility: Internal implementations can change without affecting APIs

Three-Layer Architecture

┌─────────────────────────────────────────────────────────────┐
│                     Layer 3: API Layer                       │
│  Canister endpoints, Candid interface, access control        │
│  (DbmsCanister macro, ACL guards, request/response types)    │
├─────────────────────────────────────────────────────────────┤
│                     Layer 2: DBMS Layer                      │
│  Tables, CRUD operations, transactions, foreign keys         │
│  (TableRegistry, TransactionManager, query execution)        │
├─────────────────────────────────────────────────────────────┤
│                    Layer 1: Memory Layer                     │
│  Stable memory management, encoding/decoding, page allocation│
│  (MemoryProvider, MemoryManager, Encode trait)               │
└─────────────────────────────────────────────────────────────┘
                              │
                              ▼
                    ┌─────────────────┐
                    │  IC Stable      │
                    │    Memory       │
                    │   (or Heap)     │
                    └─────────────────┘

Layer 1: Memory Layer

Crate: wasm-dbms-memory

Responsibilities:

• Manage stable memory allocation (64 KiB pages)
• Encode/decode data to/from binary format
• Track free space and handle fragmentation
• Provide abstraction for testing (heap vs stable memory)

Key components:

Memory layout:

Page 0: Schema Registry (table → page mapping)
Page 1: ACL (allowed principals)
Page 2+: Table data (Page Ledger, Free Segments, Index Ledger, Autoincrement Ledger, Records, B-tree nodes)

See Memory Documentation for detailed technical information.

Layer 2: DBMS Layer

Crate: wasm-dbms

Responsibilities:

• Implement CRUD operations
• Manage transactions with ACID properties
• Enforce foreign key constraints
• Handle sanitization and validation
• Execute queries with filters

Key components:

Transaction model:

┌──────────────────────────────────────────┐
│           Active Transactions             │
│  ┌─────────┐  ┌─────────┐  ┌─────────┐   │
│  │  Tx 1   │  │  Tx 2   │  │  Tx 3   │   │
│  │ (overlay)│  │(overlay)│  │(overlay)│   │
│  └────┬────┘  └────┬────┘  └────┬────┘   │
│       │            │            │         │
│       └────────────┼────────────┘         │
│                    │                      │
│                    ▼                      │
│         ┌─────────────────┐              │
│         │  Committed Data │              │
│         │   (in memory)   │              │
│         └─────────────────┘              │
└──────────────────────────────────────────┘

Transactions use an overlay pattern:

• Changes are written to an overlay (in-memory)
• Reading checks overlay first, then committed data
• Index changes are tracked in a separate IndexOverlay per table
• IndexReader merges base index results with overlay additions/removals
• Commit merges overlay to committed data and flushes index changes to B-trees
• Rollback discards the overlay — on-disk B-trees remain untouched

Layer 3: API Layer

Crate: ic-dbms-canister (IC-specific)

Responsibilities:

• Expose Candid interface
• Handle request/response encoding
• Enforce access control (ACL)
• Route requests to DBMS layer
• Generate table-specific endpoints

Key components:

Generated API structure:

#![allow(unused)]
fn main() {
// For each table "users":
insert_users(UserInsertRequest, Option<TxId>) -> Result<()>
select_users(Query, Option<TxId>) -> Result<Vec<UserRecord>>
update_users(UserUpdateRequest, Option<TxId>) -> Result<u64>
delete_users(DeleteBehavior, Option<Filter>, Option<TxId>) -> Result<u64>

// Untyped select (supports joins):
select(table: String, Query, Option<TxId>) -> Result<Vec<Vec<(JoinColumnDef, Value)>>>

// Global operations:
begin_transaction() -> TxId
commit(TxId) -> Result<()>
rollback(TxId) -> Result<()>
acl_add_principal(Principal) -> Result<()>
acl_remove_principal(Principal) -> Result<()>
acl_allowed_principals() -> Vec<Principal>
}

Crate Organization

wasm-dbms/
├── crates/
│   ├── wasm-dbms/                  # Generic WASM DBMS crates
│   │   ├── wasm-dbms-api/          # Shared types and traits
│   │   ├── wasm-dbms-memory/       # Memory abstraction and page management
│   │   ├── wasm-dbms/              # Core DBMS engine
│   │   └── wasm-dbms-macros/       # Procedural macros (Encode, Table, CustomDataType, DatabaseSchema)
│   │
│   └── ic-dbms/                    # IC-specific crates
│       ├── ic-dbms-api/            # IC-specific types (re-exports wasm-dbms-api)
│       ├── ic-dbms-canister/       # Core IC canister implementation
│       ├── ic-dbms-macros/         # IC-specific macros (DatabaseSchema, DbmsCanister)
│       ├── ic-dbms-client/         # Client libraries
│       ├── example/                # Reference implementation
│       └── integration-tests/      # PocketIC integration tests
│
└── .artifact/                      # Build outputs (.wasm, .did, .wasm.gz)

Dependency Graph

wasm-dbms-macros <── wasm-dbms-api <── wasm-dbms-memory <── wasm-dbms
                                                                 ^
ic-dbms-macros <── ic-dbms-canister ─────────────────────────────┘
                        ^
                   ic-dbms-client

Generic Layer (wasm-dbms)

wasm-dbms-api

Purpose: Runtime-agnostic shared types and traits

Contents:

• Data types (Uint32, Text, DateTime, etc.)
• Value enum for runtime values
• Filter, Query, and Join types
• Database trait
• Sanitizer and Validator traits
• CustomDataType trait and CustomValue
• Error types (DbmsError, DbmsResult)

Dependencies: Minimal (serde, thiserror). Candid support via optional candid feature.

wasm-dbms-memory

Purpose: Memory abstraction and page management

Contents:

• MemoryProvider trait
• HeapMemoryProvider (testing)
• MemoryManager (page-level operations)
• SchemaRegistry (table-to-page mapping)
• AccessControl trait (identity-based ACL abstraction)
• AccessControlList (default AccessControl impl with Vec<u8> identity)
• NoAccessControl (no-op ACL for runtimes that don’t need access control)
• TableRegistry (record-level operations)

wasm-dbms

Purpose: Core DBMS engine (runtime-agnostic)

Contents:

• DbmsContext<M, A = AccessControlList> (owns all mutable state)
• WasmDbmsDatabase<'ctx, M, A> (session-scoped operations)
• Transaction management (overlay pattern)
• Foreign key integrity checks
• JOIN execution engine
• DatabaseSchema trait for dynamic dispatch

wasm-dbms-macros

Purpose: Generic procedural macros

Macros:

• #[derive(Encode)] - Binary serialization
• #[derive(Table)] - Table schema and related types
• #[derive(CustomDataType)] - Custom data type bridge
• #[derive(DatabaseSchema)] - Generates DatabaseSchema<M> trait implementation for schema dispatch

IC Layer (ic-dbms)

ic-dbms-api

Purpose: IC-specific types, re-exports generic API

Contents:

• Re-exports all types from wasm-dbms-api
• Principal custom data type (wraps candid::Principal)
• IcDbmsCanisterArgs init/upgrade arguments
• IcDbmsError / IcDbmsResult type aliases

ic-dbms-canister

Purpose: Thin IC adapter over wasm-dbms

Contents:

• IcMemoryProvider (IC stable memory)
• DBMS_CONTEXT thread-local wrapping DbmsContext<IcMemoryProvider>
• Canister API layer with ACL guards

Dependencies: ic-dbms-api, ic-dbms-macros, wasm-dbms, wasm-dbms-memory, ic-cdk

ic-dbms-macros

Purpose: IC-specific code generation

Macros:

• #[derive(DatabaseSchema)] - Generates DatabaseSchema<M> trait implementation (IC-specific paths)
• #[derive(DbmsCanister)] - Generates complete canister API

ic-dbms-client

Purpose: Client libraries for canister interaction

Implementations:

• IcDbmsCanisterClient - Inter-canister calls
• IcDbmsAgentClient - External via ic-agent (feature-gated)
• IcDbmsPocketIcClient - Testing with PocketIC (feature-gated)

Data Flow

Insert Operation

1. Client calls insert_users(request, tx_id)
              │
2. ACL guard checks caller authorization
              │
3. API layer deserializes request
              │
4. DBMS layer:
   a. Apply sanitizers to values
   b. Apply validators to values
   c. Check primary key uniqueness
   d. Validate foreign key references
   e. If tx_id: write to transaction overlay
      (index overlay tracks added keys)
      Else: write directly
              │
5. Memory layer:
   a. Encode record to bytes
   b. Find space (free segment or new page)
   c. Write to stable memory
   d. Update all indexes with the new
      key → RecordAddress mapping
              │
6. Return Result<()>

Select Operation

1. Client calls select_users(query, tx_id)
              │
2. ACL guard checks caller authorization
              │
3. API layer deserializes query
              │
4. DBMS layer:
   a. Parse filters
   b. Analyze filter for index plan
      (equality, range, or IN on indexed column)
   c. If index plan found:
      - Use IndexReader to get RecordAddresses
      - Load only matching records
      - Apply remaining filter as residual
   d. If no index plan (fallback):
      - Full table scan across all pages
   e. If tx_id: merge with overlay
   f. Apply ordering
   g. Apply limit/offset
   h. Select requested columns
   i. Handle eager loading
              │
5. Memory layer:
   a. Read pages
   b. Decode records
              │
6. Return Result<Vec<Record>>

Select with Join

1. Client calls select(table, query_with_joins, tx_id)
              │
2. ACL guard checks caller authorization
              │
3. API layer checks query.has_joins()
              │ (true)
4. JoinEngine:
   a. Read all rows from FROM table
   b. For each JOIN clause:
      - Read all rows from joined table
      - Resolve column references
      - Execute nested-loop join
   c. Apply filter on combined rows
   d. Apply ordering
   e. Apply offset/limit
   f. Flatten to output with JoinColumnDef
              │
5. Return Result<Vec<Vec<(JoinColumnDef, Value)>>>

See Join Engine for implementation details.

Transaction Flow

begin_transaction():
  1. Generate transaction ID
  2. Create empty overlay
  3. Record owner (caller identity)
  4. Return transaction ID

Operation with tx_id:
  1. Verify caller owns transaction
  2. Read from: overlay first, then committed
  3. Write to: overlay only

commit(tx_id):
  1. Verify caller owns transaction
  2. For each change in overlay:
     - Write to committed data (stable memory)
  3. Delete overlay
  4. Transaction ID becomes invalid

rollback(tx_id):
  1. Verify caller owns transaction
  2. Delete overlay (discard all changes)
  3. Transaction ID becomes invalid

Extension Points

ic-dbms provides several extension points for customization:

Custom Sanitizers

Implement the Sanitize trait:

#![allow(unused)]
fn main() {
pub trait Sanitize {
    fn sanitize(&self, value: Value) -> DbmsResult<Value>;
}
}

Custom Validators

Implement the Validate trait:

#![allow(unused)]
fn main() {
pub trait Validate {
    fn validate(&self, value: &Value) -> DbmsResult<()>;
}
}

Custom Data Types

Define custom data types with the CustomDataType derive macro:

#![allow(unused)]
fn main() {
#[derive(Encode, CustomDataType, Clone, Debug, PartialEq, Eq)]
#[type_tag = "status"]
pub enum Status {
    Active,
    Inactive,
}
}

Memory Provider

Implement MemoryProvider for custom memory backends:

#![allow(unused)]
fn main() {
pub trait MemoryProvider {
    const PAGE_SIZE: u64;
    fn size(&self) -> u64;
    fn pages(&self) -> u64;
    fn grow(&mut self, new_pages: u64) -> MemoryResult<u64>;
    fn read(&mut self, offset: u64, buf: &mut [u8]) -> MemoryResult<()>;
    fn write(&mut self, offset: u64, buf: &[u8]) -> MemoryResult<()>;
}
}

Built-in providers:

• IcMemoryProvider - Uses IC stable memory (IC production)
• WasiMemoryProvider - Uses a single flat file (WASI production)
• HeapMemoryProvider - Uses heap memory (testing)

Memory Management

• Memory Management
  • Overview
  • How Internet Computer Memory Works
  • Memory Model
  • Memory Provider
  • Memory Manager and MemoryAccess
  • Unclaimed Pages Ledger
  • Encode Trait
  • Schema Registry
  • ACL Storage
  • Table Registry
    • Schema Snapshot Ledger
    • Page Ledger
    • Free Segments Ledger
    • Autoincrement Ledger
  • Record Storage
    • Record Encoding
    • Record Alignment
    • Table Reader
  • Index Registry
    • Index Ledger
    • B-Tree Structure
    • Internal Node Layout
    • Leaf Node Layout
    • Index Maintenance
    • Index Tree Walker

Overview

This document provides the technical details of memory management in wasm-dbms, also known as Layer 0 (the Memory Layer). Understanding this layer is useful for:

• Performance optimization
• Debugging memory issues
• Contributing to wasm-dbms
• Understanding storage costs

How Internet Computer Memory Works

On the Internet Computer, canisters have access to stable memory that persists across upgrades. Key characteristics:

• Page-based: Memory is divided into 64 KiB (65,536 bytes) pages
• Growable: Canisters start small and can allocate additional pages
• Persistent: Survives canister upgrades
• Limited: Subject to subnet memory limits

wasm-dbms uses stable memory directly (not the heap) to ensure data persistence.

Memory Model

┌──────────────────────────────────────────────────┐
│ Page 0: Schema Registry (65 KiB)                 │
│   - Table name hashes → table registry pages     │
├──────────────────────────────────────────────────┤
│ Page 1: ACL Table (65 KiB)                       │
│   - List of allowed principals                   │
├──────────────────────────────────────────────────┤
│ Page 2: Unclaimed Pages Ledger (65 KiB)          │
│   - Stack of pages released by destructive ops   │
├──────────────────────────────────────────────────┤
│ Page 3: Table "users" Schema Snapshot            │
├──────────────────────────────────────────────────┤
│ Page 4: Table "users" Page Ledger                │
├──────────────────────────────────────────────────┤
│ Page 5: Table "users" Free Segments Ledger       │
├──────────────────────────────────────────────────┤
│ Page 6: Table "users" Index Ledger               │
├──────────────────────────────────────────────────┤
│ Page 7: Table "users" Autoincrement Ledger (*)   │
├──────────────────────────────────────────────────┤
│ Page 8: Table "posts" Schema Snapshot            │
├──────────────────────────────────────────────────┤
│ Page 9: Table "posts" Page Ledger                │
├──────────────────────────────────────────────────┤
│ Page 10: Table "posts" Free Segments Ledger      │
├──────────────────────────────────────────────────┤
│ Page 11: Table "posts" Index Ledger              │
├──────────────────────────────────────────────────┤
│ Page 12: Table "users" Records - Page 1          │
├──────────────────────────────────────────────────┤
│ Page 13: Table "users" Records - Page 2          │
├──────────────────────────────────────────────────┤
│ Page 14: B-Tree Node (index on users.id)         │
├──────────────────────────────────────────────────┤
│ Page 15: B-Tree Node (index on users.email)      │
├──────────────────────────────────────────────────┤
│ Page 16: Table "posts" Records - Page 1          │
├──────────────────────────────────────────────────┤
│ ...                                              │
└──────────────────────────────────────────────────┘
(*) Only allocated for tables with #[autoincrement] columns

Layout characteristics:

• Reserved pages (0-2) are allocated at initialization
• Each table gets a Schema Snapshot, Page Ledger, Free Segments Ledger, and Index Ledger
• Tables with #[autoincrement] columns also get an Autoincrement Ledger page
• Record pages and B-tree node pages are allocated on demand
• Pages can be interleaved between tables
• Pages released by destructive ops (e.g. DropTable) are returned to the Unclaimed Pages Ledger and reused by subsequent claim_page calls before the high-water mark is bumped

Memory Provider

The MemoryProvider trait abstracts memory access:

#![allow(unused)]
fn main() {
pub trait MemoryProvider {
    /// Size of a memory page in bytes (64 KiB for IC)
    const PAGE_SIZE: u64;

    /// Current memory size in bytes
    fn size(&self) -> u64;

    /// Number of allocated pages
    fn pages(&self) -> u64;

    /// Grow memory by new_pages
    /// Returns previous size on success
    fn grow(&mut self, new_pages: u64) -> MemoryResult<u64>;

    /// Read bytes from memory at offset
    fn read(&mut self, offset: u64, buf: &mut [u8]) -> MemoryResult<()>;

    /// Write bytes to memory at offset
    fn write(&mut self, offset: u64, buf: &[u8]) -> MemoryResult<()>;
}
}

Implementations:

#![allow(unused)]
fn main() {
// Production: Uses IC stable memory APIs
pub struct IcMemoryProvider;

#[cfg(target_family = "wasm")]
impl MemoryProvider for IcMemoryProvider {
    const PAGE_SIZE: u64 = ic_cdk::stable::WASM_PAGE_SIZE_IN_BYTES;

    fn grow(&mut self, new_pages: u64) -> MemoryResult<u64> {
        ic_cdk::stable::stable_grow(new_pages)
            .map_err(MemoryError::ProviderError)
    }

    fn read(&mut self, offset: u64, buf: &mut [u8]) -> MemoryResult<()> {
        ic_cdk::stable::stable_read(offset, buf);
        Ok(())
    }

    fn write(&mut self, offset: u64, buf: &[u8]) -> MemoryResult<()> {
        ic_cdk::stable::stable_write(offset, buf);
        Ok(())
    }
}

// Testing: Uses heap memory
pub struct HeapMemoryProvider {
    memory: Vec<u8>,
}
}

Memory Manager and MemoryAccess

The MemoryManager builds on MemoryProvider to handle page allocation. Its page-level read/write operations are exposed through the MemoryAccess trait, which allows the DBMS layer to substitute a journaled writer for atomic transactions (see Atomicity).

#![allow(unused)]
fn main() {
/// Abstracts page-level read/write operations.
///
/// `MemoryManager` implements this trait directly. The DBMS layer provides
/// `JournaledWriter`, which wraps a `MemoryManager` and records original
/// bytes before each write for rollback support.
pub trait MemoryAccess {
    fn page_size(&self) -> u64;
    /// Hand out a page — pops from the unclaimed-pages ledger if one is
    /// available, otherwise grows the underlying provider.
    fn claim_page(&mut self) -> MemoryResult<Page>;
    /// Return a page to the unclaimed-pages ledger after zeroing it.
    fn unclaim_page(&mut self, page: Page) -> MemoryResult<()>;
    /// Grow the provider by exactly one zero-initialized page (primitive).
    fn grow_one_page(&mut self) -> MemoryResult<Page>;
    /// Zero an entire allocated page (primitive).
    fn zero_page(&mut self, page: Page) -> MemoryResult<()>;
    fn read_at<D: Encode>(&mut self, page: Page, offset: PageOffset) -> MemoryResult<D>;
    fn write_at<E: Encode>(&mut self, page: Page, offset: PageOffset, data: &E) -> MemoryResult<()>;
    fn zero<E: Encode>(&mut self, page: Page, offset: PageOffset, data: &E) -> MemoryResult<()>;
    fn read_at_raw(&mut self, page: Page, offset: PageOffset, buf: &mut [u8]) -> MemoryResult<usize>;
}
}

claim_page and unclaim_page ship as default trait methods built on top of the four primitives — every implementor automatically inherits the unclaimed-pages-aware allocation strategy. JournaledWriter overrides only grow_one_page (intentionally not journaled, since extending the high-water mark cannot be replayed in reverse) and zero_page (records the full pre-zero page contents so a rollback restores them).

#![allow(unused)]
fn main() {
pub struct MemoryManager<P: MemoryProvider> {
    provider: P,
}

// Global instance (thread-local for IC)
// All state is consolidated in a single DbmsContext:
thread_local! {
    pub static DBMS_CONTEXT: DbmsContext<IcMemoryProvider> =
        DbmsContext::new(IcMemoryProvider::default());
}

impl<P: MemoryProvider> MemoryManager<P> {
    /// Initialize and allocate reserved pages
    fn init(provider: P) -> Self;

    /// ACL page number (always 1)
    pub const fn acl_page(&self) -> Page;

    /// Schema registry page (always 0)
    pub const fn schema_page(&self) -> Page;
}

// MemoryAccess is implemented for MemoryManager<P>,
// delegating directly to the underlying MemoryProvider.
impl<P: MemoryProvider> MemoryAccess for MemoryManager<P> { /* ... */ }
}

All table-registry and ledger functions are generic over impl MemoryAccess rather than taking &[mut] MemoryManager directly. This makes it possible to intercept writes at the DBMS layer without modifying any memory-crate code.

Unclaimed Pages Ledger

The unclaimed-pages ledger lives on reserved page 2 (UNCLAIMED_PAGES_PAGE). It is a LIFO stack of [Page] numbers that destructive operations have released. claim_page consults this stack before bumping the high-water mark; unclaim_page zeroes the page and pushes it onto the stack.

Serialization format:

Offset   Size   Field
0        4      Number of unclaimed pages (u32, little-endian)
4+       var    Sequence of page numbers (u32, little-endian) — newest last

The single reserved page can hold up to UNCLAIMED_PAGES_CAPACITY = 16382 entries (the encoded ledger size must fit in MSize = u16). Pushing beyond capacity returns MemoryError::UnclaimedPagesFull. A future extension may chain additional pages once a real workload exhausts the single-page budget; for now the v1 limit is enough to absorb typical drop-and-recreate cycles.

Behavior:

• claim_page pops the most recently unclaimed page (LIFO ordering keeps hot pages cache-warm). When the ledger is empty it grows the provider by exactly one zero-initialized page.
• unclaim_page zeroes the page in full before pushing — released pages never leak residual record bytes. The zero is journaled when invoked through JournaledWriter, so a rolled-back transaction restores the page contents and the ledger update.
• The default MemoryAccess impls of claim_page and unclaim_page drive the ledger entirely through the trait’s read/write methods, so every interceptor (journal, future overlays) automatically participates.
• MigrationOp::DropTable walks every page owned by the dropped table — record pages, page-ledger / free-segments / index-ledger pages, every B-tree node, schema-snapshot and (optional) autoincrement pages — and hands each one to unclaim_page before clearing the table from the schema registry.

Rollback semantics:

Inside an atomic block:

• A successful unclaim_page whose surrounding transaction rolls back is fully reversed: the page contents reappear and the ledger does not contain the page.
• A claim_page that hits the ledger pops a page; on rollback the ledger is restored and the page is “back” in the unclaimed pool. Any data the caller wrote into that page is also reverted by the journal.
• A claim_page that grows the provider returns a page whose existence is not journaled; on rollback the page stays grown but unreferenced (it leaks until the next process lifetime). This matches the previous behavior of allocate_page and is unchanged by this design.

Encode Trait

All data stored in memory implements the Encode trait:

#![allow(unused)]
fn main() {
pub trait Encode {
    /// Size characteristic: Fixed or Dynamic
    const SIZE: DataSize;

    /// Memory alignment in bytes
    /// - For Fixed: must equal size
    /// - For Dynamic: minimum 8, default 32
    const ALIGNMENT: PageOffset;

    /// Encode to bytes
    fn encode(&'_ self) -> Cow<'_, [u8]>;

    /// Decode from bytes
    fn decode(data: Cow<[u8]>) -> MemoryResult<Self>
    where
        Self: Sized;

    /// Size of encoded data
    fn size(&self) -> MSize;
}

pub enum DataSize {
    /// Fixed size in bytes (e.g., integers)
    Fixed(MSize),
    /// Variable size (e.g., strings, blobs)
    Dynamic,
}
}

Examples:

Schema Registry

The Schema Registry maps tables to their storage pages:

#![allow(unused)]
fn main() {
/// Information about a table's storage pages
pub struct TableRegistryPage {
    pub schema_snapshot_page: Page,                 // Schema Snapshot Ledger location
    pub pages_list_page: Page,                      // Page Ledger location
    pub free_segments_page: Page,                   // Free Segments Ledger location
    pub index_registry_page: Page,                  // Index Ledger location
    pub autoincrement_registry_page: Option<Page>,  // Autoincrement Ledger (if needed)
}

/// Maps table fingerprints to storage locations
pub struct SchemaRegistry {
    tables: HashMap<TableFingerprint, TableRegistryPage>,
}
}

The autoincrement_registry_page is only allocated when a table has at least one column with the #[autoincrement] attribute. For tables without autoincrement columns, this field is None, avoiding unnecessary page allocation.

Table Fingerprint:

• Hash of TableSchema::table_name() — stable across rebuilds and schema evolution
• Used as the key into the schema registry’s HashMap
• Enables multiple tables in one canister

Name collision detection:

SchemaRegistry::register_table is collision-aware. When the fingerprint slot is already occupied, the registry loads the persisted Schema Snapshot from that table’s schema_snapshot_page and compares its name against the candidate’s TableSchema::table_name():

• Same name: the entry is the same logical table — return the existing pages, no allocation
• Different name: two distinct names hashed to the same value — return MemoryError::NameCollision { candidate, existing } without allocating any page

ACL Storage

The Access Control List is stored in Page 1. Access control is abstracted behind the AccessControl trait, which allows different runtimes to use different identity types (e.g., Principal on IC, Vec<u8> for generic use).

#![allow(unused)]
fn main() {
pub trait AccessControl: Default {
    type Id;

    fn load<M>(mm: &MemoryManager<M>) -> MemoryResult<Self>
    where
        M: MemoryProvider,
        Self: Sized;

    fn is_allowed(&self, identity: &Self::Id) -> bool;
    fn allowed_identities(&self) -> Vec<Self::Id>;
    fn add_identity<M>(&mut self, identity: Self::Id, mm: &mut MemoryManager<M>) -> MemoryResult<()>
    where
        M: MemoryProvider;
    fn remove_identity<M>(&mut self, identity: &Self::Id, mm: &mut MemoryManager<M>) -> MemoryResult<()>
    where
        M: MemoryProvider;
}
}

The default implementation AccessControlList uses Vec<u8> as its identity type. NoAccessControl is a no-op implementation (with type Id = ()) for runtimes that don’t require ACL. The IC layer provides IcAccessControlList which wraps AccessControlList and uses Principal as its identity type.

Table Registry

Each table has a TableRegistry managing its records, plus an optional AutoincrementLedger for tables with autoincrement columns:

#![allow(unused)]
fn main() {
pub struct TableRegistry {
    schema_snapshot_ledger: SchemaSnapshotLedger,
    page_ledger: PageLedger,
    free_segments_ledger: FreeSegmentsLedger,
    index_ledger: IndexLedger,
    auto_increment_ledger: Option<AutoincrementLedger>,
}
}

Schema Snapshot Ledger

The SchemaSnapshotLedger persists a single TableSchemaSnapshot on the table’s schema_snapshot_page. The snapshot is the frozen, comparable view of the table’s compile-time schema — name, primary key, alignment, columns, and indexes — used for drift detection and migration planning.

#![allow(unused)]
fn main() {
pub struct SchemaSnapshotLedger {
    snapshot: TableSchemaSnapshot,  // cached copy of the on-disk snapshot
}

impl SchemaSnapshotLedger {
    pub fn init<Schema: TableSchema>(page: Page, mm: &mut impl MemoryAccess) -> MemoryResult<()>;
    pub fn load(page: Page, mm: &mut impl MemoryAccess) -> MemoryResult<Self>;
    pub fn write(&mut self, page: Page, snapshot: TableSchemaSnapshot, mm: &mut impl MemoryAccess) -> MemoryResult<()>;
    pub fn get(&self) -> &TableSchemaSnapshot;
}
}

Behavior:

• init is called exactly once per table by SchemaRegistry::register_table, capturing the snapshot from TableSchema::schema_snapshot() and writing it to the dedicated page
• load decodes the persisted snapshot and caches it in memory
• write replaces the persisted snapshot (used after a successful migration) and updates the cache; on write error the cache is left untouched
• get returns the cached snapshot — no I/O on the hot path

Serialization format (TableSchemaSnapshot):

Offset   Size     Field
0        1        Snapshot format version (u8, current = 0x01)
1        1        Table name length (u8)
2+       N1       UTF-8 table name
+        1        Primary key column name length (u8)
+        N2       UTF-8 primary key column name
+        4        Record alignment (u32, little-endian)
+        2        Column count (u16, little-endian)
+        var      For each column:
                    - 2 bytes: encoded column size (u16, little-endian)
                    - var bytes: encoded `ColumnSnapshot`
+        2        Index count (u16, little-endian)
+        var      For each index:
                    - 2 bytes: encoded index size (u16, little-endian)
                    - var bytes: encoded `IndexSnapshot`

ColumnSnapshot encodes name, data-type tag (with optional payload for Custom), nullable / auto-increment / unique / primary-key flags, optional foreign key, and optional default value. IndexSnapshot encodes the covered column names plus the unique flag.

Stability rules:

• DataTypeSnapshot discriminants are frozen — never reordered, never reused
• New fields append at the tail and bump the container version; old readers stop at the previous length prefix
• Removed fields leave their slot reserved; later fields do not shift

See Schema Reference for the full per-field layout.

Page Ledger

Tracks which pages contain records for this table:

#![allow(unused)]
fn main() {
pub struct PageLedger {
    ledger_page: Page,      // Where this ledger is stored
    pages: PageTable,       // List of data pages with free space info
}

impl PageLedger {
    /// Load from memory
    pub fn load(page: Page) -> MemoryResult<Self>;

    /// Get page for writing a record
    /// Returns existing page with space or allocates new
    pub fn get_page_for_record<R: Encode>(&mut self, record: &R) -> MemoryResult<Page>;

    /// Commit allocation (update free space tracking)
    pub fn commit<R: Encode>(&mut self, page: Page, record: &R) -> MemoryResult<()>;
}
}

Free Segments Ledger

Tracks free space from deleted/moved records:

#![allow(unused)]
fn main() {
pub struct FreeSegmentsLedger {
    free_segments_page: Page,
    tables: PagesTable,  // Pages containing FreeSegmentsTables
}

pub struct FreeSegment {
    pub page: Page,
    pub offset: PageOffset,
    pub size: MSize,
}

impl FreeSegmentsLedger {
    /// Insert a free segment (when record is deleted)
    pub fn insert_free_segment<E: Encode>(
        &mut self,
        page: Page,
        offset: PageOffset,
        record: &E,
    ) -> MemoryResult<()>;

    /// Find reusable space for a record
    pub fn find_reusable_segment<E: Encode>(
        &self,
        record: &E,
    ) -> MemoryResult<Option<FreeSegmentTicket>>;

    /// Commit reused space
    pub fn commit_reused_space<E: Encode>(
        &mut self,
        record: &E,
        segment: FreeSegmentTicket,
    ) -> MemoryResult<()>;
}
}

Space reuse logic:

1. When a record is deleted, its space is added to free segments
2. When inserting, check for suitable free segment first
3. If found, reuse the space; remaining space becomes new free segment
4. Adjacent free segments are merged to reduce fragmentation

Autoincrement Ledger

Tables with #[autoincrement] columns have a dedicated page storing the current counter value for each autoincrement column. The AutoincrementLedger manages these counters:

#![allow(unused)]
fn main() {
pub struct AutoincrementLedger {
    page: Page,
    registry: AutoincrementRegistry,  // column name → current Value
}
}

Serialization format:

Offset   Size    Field
0        1       Number of entries (u8)
1+       var     For each entry:
                 - 1 byte: column name length (u8)
                 - N bytes: UTF-8 column name
                 - var bytes: encoded Value (type-tagged)

Behavior:

• Initialized with zero values matching each column’s integer type when a table is registered
• next() increments the counter by one and persists the updated value to memory
• Uses checked_add — returns MemoryError::AutoincrementOverflow when a column reaches its type’s maximum value, preventing duplicate key generation
• Each column’s counter is independent; advancing one does not affect others
• State survives across load()/save() cycles (persisted to the dedicated page)

Supported types:

Record Storage

Record Encoding

Records are wrapped in RawRecord with a length header:

┌─────────────────────────────────────────┐
│  2 bytes: Data length (little-endian)   │
├─────────────────────────────────────────┤
│  N bytes: Encoded data                  │
├─────────────────────────────────────────┤
│  Padding to alignment boundary          │
└─────────────────────────────────────────┘

Dynamic size example (alignment=32, data=24 bytes):

Bytes 0-1:   Data length (24)
Bytes 2-25:  Data (24 bytes)
Bytes 26-31: Padding (6 bytes)
Total: 32 bytes (aligned)

Fixed size example (size=14 bytes):

Bytes 0-1:   Data length (14)
Bytes 2-15:  Data (14 bytes)
Total: 16 bytes (no padding for fixed)

Record Alignment

Alignment ensures efficient memory access:

#![allow(unused)]
fn main() {
impl<E: Encode> Alignment for E {
    fn alignment() -> usize {
        match E::SIZE {
            DataSize::Fixed(size) => size as usize,
            DataSize::Dynamic => E::ALIGNMENT as usize,
        }
    }
}

fn align_up<E: Encode>(size: usize) -> usize {
    let align = E::alignment();
    (size + align - 1) / align * align
}
}

Configuring alignment:

#![allow(unused)]
fn main() {
#[derive(Table, ...)]
#[table = "large_records"]
#[alignment = 64]  // Custom alignment for this table
pub struct LargeRecord {
    // ...
}
}

Table Reader

Reading records from a table:

#![allow(unused)]
fn main() {
impl<E: Encode> TableRegistry<E> {
    pub fn read_all(&self) -> MemoryResult<Vec<E>> {
        let mut records = Vec::new();

        for page in self.page_ledger.pages() {
            let mut offset = 0;

            while offset < PAGE_SIZE {
                // Read length header
                let len = read_u16_le(page, offset);

                if len == 0 {
                    // Skip empty slot
                    offset += E::alignment();
                    continue;
                }

                // Read and decode record
                let data = read_bytes(page, offset + 2, len);
                let record = E::decode(data)?;
                records.push(record);

                // Move to next aligned position
                offset += align_up::<E>(len + 2);
            }
        }

        Ok(records)
    }
}
}

Read process:

1. Read 2 bytes at offset for data length
2. If length is 0, skip to next aligned position
3. Read length bytes of data
4. Decode data into record
5. Move to next aligned position
6. Repeat until end of page

Index Registry

Each table has an IndexLedger that maps index definitions (column sets) to B-tree root pages. Indexes are always B+ trees where each node occupies exactly one memory page (64 KiB).

Every table automatically gets an index on its primary key. Additional indexes can be declared with the #[index] attribute (see Schema Reference).

Index Ledger

The IndexLedger is stored in a single page per table and maps column sets to B-tree root pages:

#![allow(unused)]
fn main() {
pub struct IndexLedger {
    ledger_page: Page,
    tables: HashMap<Vec<String>, Page>,  // column names → root page
}
}

Serialization format:

Offset   Size    Field
0-7      8       Number of indexes (u64)
8+       var     For each index:
                 - 8 bytes: column count (u64)
                 - For each column name:
                   - 1 byte: name length (u8)
                   - N bytes: UTF-8 column name
                 - 4 bytes: root page (u32)

When a table is registered via SchemaRegistry::register_table(), the index ledger is initialized by allocating one root page per index definition. The ledger supports insert, delete, update, exact-match search, and range scan operations — all delegated to the underlying B-tree for the appropriate column set.

B-Tree Structure

Indexes use a B+ tree where values (record pointers) are stored only in leaf nodes. Internal nodes contain separator keys that guide traversal. Each node is a single page.

#![allow(unused)]
fn main() {
struct RecordAddress {
    page: Page,         // 4 bytes, u32
    offset: PageOffset, // 2 bytes, u16
}
}

RecordAddress is the pointer stored in leaf entries, pointing to the exact location of the record in the table’s data pages. It is 6 bytes when serialized.

Key characteristics:

• Variable-size keys: Entries are packed as many as fit in a 64 KiB page
• Non-unique: The same key can map to multiple RecordAddress values
• Linked leaves: Leaf nodes form a doubly-linked list for range scans
• Node type tag: Byte 0 distinguishes internal (0x00) from leaf (0x01) nodes

Internal Node Layout

┌──────────────────────────────────────────────────────┐
│ Byte 0:     Node type (0x00 = INTERNAL)              │
│ Bytes 1-4:  Parent page (u32, u32::MAX if root)      │
│ Bytes 5-6:  Entry count (u16)                        │
│ Bytes 7-10: Rightmost child page (u32)               │
├──────────────────────────────────────────────────────┤
│ Entry 0:                                             │
│   Bytes 0-1: Key size (u16)                          │
│   Bytes 2+:  Key data (variable)                     │
│   Next 4:    Child page (u32)                        │
├──────────────────────────────────────────────────────┤
│ Entry 1: ...                                         │
├──────────────────────────────────────────────────────┤
│ ...                                                  │
└──────────────────────────────────────────────────────┘

Header size: 11 bytes. Entries are sorted by key. A search for key K routes to the child page of the first entry whose key is >= K, or to rightmost_child if K is greater than all entries.

Leaf Node Layout

┌──────────────────────────────────────────────────────┐
│ Byte 0:      Node type (0x01 = LEAF)                 │
│ Bytes 1-4:   Parent page (u32, u32::MAX if root)     │
│ Bytes 5-6:   Entry count (u16)                       │
│ Bytes 7-10:  Previous leaf page (u32, u32::MAX=none) │
│ Bytes 11-14: Next leaf page (u32, u32::MAX=none)     │
├──────────────────────────────────────────────────────┤
│ Entry 0:                                             │
│   Bytes 0-1: Key size (u16)                          │
│   Bytes 2+:  Key data (variable)                     │
│   Next 6:    RecordAddress (4-byte page + 2-byte     │
│              offset)                                 │
├──────────────────────────────────────────────────────┤
│ Entry 1: ...                                         │
├──────────────────────────────────────────────────────┤
│ ...                                                  │
└──────────────────────────────────────────────────────┘

Header size: 15 bytes. Entries are sorted by (key, record address). The prev_leaf / next_leaf pointers form a doubly-linked list across all leaves, enabling efficient forward and backward range scans.

Index Maintenance

Indexes are updated eagerly on every write operation:

• INSERT: After writing the record and obtaining its RecordAddress, the key is inserted into every index defined on the table.
• DELETE: After removing the record, the key-pointer pair is removed from every index.
• UPDATE: If indexed columns changed, those indexes are updated (delete old key + insert new key). If the record moved (size change), all indexes are updated with the new RecordAddress.

When a leaf node overflows during insertion, it splits at its midpoint. The first key of the new right sibling is promoted to the parent internal node. If the parent also overflows, the split propagates upward. When the root splits, a new root is created and the tree height increases by one.

When a leaf becomes empty after deletion (and is not the root), it is unlinked from the leaf chain and its parent is updated.

Index Tree Walker

Range scans use an IndexTreeWalker that iterates through leaf entries across linked leaf pages:

#![allow(unused)]
fn main() {
pub struct IndexTreeWalker<K: Encode + Ord> {
    entries: Vec<LeafEntry<K>>,   // Current leaf's entries
    cursor: usize,                // Position within current leaf
    next_leaf: Option<Page>,      // Next leaf page for continuation
    end_key: Option<K>,           // Optional upper bound (inclusive)
}
}

The walker starts at the first leaf entry >= start_key and advances through the linked-leaf chain until it reaches an entry > end_key (or exhausts all leaves). This provides efficient iteration for range queries without revisiting internal nodes.

Join Engine

• Overview
• Architecture
• Processing Pipeline
• Nested-Loop Join Algorithm
• NULL Padding
• Column Resolution
• Output Format
• Limitations

Overview

The join engine executes cross-table join queries, combining rows from two or more tables based on column equality conditions. It supports four join types — INNER, LEFT, RIGHT, and FULL — and integrates with the existing query pipeline for filtering, ordering, pagination, and column selection.

The implementation lives in crates/ic-dbms-canister/src/dbms/join.rs.

Architecture

The engine is implemented as a generic struct:

#![allow(unused)]
fn main() {
pub struct JoinEngine<'a, Schema: ?Sized>
where
    Schema: DatabaseSchema,
{
    schema: &'a Schema,
}
}

Key design decisions:

• Schema: ?Sized — The ?Sized bound allows the engine to work with Box<dyn DatabaseSchema>, which is how the API layer passes the schema at runtime.
• Borrows DatabaseSchema — The engine borrows the schema to read rows from tables via schema.select(dbms, table, query).
• Stateless — The engine holds no mutable state; it takes a Query and returns results in a single call.

The DatabaseSchema trait provides the select method that the engine uses to read all rows from each table involved in the join.

Processing Pipeline

The join() method processes a query through these steps:

                    ┌──────────────────────────┐
                    │ 1. Read FROM table rows  │
                    └────────────┬─────────────┘
                                 │
                    ┌────────────▼─────────────┐
                    │ 2. For each JOIN clause:  │◄──── left-to-right
                    │    Read right table rows  │
                    │    Nested-loop join       │
                    └────────────┬─────────────┘
                                 │
                    ┌────────────▼─────────────┐
                    │ 3. Apply filter           │
                    └────────────┬─────────────┘
                                 │
                    ┌────────────▼─────────────┐
                    │ 4. Apply ordering         │
                    └────────────┬─────────────┘
                                 │
                    ┌────────────▼─────────────┐
                    │ 5. Apply offset           │
                    └────────────┬─────────────┘
                                 │
                    ┌────────────▼─────────────┐
                    │ 6. Apply limit            │
                    └────────────┬─────────────┘
                                 │
                    ┌────────────▼─────────────┐
                    │ 7. Flatten to output      │
                    └──────────────────────────┘

1. Read FROM table: All rows from the primary table are loaded using an unfiltered Query::builder().all().build().
2. Process JOINs: Each Join clause is processed left-to-right. For each clause, the right table is read in full, column references are resolved, and the nested-loop join is executed against the accumulated result.
3. Filter: The query’s filter is applied to the combined rows using filter.matches_joined_row(), which supports qualified table.column references.
4. Order: Order-by clauses are applied in reverse (stable sort), so the primary sort key ends up correctly ordered.
5. Offset: Rows are skipped according to the offset value.
6. Limit: The result is truncated to the limit.
7. Flatten: Each joined row is converted from the internal JoinedRow representation to the output Vec<(JoinColumnDef, Value)> format, applying column selection.

Nested-Loop Join Algorithm

All four join types are handled by a single nested_loop_join method using two boolean flags:

The algorithm:

1. For each left row, iterate over all right rows.
2. If the left column value equals the right column value (and is not None), emit a combined row and mark the right row as matched.
3. After scanning all right rows for a given left row: if keep_unmatched_left is true and no match was found, emit the left row with NULL-padded right columns.
4. After all left rows are processed: if keep_unmatched_right is true, emit each unmatched right row with NULL-padded left columns.

This unified approach avoids code duplication across join types while keeping the logic straightforward.

NULL Padding

When a row has no match on the opposite side (in LEFT, RIGHT, or FULL joins), the missing columns are filled with Value::Null. The engine determines which columns to pad by inspecting a sample row from the opposite table:

#![allow(unused)]
fn main() {
fn null_pad_columns(&self, sample_row: &[(ColumnDef, Value)]) -> Vec<(ColumnDef, Value)> {
    sample_row
        .iter()
        .map(|(col, _)| (*col, Value::Null))
        .collect()
}
}

This preserves the correct column definitions (name, type, nullability) while setting every value to NULL. If the opposite table is empty (no sample row available), the padded group has zero columns.

Column Resolution

Column references in join ON conditions, filters, and ordering can be either qualified or unqualified:

• Qualified: "users.id" — explicitly specifies the table.
• Unqualified: "id" — defaults to the FROM table (for ON left-column) or the joined table (for ON right-column).

Resolution is handled by resolve_column_ref:

#![allow(unused)]
fn main() {
fn resolve_column_ref(&self, field: &str, default_table: &str) -> (String, &str) {
    if let Some((table, column)) = field.split_once('.') {
        (table.to_string(), column)
    } else {
        (default_table.to_string(), field)
    }
}
}

For filters and ordering on joined results, the same qualified/unqualified pattern applies. Unqualified names are searched across all table groups in the row, returning the first match.

Output Format

Join results use JoinColumnDef instead of ColumnDef:

#![allow(unused)]
fn main() {
pub struct JoinColumnDef {
    pub table: Option<String>,  // Source table name
    pub name: String,
    pub data_type: DataTypeKind,
    pub nullable: bool,
    pub primary_key: bool,
}
}

The table field is Some(table_name) for join results, allowing consumers to distinguish columns that share the same name across different tables.

At the API layer, the generated select endpoint checks query.has_joins():

• With joins: Routes to select_join, which uses JoinEngine.
• Without joins: Routes to select_raw, the standard single-table path.

Both paths return Vec<Vec<(JoinColumnDef, Value)>>, but for non-join queries the table field is None.

Limitations

• O(n*m) nested-loop join: Each join performs a full nested-loop comparison. For two tables of size n and m, this is O(n*m) per join clause.
• Full table scans for join matching: The join ON condition itself does not use indexes — both sides are compared via linear scan. However, if the query has a filter, the individual table reads that feed the join may use indexes (via the standard select path).
• All rows loaded into memory: Every table involved in the join is fully materialized in memory before processing. This can be a concern for very large tables on the IC.
• Equality joins only: The ON condition only supports column equality (left_col = right_col). Range conditions, expressions, and multi-column ON clauses are not supported.

Atomicity

• Atomicity
  • Overview
  • The Problem
  • Write-Ahead Journal
    • Architecture
    • Journal Flow
    • What is Journaled
  • Transaction Commit Atomicity
  • Edge Cases
    • Page Allocation
    • Nested Atomic Calls
    • Rollback Failure

Overview

A DBMS must guarantee atomicity: either all writes in an operation succeed, or none of them persist. Without atomicity, a crash or error mid-operation can leave the database in an inconsistent state (e.g., a record written but the page ledger not updated, or half the rows in a transaction committed while the rest are lost).

The Problem

The original atomic() implementation relied on panic semantics:

#![allow(unused)]
fn main() {
fn atomic<F, R>(&self, f: F) -> R
where
    F: FnOnce(&WasmDbmsDatabase<M, A>) -> DbmsResult<R>,
{
    match f(self) {
        Ok(res) => res,
        Err(err) => panic!("{err}"),
    }
}
}

On the Internet Computer, a panic (trap) automatically reverts all stable-memory writes made during that call. This gave IC canisters free atomicity. However, on non-IC WASM runtimes (e.g., Wasmtime, Wasmer, browser WASM), a panic does not revert memory. The host simply sees the guest abort, and any writes already flushed to linear memory remain. This made wasm-dbms effectively IC-only for write operations.

Write-Ahead Journal

The fix is a write-ahead journal. Before overwriting any bytes, the journal saves the original content at that offset. On error, the journal replays saved entries in reverse order, restoring every modified byte.

Architecture

The journal lives in the wasm-dbms crate’s transaction module, not in the memory layer. This separation keeps the memory crate (wasm-dbms-memory) focused on page-level I/O while the DBMS layer owns the transaction concern.

The key types are:

• MemoryAccess trait (in wasm-dbms-memory): Abstracts page-level read/write operations. MemoryManager implements this trait with direct writes.
• Journal (in wasm-dbms): A heap-only collection of JournalEntry records. Each entry stores the page, offset, and original bytes before a write.
• JournaledWriter (in wasm-dbms): Wraps a &mut MemoryManager and a &mut Journal, implementing MemoryAccess. Every write_at or zero call reads the original bytes first, records them in the journal, then delegates to the underlying MemoryManager.

All memory-crate functions that perform writes (in TableRegistry, PageLedger, FreeSegmentsLedger, etc.) are generic over impl MemoryAccess. When called with a plain MemoryManager, writes go directly to memory. When called with a JournaledWriter, writes are automatically recorded for rollback.

Journal Flow

┌─────────────────┐
│  Journal::new() │   Creates empty journal
└────────┬────────┘
         │
         ▼
┌─────────────────────────┐
│  JournaledWriter wraps  │
│  MemoryManager + Journal│
└────────┬────────────────┘
         │
         ▼
┌──────────────┐
│   write_at   │──► Reads original bytes, records in journal, then writes new data
│     zero     │──► Reads original bytes, records in journal, then writes zeros
└──────┬───────┘
       │
       ├── success ──► journal.commit()   ──► Drops entries (no-op)
       │
       └── error   ──► journal.rollback() ──► Replays entries in reverse via MemoryManager

Each journal entry is:

#![allow(unused)]
fn main() {
struct JournalEntry {
    page: Page,
    offset: PageOffset,
    original_bytes: Vec<u8>,
}
}