Skip to content

Many Ways ​

One chart and one Data Navigator structure, with at least six different ways to navigate. Try out our demo and all of the ways that Data Navigator enables input!

Ways to navigate

Keyboard and text are always on. Turn on any others.

Swiping takes over touch on the chart. Leave it off if you use a screen reader.

Text chat

View navigation structure

Try a path ​

  1. Enter the chart. You land on the chart summary.
  2. Drill in three times: Year, then 2015, then 62%, Chrome, 2015.
  3. Go right to 2016, then up to Firefox.
  4. Drill out to the browser to hear about Firefox across all years.

Mix inputs as you go. Type drill in, point right with your hand, then say "up to the browser". Every input moves the same position, shown by one outline on the chart and in the inspector.

"Many ways" ​

Offering many ways to do the same thing, and reach the same goals, is one of the best measures of a robust, accessible system. People use different inputs because of disability, device, setting, or preference, and those can change from one moment to the next. When every input reaches the same places, many more folks can reach the goals that they have in their own way.

This idea builds on Chartability, especially its robust and flexible principles, and on Skeleton, which calls for accommodating inputs beyond the keyboard, including touch and text.

We put this idea into the simple 2 words, "many ways," while working with the React Spectrum Charts team at Adobe. Thank you to that team for helping to shape our micro-philosophy for accessible interaction design.

How it works ​

Everything runs on one structure, one input, one rendering, and one textChat. The chart is a dual tree: every data point belongs to a year and a browser, so you can move across years or across browsers from any point.

InputHow it reaches a navigation rule
Keyboardinput.keydownValidator(event) returns the rule for the key
TexttextChat matches what you type to a rule name or label
VoiceThe browser's speech recognition passes what you said to chat.submit(text)
HandA MediaPipe gesture recognizer maps each gesture to a rule name
HeadA MediaPipe face landmarker maps head turns, nods, tilts, and expressions to rules
TouchPointer events map swipes, taps, and presses to rules

Every input ends in the same call, input.move(current, rule), so the navigation rules are defined once.

Many names for one rule ​

Speech and text need more than one name for each move. People say "drill in", "go in", "child", or "open". Each of these is its own navigation rule, added to the same edges as drill-in and pointing in the same direction. Nothing else changes. The commands below the chart list every name.

js
export const addAliases = structure => {
    Object.values(structure.edges).forEach(edge => {
        const rules = new Set(edge.navigationRules);
        commands.forEach(c => {
            if (c.alsoOn?.some(r => rules.has(r))) rules.add(c.rule);
        });
        [...rules].forEach(r => (aliases[r] || []).forEach(a => rules.add(a)));
        edge.navigationRules = [...rules];
    });
    return structure;
};

One position ​

Keyboard moves focus to the chart element, so a screen reader reads it. Other inputs move the same position through the text chat with chat.submit(), which logs the result and announces it. When the keyboard moves, chat.setCurrentNode() keeps the chat in sync. See Sharing a position with other inputs.

Camera, microphone, and swiping status (such as "No hand in view") goes to a separate status message, so it never repeats what the chat or focus already said.

Notes on each input ​

  • Voice is push to talk. Always-on listening can hear a screen reader read a result aloud and act on it again. Speech recognition works in Chrome, Edge, and Safari. Some browsers send audio to a server to recognize it.
  • Hand gestures and head movement use MediaPipe Tasks Vision, which runs in the browser, including on phones. Models download when you turn them on. Hold a gesture briefly to use it, then relax before the next one. Head movement measures your resting position first, so hold still for a moment after turning it on.
  • Touch swiping is off by default. Screen readers on phones use swipes for their own navigation, so swiping here would get in the way. Swiping also works with a mouse drag.

Focus indicators ​

Each node's outline matches what it represents: a box around a bar, a box around a year's column, an outline around each segment for a browser, and a box around the browser names for the browser grouping. See Focus Indication for how to build these, and Trouble with Focus Indication for charts drawn on a canvas.

The code ​

This demo is a ready-to-run project in assets/many-ways. Run npm install in that folder and serve it from the repository root.

js
import { data } from './data.js';
import { drawChart } from './chart.js';
import { buildStructure } from './structure.js';
import { createNavigator } from './navigator.js';
import { speechSupported, createSpeech } from './speech.js';
import { createCamera } from './vision.js';
import { startHand } from './hand.js';
import { startHead } from './head.js';
import { startSwipe } from './swipe.js';

export function mountManyWays(root) {
    const $ = id => root.querySelector(`#${id}`);
    const chartHost = $('mw-chart');
    const status = $('mw-status');
    const announce = text => {
        status.textContent = '';
        requestAnimationFrame(() => (status.textContent = text));
    };

    const geometry = drawChart(chartHost, data, { width: Math.min(640, chartHost.clientWidth || 640) });
    const nav = createNavigator({
        structure: buildStructure(data, geometry),
        description: 'Browser share chart',
        chartHost,
        chatHost: $('mw-chat'),
        inspectorHost: $('mw-inspector'),
        commandsHost: $('mw-commands'),
        onStatus: announce
    });

    const stops = {};
    const stop = way => {
        stops[way]?.();
        delete stops[way];
    };

    const speak = $('mw-speak');
    const voice = $('mw-voice');
    const speech = speechSupported()
        ? createSpeech({
              onText: nav.say,
              onStatus: announce,
              onChange: listening => (speak.textContent = listening ? 'Stop listening' : 'Speak a command')
          })
        : null;
    if (!speech) {
        voice.disabled = true;
        $('mw-voice-note').hidden = false;
    }
    voice.addEventListener('change', () => {
        speak.hidden = !voice.checked;
        if (!voice.checked) speech.stop();
        announce(voice.checked ? 'Voice on. Press Speak a command, then say a command.' : 'Voice off.');
    });
    speak.addEventListener('click', () => speech.listen());

    const video = $('mw-video');
    const camera = createCamera(video);
    const cameraPanel = $('mw-camera');
    const vision = {
        hand: { box: $('mw-hand'), start: startHand, name: 'Hand', ready: 'Show one hand to the camera.' },
        head: {
            box: $('mw-head'),
            start: startHead,
            name: 'Head',
            ready: 'Look at the camera and hold still for a moment.'
        }
    };
    const showCamera = () => (cameraPanel.hidden = !(vision.hand.box.checked || vision.head.box.checked));
    Object.entries(vision).forEach(([way, { box, start, name, ready }]) => {
        box.addEventListener('change', async () => {
            showCamera();
            if (!box.checked) {
                stop(way);
                return announce(`${name} tracking off.`);
            }
            announce(`Loading ${way} tracking.`);
            try {
                await camera.start();
                const end = await start(video, { onRule: rule => nav.run(rule, way), onStatus: announce });
                stops[way] = () => {
                    end();
                    camera.stop();
                };
                if (!box.checked) return stop(way);
                announce(`${name} tracking on. ${ready}`);
            } catch (error) {
                camera.stop();
                box.checked = false;
                showCamera();
                announce(
                    error.name === 'NotAllowedError' ? 'Camera access is blocked.' : `Couldn't start ${way} tracking.`
                );
            }
        });
    });

    const swipe = $('mw-swipe');
    swipe.addEventListener('change', () => {
        if (!swipe.checked) {
            stop('touch');
            return announce('Swiping off.');
        }
        stops.touch = startSwipe(chartHost, { onRule: rule => nav.run(rule, 'touch') });
        announce('Swiping on. Swipe, tap, or press on the chart.');
    });

    return () => {
        Object.keys(stops).forEach(stop);
        speech?.stop();
        nav.destroy();
        chartHost.querySelector('.mw-svg')?.remove();
    };
}
js
import dataNavigator from 'data-navigator';
import { Inspector } from '@data-navigator/inspector';
import { commands, commandLabels, tableRows, tableColumns } from './commands.js';

export function createNavigator({
    structure,
    description,
    chartHost,
    chatHost,
    inspectorHost,
    commandsHost,
    onStatus
}) {
    const elementData = {};
    Object.values(structure.nodes).forEach(node => {
        node.renderId = `mw-node-${node.id}`;
        elementData[node.renderId] = node;
    });
    const entryPoint = 'chart';
    let current = null;

    const rendering = dataNavigator.rendering({
        elementData,
        suffixId: 'mw',
        defaults: { cssClass: 'dn-node' },
        root: { id: chartHost.id, description, width: '100%', height: 0 },
        entryButton: { include: true, callbacks: { click: () => enter() } },
        exitElement: { include: true },
        commandsElement: {
            include: true,
            rootId: commandsHost.id,
            title: 'Commands for every input',
            commands: tableRows,
            columns: tableColumns
        }
    });
    rendering.initialize();

    const input = dataNavigator.input({
        structure,
        navigationRules: structure.navigationRules,
        entryPoint,
        exitPoint: rendering.exitElement.id
    });

    const width = Math.min(480, inspectorHost.clientWidth || 360);
    const inspector = Inspector({
        structure,
        container: inspectorHost,
        size: { width, height: Math.round(width * 0.75) },
        mode: 'tree'
    });

    const onKeydown = e => {
        const rule = input.keydownValidator(e);
        if (!rule) return;
        e.preventDefault();
        move(rule);
    };

    const show = (id, focus) => {
        const previous = current;
        current = id;
        const node = structure.nodes[id];
        let el = document.getElementById(node.renderId);
        if (!el) {
            el = rendering.render(node);
            el.addEventListener('keydown', onKeydown);
        }
        el.classList.add('is-current');
        if (focus) input.focus(node.renderId);
        if (previous && previous !== id) rendering.remove(structure.nodes[previous].renderId);
        inspector.highlight(id);
    };

    const clear = () => {
        current = null;
        rendering.clearStructure();
        inspector.clear();
    };

    const enter = () => {
        const node = input.enter();
        show(node.id, true);
        chat.setCurrentNode(node.id);
    };

    const exit = () => {
        clear();
        chat.setCurrentNode(null);
        rendering.exitElement.style.display = 'block';
        input.focus(rendering.exitElement.id);
    };

    const move = rule => {
        if (rule === 'exit') return exit();
        const next = input.move(current, rule);
        if (!next) {
            onStatus(`"${commandLabels[rule] || rule}" isn't available here.`);
            return;
        }
        show(next.id, true);
        chat.setCurrentNode(next.id);
    };

    const chat = dataNavigator.textChat({
        structure,
        container: chatHost,
        entryPoint,
        commandLabels,
        onNavigate: node => show(node.id, false),
        onExit: clear
    });

    const inChart = () => rendering.wrapper.contains(document.activeElement);

    const run = (rule, way) => {
        if (!current && rule !== 'drill-in') {
            const how = commands.find(c => c.rule === 'drill-in')[way];
            onStatus(`Not in the chart yet. To enter: ${how.toLowerCase()}.`);
            return;
        }
        if (inChart()) {
            current ? move(rule) : enter();
            return;
        }
        chat.submit(current ? rule : 'enter');
    };

    return {
        run,
        say: text => chat.submit(text),
        destroy() {
            chat.destroy();
            inspector.destroy();
            commandsHost.replaceChildren();
            rendering.wrapper.remove();
            rendering.exitElement.remove();
        }
    };
}
js
import { keyLabel } from 'data-navigator';

export const commands = [
    {
        rule: 'left',
        key: 'ArrowLeft',
        direction: 'source',
        action: 'Previous year, same browser',
        say: ['previous', 'previous year', 'backward', 'earlier'],
        hand: 'Point left',
        head: 'Turn left',
        touch: 'Swipe left'
    },
    {
        rule: 'right',
        key: 'ArrowRight',
        direction: 'target',
        action: 'Next year, same browser',
        say: ['next', 'next year', 'forward', 'later'],
        hand: 'Point right',
        head: 'Turn right',
        touch: 'Swipe right'
    },
    {
        rule: 'up',
        key: 'ArrowUp',
        direction: 'source',
        action: 'Browser above, same year',
        say: ['above', 'higher', 'browser above'],
        hand: 'Point up',
        head: 'Chin up',
        touch: 'Swipe up'
    },
    {
        rule: 'down',
        key: 'ArrowDown',
        direction: 'target',
        action: 'Browser below, same year',
        say: ['below', 'lower', 'browser below'],
        hand: 'Point down',
        head: 'Chin down',
        touch: 'Swipe down'
    },
    {
        rule: 'drill-in',
        key: 'Enter',
        direction: 'target',
        label: 'drill in',
        action: 'Drill in',
        say: ['in', 'child', 'go in', 'open', 'deeper', 'zoom in'],
        hand: 'Open palm',
        head: 'Open mouth',
        touch: 'Double tap'
    },
    {
        rule: 'drill-out',
        key: 'Backspace',
        direction: 'source',
        label: 'drill out',
        action: 'Up one level (to the year from a data point)',
        alsoOn: ['to-year', 'to-browser'],
        say: ['out', 'parent', 'back', 'go out', 'go back', 'zoom out', 'up a level'],
        hand: 'Thumb up',
        head: 'Close eyes for 1 second',
        touch: 'Long press'
    },
    {
        rule: 'to-year',
        key: 'KeyW',
        direction: 'source',
        label: 'up to the year',
        action: 'Up to the year',
        say: [
            'year',
            'column',
            'up to year',
            'drill out to year',
            'drill out to the year',
            'out to year',
            'out to the year',
            'back to year',
            'back to the year',
            'go to year',
            'go to the year'
        ],
        hand: 'Fist',
        head: 'Tilt left',
        touch: 'Not available'
    },
    {
        rule: 'to-browser',
        key: 'KeyJ',
        direction: 'source',
        label: 'up to the browser',
        action: 'Up to the browser',
        say: [
            'browser',
            'series',
            'row',
            'up to browser',
            'drill out to browser',
            'drill out to the browser',
            'out to browser',
            'out to the browser',
            'back to browser',
            'back to the browser',
            'go to browser',
            'go to the browser'
        ],
        hand: 'Victory (two fingers)',
        head: 'Tilt right',
        touch: 'Two-finger tap'
    },
    {
        rule: 'exit',
        key: 'Escape',
        direction: 'target',
        action: 'Exit the chart',
        say: [],
        hand: 'Thumb down',
        head: 'Frown',
        touch: 'Not available'
    }
];

const aliasName = phrase => phrase.replace(/\s+/g, '-');

export const navigationRules = {};
export const commandLabels = {};
export const aliases = {};

commands.forEach(c => {
    navigationRules[c.rule] = { key: c.key, direction: c.direction };
    if (c.label) commandLabels[c.rule] = c.label;
    aliases[c.rule] = c.say.map(aliasName).filter(name => name !== c.rule);
    c.say.forEach(phrase => {
        const name = aliasName(phrase);
        navigationRules[name] = { direction: c.direction };
        if (name !== phrase) commandLabels[name] = phrase;
    });
});

// #region aliases
export const addAliases = structure => {
    Object.values(structure.edges).forEach(edge => {
        const rules = new Set(edge.navigationRules);
        commands.forEach(c => {
            if (c.alsoOn?.some(r => rules.has(r))) rules.add(c.rule);
        });
        [...rules].forEach(r => (aliases[r] || []).forEach(a => rules.add(a)));
        edge.navigationRules = [...rules];
    });
    return structure;
};
// #endregion aliases

export const ruleFor = (field, value) => commands.find(c => c[field] === value)?.rule;

export const tableRows = commands.map(c => ({
    description: c.action,
    label: keyLabel(c.key),
    command: [c.label || c.rule, ...c.say].join(', '),
    hand: c.hand,
    head: c.head,
    touch: c.touch
}));

export const tableColumns = [
    { label: 'Action', id: 'description' },
    { label: 'Key', id: 'label' },
    { label: 'Say or type', id: 'command' },
    { label: 'Hand', id: 'hand' },
    { label: 'Head', id: 'head' },
    { label: 'Touch', id: 'touch' }
];
js
import dataNavigator, { boundingRectPath, unionOfRectPaths } from 'data-navigator';
import { navigationRules, addAliases } from './commands.js';
import { years, stacks, alt } from './chart.js';

const box = (rect, pad) => ({
    x: rect.x - pad,
    y: rect.y - pad,
    width: rect.width + pad * 2,
    height: rect.height + pad * 2,
    path: boundingRectPath([rect], pad)
});

const union = rects => {
    const x = Math.min(...rects.map(r => r.x));
    const y = Math.min(...rects.map(r => r.y));
    const width = Math.max(...rects.map(r => r.x + r.width)) - x;
    const height = Math.max(...rects.map(r => r.y + r.height)) - y;
    return { x, y, width, height };
};

export function buildStructure(data, { bars, boxes }) {
    // #region structure
    const topToBottom = [...stacks].reverse();
    const firstRow = division => Object.values(division.values)[0];
    const structure = dataNavigator.structure({
        data,
        idKey: 'id',
        navigationRules,
        dimensions: {
            parentOptions: { addLevel0: { id: 'chart', edges: [] } },
            values: [
                {
                    dimensionKey: 'year',
                    nodeId: 'years',
                    type: 'categorical',
                    divisionOptions: { divisionNodeIds: (key, value) => `year-${value}` },
                    behavior: { extents: 'circular', childmostNavigation: 'across' },
                    navigationRules: { sibling_sibling: ['left', 'right'], parent_child: ['to-year', 'drill-in'] }
                },
                {
                    dimensionKey: 'browser',
                    nodeId: 'browsers',
                    type: 'categorical',
                    divisionOptions: { divisionNodeIds: (key, value) => `browser-${value.toLowerCase()}` },
                    operations: {
                        sortFunction: (a, b) =>
                            topToBottom.indexOf(firstRow(a).browser) - topToBottom.indexOf(firstRow(b).browser)
                    },
                    behavior: { extents: 'circular', childmostNavigation: 'across' },
                    navigationRules: { sibling_sibling: ['up', 'down'], parent_child: ['to-browser', 'drill-in'] }
                }
            ]
        }
    });
    addAliases(structure);
    // #endregion structure

    // #region labels
    const n = structure.nodes;
    n.chart.semantics = { label: `${alt} Two groupings: year and browser.` };
    n.years.semantics = { label: 'Year. 3 years, 2015 to 2017.' };
    n.browsers.semantics = { label: 'Browser. 5 browsers, Chrome to Other.' };
    years.forEach(y => {
        const top = data.filter(d => d.year === y).sort((a, b) => b.share - a.share)[0];
        n[`year-${y}`].semantics = { label: `${y}. 5 browsers. ${top.browser} leads with ${top.share}%.` };
    });
    stacks.forEach(s => {
        const shares = data.filter(d => d.browser === s).map(d => d.share);
        const lo = Math.min(...shares);
        const hi = Math.max(...shares);
        n[`browser-${s.toLowerCase()}`].semantics = {
            label: `${s}. 3 years, ${lo === hi ? `${lo}% each year` : `${lo}% to ${hi}%`}.`
        };
    });
    data.forEach(d => {
        n[d.id].semantics = { label: `${d.share}%, ${d.browser}, ${d.year}` };
    });
    // #endregion labels

    // #region focus
    const all = Object.values(bars);
    n.chart.spatialProperties = box(boxes.chart, 0);
    n.years.spatialProperties = box(union([...all, boxes.xAxis]), 4);
    n.browsers.spatialProperties = box(boxes.labels, 4);
    years.forEach(y => {
        n[`year-${y}`].spatialProperties = box(union(data.filter(d => d.year === y).map(d => bars[d.id])), 4);
    });
    stacks.forEach(s => {
        const segments = data.filter(d => d.browser === s).map(d => bars[d.id]);
        n[`browser-${s.toLowerCase()}`].spatialProperties = {
            ...union(segments),
            path: unionOfRectPaths(segments, 3)
        };
    });
    data.forEach(d => {
        n[d.id].spatialProperties = box(bars[d.id], 2);
    });
    // #endregion focus

    return structure;
}
js
import * as d3 from 'd3';

export const years = ['2015', '2016', '2017'];
export const stacks = ['Chrome', 'Firefox', 'Safari', 'Edge', 'Other'];
export const alt =
    'Stacked bar chart of browser share from 2015 to 2017. Chrome holds most of each year, rising from 62 to 65 percent.';

export function drawChart(host, data, { width, height = 320 }) {
    const svg = d3
        .select(host)
        .append('svg')
        .attr('class', 'mw-svg')
        .attr('width', width)
        .attr('height', height)
        .attr('role', 'img')
        .attr('aria-label', alt);

    // each browser is named beside the last column, so the labels set the right margin
    const labels = svg.append('g').attr('class', 'mw-labels');
    const texts = stacks.map(name => labels.append('text').text(name));
    const labelWidth = Math.max(...texts.map(t => t.node().getComputedTextLength()));

    const margin = { top: 16, right: labelWidth + 24, bottom: 36, left: 48 };
    const x = d3.scaleBand(years, [margin.left, width - margin.right]).padding(0.3);
    const y = d3.scaleLinear([0, 100], [height - margin.bottom, margin.top]);

    svg.append('g')
        .attr('class', 'mw-axis mw-grid')
        .attr('transform', `translate(${margin.left},0)`)
        .call(
            d3
                .axisLeft(y)
                .ticks(5)
                .tickSize(-(width - margin.left - margin.right))
                .tickFormat(d => `${d}%`)
        )
        .call(g => g.select('.domain').remove());

    const rows = years.map(year => ({
        year,
        ...Object.fromEntries(data.filter(d => d.year === year).map(d => [d.browser, d.share]))
    }));
    const bars = {};
    d3.stack()
        .keys(stacks)(rows)
        .forEach(series =>
            series.forEach(([y0, y1], i) => {
                bars[`${years[i]}-${series.key.toLowerCase()}`] = {
                    key: series.key,
                    index: series.index,
                    x: x(years[i]),
                    y: y(y1),
                    width: x.bandwidth(),
                    height: y(y0) - y(y1)
                };
            })
        );

    // outlines sit inside each segment, with a 2px gap between segments
    svg.append('g')
        .selectAll('rect')
        .data(Object.values(bars))
        .join('rect')
        .attr('class', d => `mw-bar mw-c${d.index}`)
        .attr('x', d => d.x + 0.5)
        .attr('y', d => d.y + 0.5)
        .attr('width', d => d.width - 1)
        .attr('height', d => Math.max(0.5, d.height - 1 - (d.index ? 2 : 0)));

    svg.append('g')
        .attr('class', 'mw-axis')
        .attr('transform', `translate(0,${height - margin.bottom})`)
        .call(d3.axisBottom(x).tickSizeOuter(0));

    // place the labels at their segments' middles, pushed apart where segments are thin
    const last = years[years.length - 1];
    const ends = stacks
        .map((s, i) => {
            const b = bars[`${last}-${s.toLowerCase()}`];
            return { i, y: b.y + b.height / 2 };
        })
        .sort((a, b) => a.y - b.y);
    ends.forEach((e, k) => k && (e.y = Math.max(e.y, ends[k - 1].y + 14)));
    ends.forEach(e => texts[e.i].attr('x', x(last) + x.bandwidth() + 8).attr('y', e.y + 4));
    labels.raise();
    const labelBox = labels.node().getBBox();

    return {
        bars,
        boxes: {
            chart: { x: 2, y: 2, width: width - 4, height: height - 4 },
            xAxis: { x: margin.left, y: height - margin.bottom, width: width - margin.left - margin.right, height: 28 },
            labels: { x: labelBox.x, y: labelBox.y, width: labelBox.width, height: labelBox.height }
        }
    };
}
js
const Recognition = typeof window !== 'undefined' && (window.SpeechRecognition || window.webkitSpeechRecognition);

export const speechSupported = () => !!Recognition;

export function createSpeech({ onText, onStatus, onChange }) {
    let recognition = null;

    const stop = () => recognition?.abort();

    const listen = () => {
        if (recognition) return stop();
        recognition = new Recognition();
        recognition.lang = document.documentElement.lang || 'en-US';
        recognition.continuous = false;
        recognition.interimResults = false;
        recognition.maxAlternatives = 1;
        let heard = false;
        recognition.onstart = () => {
            onChange(true);
            onStatus('Listening.');
        };
        recognition.onresult = e => {
            heard = true;
            const text = e.results[0][0].transcript
                .toLowerCase()
                .replace(/[^\p{L}\p{N}\s-]/gu, '')
                .trim();
            onStatus('');
            onText(text);
        };
        recognition.onerror = e => {
            heard = true;
            if (e.error === 'aborted') return onStatus('Stopped listening.');
            if (e.error === 'not-allowed') return onStatus('Microphone access is blocked.');
            if (e.error === 'no-speech') return onStatus("Didn't hear anything. Try again.");
            onStatus(`Speech recognition stopped: ${e.error}.`);
        };
        recognition.onend = () => {
            if (!heard) onStatus("Didn't catch that. Try again.");
            recognition = null;
            onChange(false);
        };
        recognition.start();
    };

    return { listen, stop };
}
js
import { createGestureRecognizer, everyFrame } from './vision.js';

const HOLD_MS = 400;
const NO_HAND_MS = 2000;
const named = {
    Open_Palm: 'drill-in',
    Closed_Fist: 'to-year',
    Victory: 'to-browser',
    Thumb_Up: 'drill-out',
    Thumb_Down: 'exit'
};

const dist = (a, b) => Math.hypot(a.x - b.x, a.y - b.y);
const extended = (hand, tip, joint) => dist(hand[tip], hand[0]) > dist(hand[joint], hand[0]);
const pointing = hand =>
    extended(hand, 8, 6) && !extended(hand, 12, 10) && !extended(hand, 16, 14) && !extended(hand, 20, 18);

const pointDirection = hand => {
    const dx = hand[8].x - hand[5].x;
    const dy = hand[8].y - hand[5].y;
    if (Math.abs(dx) > Math.abs(dy)) return dx > 0 ? 'left' : 'right';
    return dy > 0 ? 'down' : 'up';
};

const read = result => {
    const hand = result.landmarks[0];
    const gesture = result.gestures[0]?.[0];
    if (gesture && named[gesture.categoryName] && gesture.score > 0.6) return named[gesture.categoryName];
    if (pointing(hand)) return pointDirection(hand);
    return null;
};

export async function startHand(video, { onRule, onStatus }) {
    const recognizer = await createGestureRecognizer();
    let candidate = null;
    let since = 0;
    let fired = null;
    let lastSeen = performance.now();
    let missing = false;

    const stopLoop = everyFrame(video, now => {
        const result = recognizer.recognizeForVideo(video, now);
        if (!result.landmarks.length) {
            candidate = fired = null;
            if (!missing && now - lastSeen > NO_HAND_MS) {
                missing = true;
                onStatus('No hand in view.');
            }
            return;
        }
        lastSeen = now;
        if (missing) {
            missing = false;
            onStatus('Hand found.');
        }
        const rule = read(result);
        if (rule !== candidate) {
            candidate = rule;
            since = now;
        }
        if (!rule) {
            fired = null;
            return;
        }
        if (rule !== fired && now - since >= HOLD_MS) {
            fired = rule;
            onRule(rule);
        }
    });

    return () => {
        stopLoop();
        recognizer.close();
    };
}
js
import { createFaceLandmarker, everyFrame } from './vision.js';

const NO_FACE_MS = 2000;
const CALIBRATION_FRAMES = 15;
const holdMs = { 'drill-out': 1000, exit: 500 };
const limits = { turn: 0.15, nod: 0.12, tilt: 0.25, eyesClosed: 0.5, frown: 0.3, mouthOpen: 0.45 };

const pose = (face, aspect) => {
    const [a, b, nose] = [face[33], face[263], face[1]].map(p => ({ x: p.x * aspect, y: p.y }));
    const tilt = Math.atan2(b.y - a.y, b.x - a.x);
    const eyes = Math.hypot(b.x - a.x, b.y - a.y);
    const dx = (nose.x - (a.x + b.x) / 2) / eyes;
    const dy = (nose.y - (a.y + b.y) / 2) / eyes;
    const cos = Math.cos(-tilt);
    const sin = Math.sin(-tilt);
    return { turn: dx * cos - dy * sin, nod: dx * sin + dy * cos, tilt };
};

const scores = result =>
    Object.fromEntries((result.faceBlendshapes[0]?.categories || []).map(c => [c.categoryName, c.score]));

export async function startHead(video, { onRule, onStatus }) {
    const landmarker = await createFaceLandmarker();
    const samples = [];
    let neutral = null;
    let candidate = null;
    let since = 0;
    let ready = true;
    let lastSeen = performance.now();
    let missing = false;

    const read = (p, s) => {
        if (Math.min(s.eyeBlinkLeft, s.eyeBlinkRight) > limits.eyesClosed) return 'drill-out';
        if ((s.mouthFrownLeft + s.mouthFrownRight) / 2 > limits.frown) return 'exit';
        if (s.jawOpen > limits.mouthOpen) return 'drill-in';
        const turn = p.turn - neutral.turn;
        const nod = p.nod - neutral.nod;
        const tilt = p.tilt - neutral.tilt;
        if (Math.abs(tilt) > limits.tilt) return tilt > 0 ? 'to-year' : 'to-browser';
        if (Math.abs(turn) / limits.turn > Math.abs(nod) / limits.nod) {
            if (Math.abs(turn) > limits.turn) return turn > 0 ? 'left' : 'right';
        } else if (Math.abs(nod) > limits.nod) {
            return nod > 0 ? 'down' : 'up';
        }
        return null;
    };

    const stopLoop = everyFrame(video, now => {
        const result = landmarker.detectForVideo(video, now);
        const face = result.faceLandmarks[0];
        if (!face) {
            candidate = null;
            if (!missing && now - lastSeen > NO_FACE_MS) {
                missing = true;
                onStatus('No face in view.');
            }
            return;
        }
        lastSeen = now;
        if (missing) {
            missing = false;
            onStatus('Face found.');
        }
        const p = pose(face, video.videoWidth / video.videoHeight);
        if (!neutral) {
            samples.push(p);
            if (samples.length < CALIBRATION_FRAMES) return;
            neutral = Object.fromEntries(
                Object.keys(p).map(k => [k, samples.reduce((sum, s) => sum + s[k], 0) / samples.length])
            );
            onStatus('Head tracking ready.');
            return;
        }
        const rule = read(p, scores(result));
        if (!rule) {
            candidate = null;
            ready = true;
            return;
        }
        if (rule !== candidate) {
            candidate = rule;
            since = now;
        }
        if (ready && now - since >= (holdMs[rule] || 250)) {
            ready = false;
            onRule(rule);
        }
    });

    return () => {
        stopLoop();
        landmarker.close();
    };
}
js
const SWIPE_PX = 30;
const TAP_PX = 10;
const DOUBLE_TAP_MS = 300;
const LONG_PRESS_MS = 600;

export function startSwipe(surface, { onRule }) {
    const pointers = new Map();
    let start = null;
    let twoFingers = false;
    let lastTap = 0;
    let pressTimer = 0;

    const reset = () => {
        clearTimeout(pressTimer);
        start = null;
        twoFingers = false;
    };

    const down = e => {
        pointers.set(e.pointerId, e);
        surface.setPointerCapture(e.pointerId);
        if (pointers.size === 2) {
            twoFingers = true;
            clearTimeout(pressTimer);
            return;
        }
        if (pointers.size > 2) return reset();
        start = { x: e.clientX, y: e.clientY };
        pressTimer = setTimeout(() => {
            onRule('drill-out');
            start = null;
        }, LONG_PRESS_MS);
    };

    const move = e => {
        if (start && Math.hypot(e.clientX - start.x, e.clientY - start.y) > TAP_PX) clearTimeout(pressTimer);
    };

    const up = e => {
        pointers.delete(e.pointerId);
        clearTimeout(pressTimer);
        if (pointers.size) return;
        if (twoFingers) {
            onRule('to-browser');
            return reset();
        }
        if (!start) return;
        const dx = e.clientX - start.x;
        const dy = e.clientY - start.y;
        reset();
        if (Math.max(Math.abs(dx), Math.abs(dy)) >= SWIPE_PX) {
            onRule(Math.abs(dx) > Math.abs(dy) ? (dx > 0 ? 'right' : 'left') : dy > 0 ? 'down' : 'up');
        } else if (e.timeStamp - lastTap < DOUBLE_TAP_MS) {
            lastTap = 0;
            onRule('drill-in');
        } else {
            lastTap = e.timeStamp;
        }
    };

    const cancel = e => {
        pointers.delete(e.pointerId);
        reset();
    };

    surface.classList.add('is-swiping');
    surface.addEventListener('pointerdown', down);
    surface.addEventListener('pointermove', move);
    surface.addEventListener('pointerup', up);
    surface.addEventListener('pointercancel', cancel);

    return () => {
        reset();
        pointers.clear();
        surface.classList.remove('is-swiping');
        surface.removeEventListener('pointerdown', down);
        surface.removeEventListener('pointermove', move);
        surface.removeEventListener('pointerup', up);
        surface.removeEventListener('pointercancel', cancel);
    };
}
js
const BASE = 'https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@1.1.1';
const MODELS = 'https://storage.googleapis.com/mediapipe-models';

let loading = null;
const loadVision = () => {
    loading ||= (async () => {
        const vision = await import(/* @vite-ignore */ `${BASE}/vision_bundle.mjs`);
        const fileset = await vision.FilesetResolver.forVisionTasks(`${BASE}/wasm`);
        return { vision, fileset };
    })();
    return loading;
};

const create = async (Task, model, options) => {
    const { fileset } = await loadVision();
    const base = { runningMode: 'VIDEO', ...options };
    try {
        return await Task.createFromOptions(fileset, {
            ...base,
            baseOptions: { modelAssetPath: model, delegate: 'GPU' }
        });
    } catch {
        return Task.createFromOptions(fileset, { ...base, baseOptions: { modelAssetPath: model, delegate: 'CPU' } });
    }
};

export const createGestureRecognizer = async () => {
    const { vision } = await loadVision();
    return create(
        vision.GestureRecognizer,
        `${MODELS}/gesture_recognizer/gesture_recognizer/float16/1/gesture_recognizer.task`,
        {
            numHands: 1
        }
    );
};

export const createFaceLandmarker = async () => {
    const { vision } = await loadVision();
    return create(vision.FaceLandmarker, `${MODELS}/face_landmarker/face_landmarker/float16/1/face_landmarker.task`, {
        numFaces: 1,
        outputFaceBlendshapes: true
    });
};

export function createCamera(video) {
    let stream = null;
    let users = 0;
    return {
        async start() {
            users++;
            if (stream) return;
            stream = await navigator.mediaDevices.getUserMedia({
                video: { facingMode: 'user', width: { ideal: 640 }, height: { ideal: 480 } },
                audio: false
            });
            video.muted = true;
            video.srcObject = stream;
            await video.play();
        },
        stop() {
            users = Math.max(0, users - 1);
            if (users || !stream) return;
            stream.getTracks().forEach(track => track.stop());
            stream = null;
            video.srcObject = null;
        }
    };
}

export function everyFrame(video, step) {
    let id = 0;
    let last = -1;
    const tick = () => {
        if (video.readyState >= 2 && video.currentTime !== last) {
            last = video.currentTime;
            step(performance.now());
        }
        id = requestAnimationFrame(tick);
    };
    tick();
    return () => cancelAnimationFrame(id);
}
html
<!doctype html>
<html lang="en">
    <head>
        <meta charset="utf-8" />
        <meta name="viewport" content="width=device-width, initial-scale=1" />
        <title>Many ways to navigate a chart</title>
        <link rel="stylesheet" href="./node_modules/data-navigator/text-chat.css" />
        <link rel="stylesheet" href="../../packages/inspector/style.css" />
        <link rel="stylesheet" href="./src/style.css" />
        <script>
            if (matchMedia('(prefers-color-scheme: dark)').matches) document.documentElement.classList.add('dark');
        </script>
        <script type="importmap">
            {
                "imports": {
                    "data-navigator": "./node_modules/data-navigator/dist/index.js",
                    "@data-navigator/inspector": "../../packages/inspector/src/inspector.js",
                    "d3": "https://cdn.jsdelivr.net/npm/d3@7.9.0/+esm",
                    "d3-array": "https://cdn.jsdelivr.net/npm/d3-array@3/+esm",
                    "d3-drag": "https://cdn.jsdelivr.net/npm/d3-drag@3/+esm",
                    "d3-force": "https://cdn.jsdelivr.net/npm/d3-force@3/+esm",
                    "d3-scale": "https://cdn.jsdelivr.net/npm/d3-scale@4/+esm",
                    "d3-scale-chromatic": "https://cdn.jsdelivr.net/npm/d3-scale-chromatic@3/+esm",
                    "d3-selection": "https://cdn.jsdelivr.net/npm/d3-selection@3/+esm"
                }
            }
        </script>
        <style>
            body {
                max-width: 760px;
                margin: 0 auto;
                padding: 0 16px;
                font-family: system-ui, sans-serif;
                background: Canvas;
                color: CanvasText;
            }
        </style>
    </head>
    <body>
        <main>
            <h1>Many ways to navigate a chart</h1>
            <div class="many-ways" id="mw-demo">
                <fieldset class="mw-ways">
                    <legend>Ways to navigate</legend>
                    <p class="mw-hint">Keyboard and text are always on. Turn on any others.</p>
                    <label><input type="checkbox" id="mw-voice" aria-describedby="mw-voice-note" /> Voice</label>
                    <label><input type="checkbox" id="mw-hand" /> Hand gestures</label>
                    <label><input type="checkbox" id="mw-head" /> Head movement</label>
                    <label
                        ><input type="checkbox" id="mw-swipe" aria-describedby="mw-swipe-note" /> Touch swiping</label
                    >
                    <p class="mw-hint" id="mw-voice-note" hidden>This browser does not support speech recognition.</p>
                    <p class="mw-hint" id="mw-swipe-note">
                        Swiping takes over touch on the chart. Leave it off if you use a screen reader.
                    </p>
                </fieldset>
                <div class="mw-camera" id="mw-camera" hidden>
                    <video id="mw-video" muted playsinline aria-label="Camera preview"></video>
                    <p class="mw-hint">Video is processed on your device and is not sent anywhere.</p>
                </div>
                <p class="mw-status" id="mw-status" role="status"></p>
                <div class="mw-chart" id="mw-chart"></div>
                <div>
                    <p class="mw-label" id="mw-chat-label">Text chat</p>
                    <button type="button" class="mw-speak" id="mw-speak" hidden>Speak a command</button>
                    <div id="mw-chat" role="group" aria-labelledby="mw-chat-label"></div>
                </div>
                <details class="mw-structure">
                    <summary>View navigation structure</summary>
                    <div class="mw-inspector" id="mw-inspector"></div>
                </details>
                <div class="mw-commands" id="mw-commands"></div>
            </div>
        </main>
        <script type="module">
            import { mountManyWays } from './src/main.js';
            mountManyWays(document.getElementById('mw-demo'));
        </script>
    </body>
</html>
css
.many-ways {
    --mw-text: #0b0b0b;
    --mw-muted: #52514e;
    --mw-grid: #e6e5e0;
    --mw-surface: var(--vp-c-bg, #fff);
    --mw-soft: var(--vp-c-bg-soft, #f6f6f7);
    --mw-divider: var(--vp-c-divider, #d4d4d4);
    --mw-focus: var(--mw-text);
    --mw-c0-fill: #f4f8fd;
    --mw-c0-stroke: #2a78d6;
    --mw-c1-fill: #fdf4f8;
    --mw-c1-stroke: #e46293;
    --mw-c2-fill: #f4fdfa;
    --mw-c2-stroke: #1aa674;
    --mw-c3-fill: #fffbf2;
    --mw-c3-stroke: #c48500;
    --mw-c4-fill: #fef6f3;
    --mw-c4-stroke: #eb6834;
    display: grid;
    grid-template-columns: minmax(0, 1fr);
    gap: 1rem;
    margin: 1.5rem 0 2rem;
    padding: 1rem;
    border: 1px solid var(--mw-divider);
    border-radius: 8px;
    background: var(--mw-soft);
    color: var(--mw-text);
}

.dark .many-ways {
    --mw-text: #f0efec;
    --mw-muted: #c3c2b7;
    --mw-grid: #2e2e2c;
    --mw-c0-fill: #134c90;
    --mw-c0-stroke: #5c9cea;
    --mw-c1-fill: #832044;
    --mw-c1-stroke: #dc7097;
    --mw-c2-fill: #116a4b;
    --mw-c2-stroke: #1fc78d;
    --mw-c3-fill: #a36c00;
    --mw-c3-stroke: #ffcd6b;
    --mw-c4-fill: #8b3918;
    --mw-c4-stroke: #e48a66;
}

.mw-ways {
    display: flex;
    flex-wrap: wrap;
    gap: 0.5rem 1rem;
    margin: 0;
    padding: 0;
    border: 0;
}

.mw-ways legend {
    padding: 0;
    margin-bottom: 0.25rem;
    font-weight: 600;
}

.mw-ways label {
    display: inline-flex;
    align-items: center;
    gap: 0.4rem;
    min-height: 44px;
}

.mw-ways input {
    width: 1.1rem;
    height: 1.1rem;
    margin: 0;
}

.many-ways .mw-hint {
    flex-basis: 100%;
    margin: 0;
    font-size: 0.875rem;
    color: var(--mw-muted);
}

.mw-camera[hidden],
.mw-speak[hidden],
.mw-hint[hidden] {
    display: none;
}

.mw-camera video {
    display: block;
    width: 100%;
    max-width: 320px;
    aspect-ratio: 4 / 3;
    border-radius: 6px;
    background: #000;
    transform: scaleX(-1);
}

.many-ways .mw-status {
    min-height: 1.5em;
    margin: 0;
    font-size: 0.875rem;
}

.mw-chart {
    position: relative;
    max-width: 640px;
    background: var(--mw-surface);
    border-radius: 6px;
}

.mw-chart.is-swiping {
    touch-action: none;
    user-select: none;
    -webkit-user-select: none;
    outline: 2px dashed var(--mw-muted);
    outline-offset: 2px;
}

.mw-svg {
    display: block;
    font-size: 12px;
}

.mw-svg text {
    fill: var(--mw-text);
}

.mw-axis {
    color: var(--mw-muted);
    font-family: inherit;
    font-size: 12px;
}

.mw-grid line {
    stroke: var(--mw-grid);
}

.mw-bar {
    stroke-width: 1;
}

.mw-c0 {
    fill: var(--mw-c0-fill);
    stroke: var(--mw-c0-stroke);
}
.mw-c1 {
    fill: var(--mw-c1-fill);
    stroke: var(--mw-c1-stroke);
}
.mw-c2 {
    fill: var(--mw-c2-fill);
    stroke: var(--mw-c2-stroke);
}
.mw-c3 {
    fill: var(--mw-c3-fill);
    stroke: var(--mw-c3-stroke);
}
.mw-c4 {
    fill: var(--mw-c4-fill);
    stroke: var(--mw-c4-stroke);
}

.many-ways .dn-node {
    position: absolute;
    box-sizing: border-box;
    margin: 0;
}

.many-ways .dn-node:focus {
    outline: none;
}

.many-ways .dn-node-svg {
    position: absolute;
    pointer-events: none;
    overflow: visible;
}

.many-ways .dn-node-path {
    fill: none;
    stroke: none;
}

.many-ways .dn-node.is-current .dn-node-path {
    stroke: var(--mw-focus);
    stroke-width: 3px;
}

.many-ways .dn-node.is-current .dn-node-svg {
    filter: drop-shadow(0 0 1px var(--mw-surface)) drop-shadow(0 0 1px var(--mw-surface));
}

.many-ways .dn-node-text {
    position: absolute;
    width: 1px;
    height: 1px;
    margin: -1px;
    overflow: hidden;
    clip: rect(0, 0, 0, 0);
    white-space: nowrap;
}

.many-ways .dn-entry-button,
.mw-speak,
.mw-structure summary {
    min-height: 44px;
    margin: 0.5rem 0;
    padding: 0.5rem 1rem;
    border: 1px solid var(--mw-muted);
    border-radius: 6px;
    background: var(--mw-surface);
    color: var(--mw-text);
    font: inherit;
    cursor: pointer;
}

.many-ways .dn-exit-position {
    position: absolute;
    top: 1rem;
    left: 1rem;
    padding: 0.5rem;
    border: 1px solid var(--mw-muted);
    border-radius: 4px;
    background: var(--mw-surface);
}

.mw-structure summary {
    display: list-item;
    margin: 0;
}

.mw-structure .mw-inspector {
    margin-top: 0.5rem;
}

.many-ways .mw-label {
    margin: 0 0 0.5rem;
    font-weight: 600;
}

.mw-inspector svg {
    max-width: 100%;
    height: auto;
    border-radius: 6px;
    background: var(--mw-surface);
}

.mw-inspector .dn-inspector-tooltip {
    position: static !important;
    transform: none !important;
    max-width: 100%;
    margin-top: 0.5rem;
    background: var(--mw-surface) !important;
    color: var(--mw-text) !important;
}

.mw-commands {
    overflow-x: auto;
}

.mw-commands table {
    display: table;
    width: 100%;
    margin: 0;
    font-size: 0.8125rem;
    border-collapse: collapse;
}

.mw-commands caption {
    margin-bottom: 0.5rem;
    font-weight: 600;
    text-align: left;
}

.mw-commands th,
.mw-commands td {
    padding: 0.35rem 0.5rem;
    border: 1px solid var(--mw-divider);
    text-align: left;
    vertical-align: top;
}

Released under the MIT License.