| 209 |
Feeding the Audio Graph |
In 2004, I was given an iPod.
I count this as one of the most intuitive pieces of technology I’ve ever owned. It wasn’t because of the the snazzy (colour!) menus or circular touchpad. I loved how smoothly it fitted into my life. I could plug in my headphones and listen to music while I was walking around town. Then when I got home, I could plug it into an amplifier and carry on listening there.
There was no faff. It didn’t matter if I could find my favourite mix tape, or if my WiFi was flakey - it was all just there.
Nowadays, when I’m trying to pair my phone with some Bluetooth speakers, or can’t find my USB-to-headphone jack, or even access any music because I don’t have cellular reception; I really miss this simplicity.
The Web Audio API
I think the Web Audio API feels kind of like my iPod did.
It’s different from most browser APIs - rather than throwing around data, or updating DOM elements - you plug together a graph of audio nodes, which the browser uses to generate, process, and play sounds.
The thing I like about it is that you can totally plug it into whatever you want, and it’ll mostly just work.
So, let’s get started. First of all we want an audio source.
<audio src="night-owl.mp3" controls />
(Song - Night Owl by Broke For Free)
This totally works. However, it’s not using the Web Audio API, so we can’t access or modify the sound it makes.
To hook this up to our audio graph, we can use an AudioSourceNode. This captures the sound from the element, and lets us connect to other nodes in a graph.
const audioCtx = new AudioContext()
const audio = document.querySelector('audio')
const input = audioCtx.createAudioSourceNode(audio)
input.connect(audioCtx.destination)
Great. We’ve made something that looks and sounds exactly the same as it did before. Go us.
Gain
Let’s plug in a GainNode - this allows you to alter the amplitude (volume) of an an audio stream.
We can hook this node up to an <input> element by setting the gain property of the node. (The syntax for this is kind of weird because it’s an AudioParam which has options to set values at precise intervals).
const node = audioCtx.createGain()
const input = document.querySelector('input')
input.oninput = () => node.gain.value = parseFloat(input.value)
input.connect(node)
node.connect(audioCtx.destination)
You can now see a range input, which can be dragged to update the state of our graph. This input could be any kind of element, so now you’ll be free to build the volume control of your dreams.
There’s a number of nodes that let you modify/filter an audio stream in more interesting ways. Head over to the MDN Web Audio page for a list of them.
Analysers
Something else we can add to our graph is an AnalyserNode. This doesn’t modify the audio at all, but allows us to inspect the sounds that are flowing through it. We can put this into our graph between our AudioSourceNode and the GainNode.
const analyser = audioCtx.createAnalyser()
input.connect(analyser)
analyser.connect(gain)
gain.connect(audioCtx.destination)
And now we have an analyser. We can access it from elsewhere to drive any kind of visuals. For instance, if we wanted to draw lines on a canvas we could totally do that:
const waveform = new Uint8Array(analyser.fftSize)
const frequencies = new Uint8Array(analyser.frequencyBinCount)
const ctx = canvas.getContext('2d')
const loop = () => {
requestAnimationFrame(loop)
analyser.getByteTimeDomainData(waveform)
analyser.getByteFrequencyData(frequencies)
ctx.beginPath()
waveform.forEach((f, i) => ctx.lineTo(i, f))
ctx.lineTo(0,255)
frequencies.forEach((f, i) => ctx.lineTo(i, 255-f))
ctx.stroke()
}
loop()
You can see that we have two arrays of data available (I added colours for clarity):
The waveform - the raw samples of the audio being played.
The frequencies - a fourier transform of the audio passing through the node.
What’s cool about this is that you’re not tied to any specific functionality of the Web Audio API. If it’s possible for you to update something with an array of numbers, then you can just apply it to the output of the analyser node.
For instance, if we wanted to, we could definitely animate a list of emoji in time with our music.
spans.forEach(
(s, i) => s.style.transform = `scale(${1 + (frequencies[i]/100)})`
)
🔈🎤🎤🎤🎺🎷📯🎶🔊🎸🎺🎤🎸🎼🎷🎺🎻🎸🎻🎺🎸🎶🥁🎶🎵🎵🎷📯🎸🎹🎤🎷🎻🎷🔈🔊📯🎼🎤🎵🎼📯🥁🎻🎻🎤🔉🎵🎹🎸🎷🔉🔈🔉🎷🎶🔈🎸🎸🎻🎤🥁🎼📯🎸🎸🎼🎸🥁🎼🎶🎶🥁🎤🔊🎷🔊🔈🎺🔊🎻🎵🎻🎸🎵🎺🎤🎷🎸🎶🎼📯🔈🎺🎤🎵🎸🎸🔊🎶🎤🥁🎵🎹🎸🔈🎻🔉🥁🔉🎺🔊🎹🥁🎷📯🎷🎷🎤🎸🔉🎹🎷🎸🎺🎼🎤🎼🎶🎷🎤🎷📯📯🎻🎤🎷📯🎹🔈🎵🎹🎼🔊🔉🔉🔈🎶🎸🥁🎺🔈🎷🎵🔉🥁🎷🎹🎷🔊🎤🎤🔊🎤🎤🎹🎸🎹🔉🎷
Generating Audio
So far, we’ve been using the <audio> element as a source of sound.
There’s a few other sources of audio that we can use. We’ll look at the AudioBufferNode - which allows you to manually generate a sound sample, and then connect it to our graph.
First we have to create an AudioBuffer, which holds our raw data, then we pass that to an AudioBufferNode which we can then treat just like our AudioSource node. This can get a bit boring, so we’ll use a helper method that makes it simpler to generate sounds.
const generator = (audioCtx, target) => (seconds, fn) => {
const { sampleRate } = audioCtx
const buffer = audioCtx.createBuffer(
1, sampleRate * seconds, sampleRate
)
const data = buffer.getChannelData(0)
for (var i = 0; i < data.length; i++) {
data[i] = fn(i / sampleRate, seconds)
}
return () => {
const source = audioCtx.createBufferSource()
source.buffer = audioBuffer
source.connect(target || audioCtx.destination)
source.start()
}
}
const sound = generator(audioCtx, gain)
Our wrapper will let us provide a function that maps time (in seconds) to a sample (between 1 and -1). This generates a waveform, like we saw before with the analyser node.
For example, the following will generate 0.75 seconds of white noise at 20% volume.
const noise = sound(0.75, t => Math.random() * 0.2)
button.onclick = noise
Noise
Now we’ve got a noisy button! Handy.
Rather than having a static set of audio nodes, each time we click the button, we add a new node to our graph. Although this feels inefficient, it’s not actually too bad - the browser can do a good job of cleaning up old nodes once they’ve played.
An interesting property of defining sounds as functions is that we can combine multiple function to generate new sounds. So if we wanted to fade our noise in and out, we could write a higher order function that does that.
const ease = fn => (t, s) =>
fn(t) * Math.sin((t / s) * Math.PI)
const noise = sound(0.75, ease(t => Math.random() * 0.2))
ease(noise)
And we can do more than just white noise - if we use Math.sin, we can generate some nice pure tones.
// Math.sin with period of 0..1
const wave = v => Math.sin(Math.PI * 2 * v)
const hz = f => t => wave(t * f)
const _440hz = sound(0.75, ease(hz(440)))
const _880hz = sound(0.75, ease(hz(880)))
440Hz
880Hz
We can also make our functions more complex. Below we’re combining several frequencies to make a richer sounding tone.
const harmony = f => [4, 3, 2, 1].reduce(
(v, h, i) => (sin(f * h) * (i+1) ) + v
)
const a440 = sound(0.75, ease(harmony(440)))
440Hz
880Hz
Cool.
We’re still not using any audio-specific functionality, so we can repurpose anything that does an operation on data. For example, we can use d3.js - usually used for interactive data visualisations - to generate a triangular waveform.
const triangle = d3.scaleLinear()
.domain([0, .5, 1])
.range([-1, 1, -1])
const wave = t => triangle(t % 1)
const a440 = sound(0.75, ease(harmony(440)))
440Hz
880Hz
It’s pretty interesting to play around with different functions. I’ve plonked everything in jsbin if you want to have a play yourself.
A departure from best practice
We’ve been generating our audio from scratch, but most of what we’ve looked at can be implemented by a series of native Web Audio nodes. This would be way performant (because it’s not happening on the main thread), and more flexible in some ways (because you can set timings dynamically whilst the note is playing). But we’re going to stay with this approach because it’s fun, and sometimes the fun thing to do might not technically be the best thing to do.
Making a keyboard
Having a button that makes a sound is totally great, but how about lots of buttons that make lots of sounds? Yup, totally greater-er.
The first thing we need to know is the frequency of each note. I thought this would be awkward because pianos were invented more than 250 years before the Hz unit was defined, so surely there wouldn’t be a simple mapping between the two?
const freq = note => 27.5 * Math.pow(2, (note - 21) / 12)
This equation blows my mind; I’d never really figured how tightly music and maths fit together. When you see a chord or melody, you can directly map it back to a mathematical pattern.
Our keyboard is actually an SVG picture of a keyboard, so we can traverse the elements of it and map each element to a sound generated by one of the functions that we came up with before.
Array.from(svg.querySelector('rect'))
.sort((a, b) => + a.x - b.x)
.forEach((key, i) =>
key.addEventListener('touchstart',
sound(0.75, ease(harmony(freq(i + 48))))
)
)
rect {stroke: #ddd;}
rect:hover {opacity: 0.8; stroke: #000}
Et voilà. We have a keyboard.
What I like about this is that it’s completely pure - there’s no lookup tables or hardcoded attributes; we’ve just defined a mapping from SVG elements to the sound they should probably make.
Doing better in the future
As I mentioned before, this could be implemented more performantly with Web Audio nodes, or even better - use something like Tone.js to be performant for you.
Web Audio has been around for a while, though we’re getting new challenges with immersive WebXR experiences, where spatial audio becomes really important. There’s also always support and API improvements (if you like AudioBufferNode, you’re going to love AudioWorklet)
Conclusion
And that’s about it. Web Audio isn’t some black box, you can easily link it with whatever framework, or UI that you’ve built (whether you should is an entirely different question).
If anyone ever asks you “could you turn this SVG into a musical instrument?” you don’t have to stare blankly at them any more.
(function(a,c){var b=a.createElement("script");if(!("noModule"in b)&&"on"+c in b){var d=!1;a.addEventListener(c,function(a){if(a.target===b)d=!0;else if(!a.target.hasAttribute("nomodule")||!d)return;a.preventDefault()},!0);b.type="module";b.src=".";a.head.appendChild(b);b.remove()}})(document,"beforeload"); |
2017 |
Ben Foxall |
benfoxall |
2017-12-17T00:00:00+00:00 |
https://24ways.org/2017/feeding-the-audio-graph/ |
code |
| 216 |
Styling Components - Typed CSS With Stylable |
There’s been a lot of debate recently about how best to style components for web apps so that styles don’t accidentally ‘leak’ out of the component they’re meant for, or clash with other styles on the page.
Elaborate CSS conventions have sprung up, such as OOCSS, SMACSS, BEM, ITCSS, and ECSS. These work well, but they are methodologies, and require everyone in the team to know them and follow them, which can be a difficult undertaking across large or distributed teams.
Others just give up on CSS and put all their styles in JavaScript. Now, I’m not bashing JS, especially so close to its 22nd birthday, but CSS-in-JS has problems of its own. Browsers have 20 years experience in optimising their CSS engines, so JavaScript won’t be as fast as using real CSS, and in any case, this requires waiting for JS to download, parse, execute then render the styles.
There’s another problem with CSS-in-JS, too. Since Responsive Web Design hit the streets, most designers no longer make comps in Photoshop or its equivalents; instead, they write CSS. Why hire an expensive design professional and require them to learn a new way of doing their job?
A recent thread on Twitter asked “What’s your biggest gripe with CSS-in-JS?”, and the replies were illuminating: “Always having to remember to camelCase properties then spending 10min pulling hair out when you do forget”, “the cryptic domain-specific languages that each of the frameworks do just ever so slightly differently”, “When I test look and feel in browser, then I copy paste from inspector, only to have to re-write it as a JSON object”, “Lack of linting, autocomplete, and css plug-ins for colors/ incrementing/ etc”.
If you’re a developer, and you’re still unconvinced, I challenge you to let designers change the font in your IDE to Zapf Chancery and choose a new colour scheme, simply because they like it better. Does that sound like fun? Will that boost your productivity? Thought not.
Some chums at Wix Engineering and I wanted to see if we could square this circle. Wix-hosted sites have always used CSS-in-JS (the concept isn’t new; it was in Netscape 4!) but that was causing performance problems. Could we somehow devise a method of extending CSS (like SASS and LESS do) that gives us styles that are guaranteed not to leak or clash, that is compatible with code editors’ autocompletion, and which could be pre-processed at build time to valid, cross-browser, static CSS?
After a few months and a few proofs of concept later (drumroll), yes – we could! We call it Stylable.
Introducing Stylable
Stylable is a CSS pre-processor, like SASS or LESS. It uses CSS syntax so all your development tools will work. At build time, the Stylable CSS extensions are transpiled to flat, valid, cross-browser vanilla CSS for maximum performance. There’s quite a bit to it, and this is a short article, so let’s look at the basic concepts.
Components all the way down
Stylable is designed for component-based systems. Imagine you have a Gallery component. Within that, there is a Navigation component (for example, containing a ‘next’, ‘previous’, ‘show all thumbnails’, and ‘show all albums’ controls), and within that there are NavButton components. Each component is discrete, used elsewhere in the system in different contexts, perhaps maintained by different team members or even different organisations — you can use Stylable to add a typed interface to non-Stylable component libraries, as well as using it to build an app from scratch.
Firstly, Stylable will automatically namespace styles so they only apply inside that component, by rewriting them at build time with a unique (but human-readable) prefix. So, for example,
<div className="jingle bells" /> might be re-written as <div class="header183--jingle header183--bells"></div>.
So far, so BEM-like (albeit without the headache of remembering a convention). But what else can it do?
Custom pseudo-elements
An important feature of Stylable is the ability to reach into a component and style it from the outside, without having to know about its internal structure. Let’s see the guts of a simple JSX button component in the file button.jsx:
render () {
return (
<button>
<span className="icon" />
<span className="label">Submit</span>
</button>
);
}
(Note:className is the JSX way of setting a class on an element; this example uses React, but Stylable itself is framework-agnostic.)
I style it using a Stylable stylesheet (the .st.css suffix tells the preprocessor to process this file):
/* button.st.css */
/* note that the root class is automatically placed on the root HTML
element by Stylable React integration */
.root {
background: #b0e0e6;
}
.icon {
display: block;
height: 2em;
background-image: url('./assets/btnIcon.svg');
}
.label {
font-size: 1.2em;
color: rgba(81, 12, 68, 1.0);
}
Note that Stylable allows all the CSS that you know and love to be included. As Drew Powers wrote in his review:
with Stylable, you get CSS, and every part of CSS. This seems like a “duh” observation, but this is significant if you’ve ever battled with a CSS-in-JS framework over a lost or “hacky” implementation of a basic CSS feature.
I can import my Button component into another component - this time, panel.jsx:
/* panel.jsx */
import * as React from 'react';
import {properties, stylable} from 'wix-react-tools';
import {Button} from '../button';
import style from './panel.st.css';
export const Panel = stylable(style)(() => (
<div>
<Button className="cancelBtn" />
</div>
));
In panel.st.css:
/* panel.st.css */
:import {
-st-from: './button.st.css';
-st-default: Button;
}
/* cancelBtn is of type Button */
.cancelBtn {
-st-extends: Button;
background: cornflowerblue;
}
/* targets the label of <Button className="cancelBtn" /> */
.cancelBtn::label {
color: honeydew;
font-weight: bold;
}
Here, we’re reaching into the Button component from the Panel component. Buttons that are not inside a Panel won’t be affected.
We do this by extending the CSS concept of pseudo-elements. As MDN says “A CSS pseudo-element is a keyword added to a selector that lets you style a specific part of the selected element(s)”. We don’t use a descendant selector because the label isn’t part of the Panel component, it’s part of the Button component.
This syntax allows us three important features:
Piercing the Shadow Boundary
Because, like a Matroshka doll of code, you can have components inside components inside components, you can chain pseudo-elements. In Stylable, Gallery::NavigationPanel::Button::Icon is a legitimate selector. We were worried by this (even though all Stylable CSS is transpiled to flat, valid CSS) because it’s not allowed in CSS, albeit with the note “A future version of this specification may allow multiple pseudo-elements per selector”. So I asked the CSS Working Group and was told “we intend to only allow specific combinations”, so we feel this extension to CSS is in the spirit of the language.
While we’re on the subject of those pesky Web Standards, note that the proposed ::part and ::theme pseudo-elements are meant to fulfil the same function. However, those are coming in two years (YouTube link) and, when they do, Stylable will support them.
Structure-agnostic
The second totez-groovy™ feature of Stylable’s pseudo-element syntax is that you don’t have to care about the internal structure of the component whose boundary you’re piercing. Any element with a class attribute is exposed as a pseudo-element to any component that imports it. It acts as an interface on any component, whether written in-house or by a third party.
Code completion
When we started writing Stylable, our objective was to do for CSS what TypeScript does for JavaScript. Wikipedia says
Challenges with dealing with complex JavaScript code led to demand for custom tooling to ease developing of components in the language. TypeScript developers sought a solution that would not break compatibility with the standard and its cross-platform support … [with] static typing that enables static language analysis, which facilitates tooling and IDE support.
Similarly, because Stylable knows about components, their stylable parts and states, and how they inter-relate, we can develop language services like code completion and validation. That means we can see our errors at build time or even while working in our IDE. Wave goodbye to silent run-time breakage misery, with the Stylable Intelligence VS Code extension !
An action replay of Visual Studio Code offering code completion etc, filmed in super StyloVision.
Pseudo-classes for state
Stylable makes it easy to apply styles to custom states (as well as the usual :active, :checked, :visited etc) by extending the CSS pseudo-class syntax.
We do this by declaring the possible custom states on the component:
/* Gallery.st.css */
.root {
-st-states: toggled, loading;
}
.root:toggled {
color: red;
}
.root:loading {
color: green;
}
.root:loading:toggled {
color: blue;
}
The -st-states “property” is actually a directive for the transpiler, so Stylable knows about possible pseudo-elements and can offer code completion etc. It looks like a vendor prefix by design, because it’s therefore valid CSS syntax and IDEs won’t flag it as an error, but is removed at build time. Remember, Stylable resolves to flat, valid, cross-browser CSS.
As with plain CSS, it can’t set a state, but can only react to states set externally. In the case of custom pseudo-classes, your JavaScript logic is responsible for maintaining state — by default, by setting a data-* attribute.
And there’s more!
Hopefully, I’ve shown you how Stylable extends CSS to allow you to style components and sub-components without worrying about that styles will leak, or knowing too much about internal structure. There isn’t time to tell you about mixins (CSS macros in JavaScript), variables or our theming capabilities, because I have wine to wrap and presents to mull.
We made Stylable because we ♥ CSS. But there’s a practical reason, too. As James Kyle, a core team member of Yarn, Babel and TC39 (the JavaScript Standards Technical Committee), said of Styable “pretty sure all the CSS-in-JS libraries just died for me”,
explaining
CSS could be perfectly static if given the right tools, that’s exactly what stylable does. It gives you the tools you need in CSS so that you don’t need to do a bunch of dynamic shit in JS.
Making it static is a huge performance win.
Wix is currently battle-testing Stylable in its back-office systems, before rolling it out to power Wix-hosted sites to make them more performant. There are 110 million Wix-hosted sites, so there will be a lot of Stylable on the web in a few months. And it’s open-sourced so you, dear Reader, can try it out and use it too. There’s a Stylable boilerplate based on create-react-app to get you started (more integrations are in the pipeline).
Happy Hols ‘n’ Hugz from the Stylable team: Bruce, Arnon, Tom, Ido.
Read more
Stylable documentation centre
Stylable on Twitter
A nice picture of a hedgehog |
2017 |
Bruce Lawson |
brucelawson |
2017-12-09T00:00:00+00:00 |
https://24ways.org/2017/styling-components-typed-css-with-stylable/ |
code |