00:06

all right my name is Jacob I am JCB LW

00:10

everywhere on the web and that's not bad

00:12

i don't really care about that social

00:14

website and also i'm an organizer of

00:17

jail a because i love the Jazz community

00:19

today I'm going to be talking about

00:22

visualizing audio on the web very

00:25

descriptive title nothing fancy no click

00:28

bait or anything you know you're getting

00:30

what you see there I probably can work

00:32

on that especially when we put up the

00:33

video you want to make it a little bit

00:34

more enticing and do something with it

00:37

but right now it's just visualizing the

00:39

audio on the web so before we go forward

00:43

and talk about visualizing audio I kind

00:44

of want steak a step back I mean I'm

00:46

getting older I turned 29 this month and

00:49

you know got here the older the older I

00:53

get the more I like to look back and be

00:54

nostalgic about the things that of my

00:56

path and one thing is you know

00:59

visualizing audio or visualizing music

01:02

was like it's something that I really

01:03

really liked now who remembers this yeah

01:07

so this is windows media player this was

01:10

actually my first visualizations that I

01:13

ever you know fall or probably even

01:15

dealt with as a young child and is it

01:19

like when I see this I'm like oh it

01:21

brings me back to a place where I'm

01:23

sitting in front of my computer and I'm

01:24

shoving CDs and my you know CD Drive and

01:27

like opening up a program is before

01:29

streaming or anything without now this

01:32

is a YouTube video and let's see

01:34

comments like this it really makes me

01:35

sad because it's like this is no longer

01:38

going to be available to me to be able

01:40

to use this even though granted i

01:41

probably wouldn't use it but there's no

01:43

there's no chance than we actually using

01:45

it and it's being a YouTube video of

01:47

course there is like funny comments okay

01:49

out of trip balls on your 10 this was

01:52

kind of funny because like I was

01:53

probably like 10 when i was using

01:57

windows media player and actually i

01:59

think this is a pretty good talk title

02:03

if you don't mind me i'm gonna change

02:06

this real quick

02:09

[Music]

02:13

there we go I think that's gonna that's

02:16

gonna get two clicks right there all

02:19

right going forward all right enough

02:23

about going in the past let's talk about

02:24

the present so I don't know if you guys

02:27

have checked out like modern interfaces

02:29

for playing audio pretty sure you spot a

02:31

Phi or iTunes or something and one thing

02:34

about that is that music visualizations

02:36

are usually gone from those interfaces

02:38

it's more or less like these are behind

02:41

some tab or like they're non-existent or

02:45

the some static visualization which is

02:47

like on soundcloud you actually have

02:49

like these very static visualizations

02:52

and it's pretty much just a progress bar

02:53

that you're going through and back to

02:56

myself I actually just started a new job

02:58

and it's not this really cool place

02:59

called headspace the meditation app we

03:01

actually hosted their last month but one

03:04

thing that's interesting as this is a

03:05

progression in my like career is also

03:07

that my prior places I was working with

03:10

maps and latitude longitude putting pins

03:14

on a map and if I ever wanted to do

03:17

anything with audio is really on my own

03:19

time and there was no really like

03:20

practical use for me to do it our main

03:23

product that headspace is actually audio

03:25

files so me getting to look into these

03:27

new Web Audio API is actually beneficial

03:29

for my job as well which I kind of love

03:32

that like I'm actually doing something

03:33

that I wanted you as well as it's kind

03:36

of like benefiting me a little bit and

03:38

before like started I was like checking

03:40

out the product that headspace and they

03:42

have this really cool player it's very

03:44

light but you can see that the player is

03:46

kind of moving a little bit it has like

03:47

a little bit of animation supposed to

03:49

just be smooth and like soothing it's

03:52

not really like it's not doing any audio

03:55

visualization even though you kind of

03:56

say it's like a vibe type of

03:58

visualization or something because it is

04:00

the meditation app but I thought it was

04:02

really intriguing like I usually you see

04:04

audio players and it's just a play

04:06

button and then a stop button maybe a

04:08

progress bar but you really don't have

04:10

much innovation and besides doing that

04:13

you have you know album art as big as

04:15

you can get it or something and the

04:17

first thing that went through my mind

04:18

which I think probably goes through a

04:20

lot of our minds

04:22

time to time is I just want to implement

04:24

it in JavaScript like why we actually

04:26

have a web player at head space but it

04:29

looks nothing like that cool one that I

04:31

was talking about that was like our

04:32

mobile player and that was actually a

04:34

youtube video that I was showing that so

04:37

before I started I actually took off a

04:40

month and I rode my skateboard around

04:42

New York and DC but during that time

04:44

actually stopped in a bunch of coffee

04:46

shops and I would just hack on stuff and

04:48

it was mostly trying to implement this

04:50

thing in JavaScript now the next slide

04:54

that i'm going to show you is my first

04:55

try or not first try it's the first step

04:58

in a progression of trying to implement

05:01

this player now when you look at this

05:05

and sorry for the blurry quality but

05:06

this looks nothing like that player at

05:10

all so let me kind of like go through my

05:14

thought process of this so the player is

05:16

kind of like this random blob and

05:18

there's like pieces of it moving in a

05:20

certain direction so what I did was like

05:22

I want to pick make a bunch of points

05:24

and put them around a circle and then

05:25

move them in a random direction and

05:27

that's what this is except I'm really

05:29

bad at math so as you can see it's not a

05:32

circle it actually came out something

05:34

that's more like a sideways donut I

05:37

thought it was really cool and like when

05:39

I built something really cool i liked

05:40

like at least take a screenshot or put

05:43

it on twitter or something so that's

05:46

what i did with this mess just a

05:48

progression in it as you can see i got

05:50

better at math there's another

05:52

visualization why didn't get better i

05:54

just like figured out what my problem

05:55

was and fix it and yeah so this is just

06:00

like a i'm changing the colors around as

06:02

you can see it's a little bit more

06:03

coherence getting a little bit closer

06:04

still doesn't look anything like it and

06:06

what you'll see in some of my demos as

06:08

well as just some of the other slides is

06:10

I like to like I make things very

06:12

declarative so I can just like take

06:14

these like numbers and just twist them

06:16

and like turn them to 11 and then

06:18

probably turn it back to nine because my

06:19

computer is lagging too much and that's

06:21

what you'll see in a lot of these demos

06:23

is me just kind of like messing around

06:25

with numbers and I also have another

06:29

step in here but the thing is is that I

06:31

did a lot of this without source control

06:34

and I really wish that I

06:36

I used sorts of control the entire time

06:37

because then I can show you a better

06:38

progression of this because the last

06:41

slide to this next slide is way

06:44

different so more or less like there's a

06:47

big gap in this entire history of this

06:50

Wiz players came and more or less what I

06:54

did when I connected those points made

06:56

quadratic curves between them and made

06:58

it one solid shape and then I actually

07:00

started playing audio with it and this

07:02

next step and fun so this is what it

07:07

looks like there's a little bit of music

07:09

here as you can see revision we actually

07:11

are starting to visualize audio in this

07:13

and francha this is nothing is not very

07:17

calm if you look at it this would not be

07:20

soothing to be inside of a meditation

07:22

else but more or less this was just me

07:25

taking that knob and dialing it to 11

07:27

and so this is like a very simple

07:29

visualization so I'm taking those random

07:32

points and usually looks like a small

07:34

threshold and that's kind of like at a

07:35

rest 8 and when the one there's like a

07:38

snare head or a beat drop or something

07:41

the frequency goes up and you're able to

07:43

have a larger variation and that's where

07:45

you feel like the little spikes because

07:47

it goes in and it goes out now have

07:49

another one go and turn it back on this

07:53

one is not so much a demo of the player

07:55

i'm kind of like showing off some of the

07:56

functionality because I I kind of build

07:59

this in stats so but as you can see the

08:03

animation got a little bit more subtle

08:05

this one's a little bit better you could

08:07

actually see this probably in kind of

08:08

like an interface you could say the

08:11

animation change instead of just doing

08:13

like the random points and having a

08:14

bigger threshold to them this one

08:16

actually the border gets a little bit

08:17

larger but it's actually what I found

08:19

interesting at this point is there's a

08:22

lot of different things you can do with

08:24

audio visualization that's not so

08:26

straightforward I mean this is like a

08:28

subtle animation if you thought about

08:30

windows media player that I had in the

08:31

very first slide or second slide it

08:33

actually that was like a full screen

08:35

immersion sometimes a little too much I

08:38

could say where this is a little bit

08:40

more subtle granted it probably would

08:41

get annoying after a while and this

08:44

really sparked my interest and kind of

08:46

like audio visualization so much so that

08:48

I actually built

08:50

npm module it's called sound dashboard

08:52

it's on NPM and also you can find it on

08:55

my github will actually post this to the

08:56

flak channel afterwards so you guys can

08:59

check it out if you want what this

09:00

allows me to do is load up like an audio

09:02

buffer file and then be able to like

09:04

analyze the data of it when it's like

09:06

playing and grab that data out really

09:08

conveniently instead of hooking up these

09:11

AP is now you you might be thinking like

09:15

oh that's cool and all but like how are

09:17

you actually doing this like what is

09:19

actually going on that maybe somebody

09:21

else wants to do audio visualization and

09:22

like what is the steps that I had to

09:25

take to get to that point so I have a

09:28

couple little pieces of code and I'll

09:30

try to walk through this and then i'll

09:33

answer questions at the end and i'm

09:34

pretty sure it'll be mostly about these

09:36

slides but essentially i'm using the

09:39

audio contacts API it's I want to say

09:42

it's you but it's been around for a

09:44

couple years now the the support for it

09:48

is pretty good there is some methods

09:50

that are a little bit spotty on in terms

09:52

of browser support and more or less like

09:55

my demos only work in chrome and firefox

09:57

right now but more or less I'm creating

10:01

the audio context and that's what I'm

10:02

doing that new audio context I'm

10:04

creating a buffer source what a buffer

10:06

source is essentially I can attach like

10:08

an audio buffer to it and that will

10:12

allow me to like play and stop the audio

10:15

buffer and then I'm also creating an

10:17

analyzer now the analyzer does exactly

10:19

what it sounds like it's analyzing the

10:21

data that is going through it in the way

10:24

that I can pump data through the

10:25

analyzer is down below as you can see i

10:29

am adding the audio buffer now that's

10:32

like a un aid array which is like a

10:34

array of arrays that's holding like

10:36

binary data inside of it and i'm

10:39

connecting it to the analyzer which i

10:41

think it's spelled one of these wrong

10:43

but analyzer connect and then i'm

10:46

connecting the analyzer straight to the

10:47

destination you can think of this as

10:48

with speakers of your computer and then

10:50

i'm just starting it you can add a bunch

10:52

of params for that to start it at a

10:54

certain point

10:55

now to get the data out of the analyzed

10:59

because I just showed you how to connect

11:00

the analyzer this is how you would get

11:02

it out so you get a buffered lang that's

11:04

pretty much a the audio that it's

11:07

playing at a certain time because we're

11:10

going to call this function over and

11:11

over and over again so whenever you're

11:13

playing like an audio file and you want

11:14

to get a certain data out of a certain

11:17

time when it's playing you can get the

11:19

length of the actual binary how long it

11:22

is and then I'm going to create a UN 28

11:23

array with that length which is going to

11:26

just create a blank one and then call it

11:28

calling audio analyzer get byte time

11:31

domain data I really love that method

11:33

name but then call it coin it with data

11:36

array is actually going to hydrate that

11:38

data array with a bunch of binary data

11:40

from that current buffer and then as you

11:42

can see here I'm just using

11:43

requestanimationframe to call this thing

11:45

over and over and over again and get out

11:46

the the data for each sequence and this

11:52

is what actually creating the

11:53

visualization looks like now like I said

11:55

I'm not good at math so I don't know

11:57

exactly what's going on here granted

11:58

there's only a little bit of code but

12:00

more or less what's happening is we're

12:02

we're calculating a waveform and we're

12:05

just getting x and y coordinates for it

12:07

that's what these are and then you can

12:10

take those x and y coordinates and you

12:11

can put absolute this gives or more

12:14

likely you probably want to render those

12:15

to a canvas and then you would have like

12:17

an actual waveform if you just rendered

12:20

those that's like circles on a canvas

12:21

which is really cool because it's like

12:23

that's not that much code to be able to

12:25

render out a waveform and if you want to

12:29

look at more there's actually a good mdn

12:31

doc that i have in like some of my

12:33

github repos that is the place where i

12:36

actually learned how to do a lot of this

12:37

stuff because they have a doc on how to

12:39

analyze audio data on the web and that's

12:43

where a lot of this code came from now

12:45

as a few demos the first one I have here

12:48

I was just playing around with stuff

12:50

cranking the knobs on things and this is

12:54

like a very sporadic like it's exactly

12:56

like a small snippet of like an

12:58

animation that you'll see later on but

13:02

more or less like I call this one that

13:03

event horizon I just thought it was cool

13:06

yeah it's a very chaotic

13:08

and then now I have another demo and

13:13

this is actually turn this is actually

13:19

the blob player that I was showing you i

13:22

just have it down here kind of as a

13:23

player it's not I mean the whole

13:26

interface that doesn't revolve around

13:28

this audio visualization exactly there's

13:30

a lot of times where I think audio

13:32

visualization is very good at when it's

13:34

subtle just these subtle things I mean

13:36

you talking to your Google assistant or

13:38

theory or something they're doing a

13:40

little bit of visualization of the input

13:41

that's coming into your phone and it's

13:43

nice that you're getting that feedback

13:44

that there actually is like audio going

13:46

through it so I kind of like you know

13:48

just doing these subtle things but then

13:50

again I don't always do very subtle

13:53

things and I have some demos that are a

13:55

little bit more chaotic now this is a 3d

14:02

demo it's very it moves very quickly and

14:06

it's very jagged and this is one trouble

14:08

that I've been having with visualizing

14:10

audio it's just that you're getting so

14:11

much data coming in I like you saw I'm

14:14

actually i'm just doing

14:15

requestanimationframe i could delay that

14:17

but the more i delay it where's the

14:19

visualizations become and when you have

14:22

these things happening in every like 100

14:25

milliseconds or probably shorter than

14:26

that it's hard to do tween between the

14:29

different points and be able to do

14:30

smooth animations it's possible but it's

14:33

not that easy but this is showing you

14:35

you can actually like integrate this

14:36

kind of stuff into anything even like 3d

14:39

objects then I have one more same song

14:46

but this is actually just a waveform but

14:49

it's around a surrounded circle so

14:53

essentially took that same math that i

14:54

was able to like to put points around

14:56

the circle and then I just apply to

14:58

waveform to it and then that way when it

15:00

hits like a snare drum or beat drop or

15:02

something you'll see that the kind of

15:05

like fan out I'm actually caching like

15:07

five versions of you know the waveform

15:10

and then cycling through those and then

15:12

this outer one is the oldest and the

15:15

inner

15:15

it's the newest and actually i'll show

15:17

you really quick how easy it is to like

15:21

change this around and this is this is

15:23

why i have so many just random looking

15:26

visualization I want to go to my code

15:28

real quick and as you can see I just

15:30

have like a bunch of variables up at the

15:31

top but what's cool is that I could just

15:33

say like 100 and then that should update

15:39

and that's not I can refresh as you see

15:44

it changed around the whole

15:45

visualization I mean granted i think the

15:47

other one was cooler looking this is not

15:49

bad as you can see this is actually the

15:51

one that the event horizon originally

15:53

came from because you see the lines from

15:55

the circles the outline and then i could

15:58

even do things like this which this is

16:00

actually going to just make my computer

16:02

go super slow so now i'm catching up to

16:09

10 of these things and doing the

16:10

visualizations with that but it once you

16:13

get something like this it actually

16:15

starts to slow down the animation it

16:16

looks very laggy but it's you know it

16:19

looks really cool especially when you're

16:20

not hearing the audio because when

16:22

you're not hearing the audio it just

16:23

looks like a cool animation but that is

16:28

it and thank you we have the second mic

16:46

up here yeah he's got the mic for you

16:57

um I was using at weaning library uh and

17:02

the it was just in a blob demo a lot of

17:07

those things I'm just taking the raw

17:08

audio data and then I'm trying to make a

17:09

some type of wave for more like

17:12

frequency something and then just

17:15

putting that straight into the points

17:17

with the blob player and I I'm not

17:19

really using any animations and like the

17:21

main thing here but you'll see when I

17:24

actually press like pause it's kind of

17:27

like you know tweens out and from those

17:31

different points that's the only place

17:33

when i'm actually using animation

17:34

library at all but definitely if i

17:36

wanted to smooth things out for instance

17:38

in like a 3d one it's very jarring

17:39

sometimes and I'm definitely thinking

17:42

about integrating some types of

17:44

animation library or some tweeting

17:45

library to be able to submit those

17:46

things out

17:54

are you kidding a drum so really a lot

18:05

of the visualizations are more or less

18:07

based off of what would be the part of

18:10

the waveform I'm taking like a highest

18:12

peak of the wave form and saying that's

18:13

kind of like the highest bit inside of

18:16

so essentially like based off of that

18:18

I'm making like the border fatter or the

18:20

threshold for the the random points

18:23

larger because you can't really like I

18:25

mean you can get into a lot of like and

18:26

like breaking apart the the song and

18:28

whatnot you can actually do that with

18:30

the stuff smarter people than I am can

18:32

do that with that stuff I really just

18:34

like to make things look cool right off

18:36

the bat but more or less like what

18:38

you're seeing here has nothing to do

18:40

with like separating any of the sounds

18:41

it's just more or less heavy taking part

18:44

of the waveform at its peak and being

18:46

like okay this is a very loud part of

18:48

the song or something like that no

19:02

that's a good question because I was

19:04

going to say something but sometimes I

19:06

forget to say something so you might see

19:09

in this slide I actually connect instead

19:12

of connecting I could connect the source

19:14

directly to the destination and I that's

19:19

actually I think inside of the the mdn

19:22

doc that I was referencing earlier

19:23

that's how they did it one thing I

19:26

noticed when I did that though is that

19:27

the visualization sometimes got out of

19:30

sync so the way that I was able to kind

19:33

of fix it was that I would straight from

19:36

the analyzer go to Z the speaker so that

19:39

way it always had to go through the

19:40

analyzer before it hit the speakers so a

19:42

lot of that latency wasn't there just

19:44

because the analyzer had already gotten

19:46

the data out of the buffer that was

19:49

playing at that current time by the time

19:51

it hit the speakers

19:56

alright

20:00

[Applause]

Visualizing sound on the web

Transcripts (auto-generate)

Subscribe to my newsletter