How Google Maps, Spotify, Shazam and More Work | WSJ Tech Behind

Behind the

tech

nology that consumers know and experience lies a simple question: how does it

work

? From sophisticated software that gets us from point A to point B, to algorithms that suggest a specific song along the path we explore, to the engineering and science of

tech

nology that catches our attention. and talk to the key players who make all of this happen. Let's start here with how Google remapped the world. If you need to get somewhere, let's say a new building puts it on

maps

and Google gives you a root in seconds, but behind that seemingly simple answer is a complex system of data collected from users, satellites, cars and even camels, Today,

more

than a billion people use Google Maps every month and in 17 years it has evolved from a desktop application to a massive mobile platform that continually updates information about a location that we have

more

than 150 million. people from all over the world contributing information with 50 million updates a day from the public sector community, so think the map is alive, but the amount of data it collects has also attracted criticism from privacy experts.

This is how in Google's quest to remap the world it had to revolutionize the technology we used to see it this is the technology behind Google Maps you can search for a place like an address an intersection almost anything when it comes to digital

maps

Google was not the first on the scene or even the second, but when Maps launched in the mid-2000s, it had technology that many people had never seen before. I remember the day Google Maps was launched with satellite images in the navigator, everyone went and looked in their own home. Craig is a professor in the department.

A Geography graduate from Hoffer University who studies data-driven geotechnology, no one had ever seen his own house in satellite images, let alone color satellite images before this was a new sight on the world map. The success was due in part to the acquisitions he made early on, as we do to technologies. and Keyhole that made mapping more accessible to consumers. By purchasing Keyhole, they got the software needed to have an easy-to-use satellite image reviewer on a global scale, where two technologies gave them a much better user interface than what was available elsewhere and by combining those with just the sheer amount of capital at Google allowed them to spend and develop things not seen before Maps expanded its technology, quickly opening it up to developers integrating Street View and launching Android and iOS apps from the beginning, but how?

Exactly did all of this come together to get you from point A to point B? Christopher Phillips, the current head of Google's Geo division, gave us a look inside Google's campus. The fundamental piece was having a model of the real world, the digital truth of what is happening. In the real world, including where the streets are, let's take a look at a real route in San Francisco that Chris walked us on. Let's say I'm at the San Francisco ferry building, the farmers market, and I need to get to San Francisco International Airport. San Francisco to Get that real world model that Chris was talking about.

Google starts with aerial and satellite images to create an accurate map. This is overlaid with Street View data. To do this, they use a process called photogrammetry. They use this GPS NG data and can identify the correct coordinates. For the image, if we take a quick detour from our route to France, you can see what the layering process looks like using the Arc de Triomphe. Think of it as a big puzzle where we can put these images together and we understand the distance between the images. and where they sit with the actual location, GPS location. Google maintains a variety of cameras that have been used over time to capture images in the Street View garage on its campus in California.

It all started with a 500-pound camera that was placed with a wheelbarrow on a You may have seen at some point a Street View car with a 360° camera on top. Some versions are equipped with liar sensors that measure the distance between the camera and objects around it. They are also equipped with GPS for precise location. exactly where the camera is placed, but to reach more remote locations, they have adapted the cameras to suit the terrain. Google has put cameras in backpacks to reach places cars can't reach. They have also used transportation such as bicycles and snowmobiles in some rarer cases.

In some cases, Google has obtained images of camels, divers and even astronauts. Their newest camera can be detached from cars for transport. We move on to the newest one, which is 15 pounds and very light and can be powered off from a phone instead of a large computing unit. That's in the car These images aren't just for looking at, they actually help keep your maps up to date by detecting changes since the last time images were taken, not only do we capture what a place looks like, we can also detect that a business could have changed, we can detect new signs, now there's a stop sign here or a traffic light here, so understanding the new physical attributes of the real world is what we've been able to derive once they have all the information they need to figure out which It's the map.

It appears that Google aggregates information about traffic routes and businesses to do this, they get data from public bus and train schedules from local municipalities and from the businesses themselves. This is a 2013 recording of the software Google still uses to edit Maps, a combination of machine learning and map operators. edit public data, such as road net

work

s, to exactly match what you see in the images layered from 2013. Google says they've been able to automate most of this process, so for Chris's route from the San Francisco ferry building to the airport right now I can see that. It will take 21 minutes to predict how long it takes to get somewhere or how busy a place is.

Google relies on insights from reviews and anonymized location data and uses algorithms to analyze historical traffic patterns and analyzes current trends based on your Anonymous Location aggregated with millions of other users so we can overlay the trend with current activity and then create a prediction. The company also uses your approximate location and mapping tools to target ads and advertise businesses, which is one of the ways the platform makes money. We have the ability for businesses to promote themselves on the map, so whether you're looking for a place to go and if it's relevant to people who are in that area, we can show it as one of the results that Google has mapped so far. more than 250 countries and territories, but as it adds more data and becomes embedded in our lives and culture, users and regulators have pushed back and demanded security barriers.

Privacy is a big problem with any type of geographic technology and Google Maps is much more so. Because Google Maps narrows down to an individual address and individual people when we connect it to things like your Gmail account or your wallet or your Wi-Fi network, Google has been criticized and in some cases taken to court for things like amount. of data, collects Street View images showing private property, and takes data from unsecured Wi-Fi networks with Street View cars. Google says its data it uses to report traffic and busyness is anonymous. We use a technological treatment called differential privacy.

Differential privacy allows us to separate you and your activity from what we aggregate from millions of people as signals to get information about activity level and how long the walk or trip may take, but some data is used to tailor your map. personal, like noting that you've been to a location before, don't they say that that data is not sold or shared with third parties? Google also says it gives users privacy controls like turning location history on or off, searching in incognito mode, and automatically deleting location data. improve how we manage the information we capture to protect people's privacy and ensure we only have the critical amount of data to help drive the most useful products with your advances in technology, your advertising priorities and even your reviews.

Google Maps has helped change the way. We navigate the world The biggest improvement that Google Maps has made is to make geographic information much more accessible to a large number of people. The biggest drawback is that there is something that pays much more attention to the details of every movement you make when you carry your Google phone. says the future of maps aims to be more comprehensive and immersive, we are adopting technology that allows us to combine and merge satellite images, aerial images, street view images, user contribution to create a photorealistic view of how you would feel when visiting a park. the vibe what's the vibe on a different day immersive view is Google's next step in using its database of information and images to map the world if you're going to a big stadium concert like Taylor's Aras tour Swift or watch your basketball team score hear a buzzer ring a three-pointer in the playoffs or attend a K-pop show.

Light Up Sticks and bracelets will likely be part of the experience by making the audience part of the show with synchronized light displays at the touch of a button but how are they made? Small devices create these huge immersive images. It's funny because a lot of times people think that there's a GPS in every single device or that there's a lot of advanced AI technology, but we really like old school technology and let's just be creative with it, you know? this is the technology behind the LED bracelets pixmob, based in Montreal, is a leading event-led technology company pixmob will not comment on whether or not they are involved in the ays tour, but they have done previous tours of Taylor Swift and Coldplay The Weekend.

Bad Bunny we do most of the big tours. In North America we work a lot with the NBA NHL and to do so they use two different technologies. Let's start with the simplest RF bracelets that receive a radio frequency that communicates the precise time and colors for each one. Put the two white squares here, the two LEDs and here you have the little computer that controls those LEDs and then you have this other black thing that receives radio frequency and gives it to the computer and then the computer will turn on the LEDs. In the RF commands he received, this is what pixmob used for the fifth game of the ncks Heat series when fans went to their seats at Madison Square Garden and found one of these RF bracelets waiting for them, each one pre-programmed to be part of a different group, so a bracelet in this section, for example, is programmed to be in group one for one effect, group two for another, near the entertainment control room operator, AAL serves as the driver when one of the programmed effects is activated.

Different groups light up in sync to create different patterns within the stadium or you can manually play with them depending on the moment. The bold colors of NYX orange blue orange blue it's so much fun to watch the crowd come in and get excited that they're part of the show what's what's special the signal is emitted from this really small box we have a very simple transmitter it's like the size of a tissue box that we can connect to a light board we like that technology because you can literally travel with the entire control system in your suitcase RF technology is what most other portable LED companies use, such as Zyo bands They do everything from Eurovision to corporate events, but the most advanced bracelets, like the ones you see at Superbow or Lady Gaga, use infrared technology, so the same thing you wear.

Turning on your TV is kind of a technology that's like 50 years old, it's really old school technology, but we've managed to transform it in a way that makes it quite new and magical. Because we use infrared we can send data to specific location for you or the person next to you and it doesn't have to be the same information, that's how we create those specialized effects as we call them where we can almost turn a crowd into a canvas of video, these signals come from robotic transmitters placed all over the arena, you can see it here at a Coldplay concert last year, these transmitters were placed on sound towers and the stage, this is what it looked like in the pix mob visualizer, which is how you design what the shows will look like and and what the bracelets the company calls pixels will do, these two lines are basically the center of each of the transmitters, so even if we have two transmitters right now swiping, still looks like a wave with justadd, you know more about these movements.

EDS on this, in a setup like this, we could do a more complex animation when pixmob wants to show a shape like for example these hearts, it just puts on a mask like this to shine the infrared signal through it if it's low the infrared ray and it tells you to turn red, then you turn red and then when the ray moves away from you you go back to the color you were or black or something, so it's like the bracelets are pretty stupid in a sense , but we like it. That way you know, and the whole kind of intelligence of the technology is really in the transmitters, the audience, the concert lighting really started in Korea with K-pop bands back in the '90s and now they're also becoming quite creatives and those light sticks. they are not given out as part of the event they are MOS that fans sometimes spend over $100 on they work a little different you download an app you put in your seat information and connect the light via Bluetooth that's how they can be very detailed in its Arena siiz designs, fans call it ocean, but whether Bluetooth RF or infrared, the experience of these wearable pieces of technology only becomes more detailed so that every fan feels part of the show with 500 million monthly users.

Spotify is the world. The largest music streaming service Spotify is the home of audio, it is known for its personalized playlists created with its recommendation algorithm. Think of users as this raw material and then on top of the data layer we can build shared models but relying so much on artificial intelligence has also drawn criticism from some industry experts concerned about algorithmic bias here's how Spotify uses intelligence artificial to personalize users' experiences on the platform this is the technology behind Spotify in the early 2000s, many people found music recommendations through charts and early streaming platforms like Pandora and Last FM with the Last FM app on the App Store you can listen to great bands, so when Spotify came on the scene in 2008 it wasn't so much that they were the first people to start using analytics to recommend. music, but it was the way they combined various computational techniques to make their recommendations feel more real.

Thomas Hardson studies algorithms and artificial intelligence with a focus on how new technologies from music streaming companies impact fans of artists who listen to weekly and daily discoveries. mix up the way they talk about them in very human terms find out weekly you're great, you've done it again in 2014 Spotify acquired music analytics firm echonest, which combined machine learning and natural language processing to create a database of songs and artists, says Spotify This technology marked an important step in the evolution of its recommendation system. So how does that system work? It starts with a process called collaborative filtering.

Collaborative filtering analyzes the pattern of all this data and tries to understand when tracks are included in the playlist. Very often you can think of it as building a music and podcast map, that map looks like this, each dot represents a different track in the Spotify catalog and the location of each dot is determined by collaborative filtering, which means that these tracks go together according to the way users have playlisted and listened to them, so if these two songs are included together frequently, they will be close to each other in this map, while that if these songs are never included in the playlist together, they will be further apart on the map, but the recommendations are based solely on collaborative filtering is not perfect, for example, during the holidays.

Mariah is wearing All I Want For Christmas Is You. It might appear in the playlist more often with Silent Night, even though it sounds like a pop song and it sounds like a Christmas carol. If Spotify only generated recommendations based on proximity, then users who like Mariah Carey might be recommend Silent Night when they are not interested in Christmas carols to avoid this. Spotify adds another layer of analysis called content-based filtering. This algorithm collects metadata such as release date and tag and runs raw audio. The analysis uses metrics such as danceability and volume to describe the sonic characteristics of the track.

These are the results for Uptown Funk, which sounds like this and has a danceability score of 0.856 on a scale of 0 to 1. The algorithm also analyzes the temporal structure of each track. Here is a visual representation of Taylor Swift's antihero, these are the beads, bars and sections. Content-based filtering also takes into account the cultural context of the track, which means studying the lyrics and analyzing the adjectives used to describe the track in articles and blogs, these filtering techniques are not unique to Spotify, but experts Industry experts say what sets the platform apart is the amount of user data it has and the products it creates from it.

Spotify says its content-based filtering technology has evolved over the years and now includes more advanced technologies. proprietary features, but Hodson says the danger with algorithms is that they could reinforce existing biases. This could mean that a particular catalog of music has more male artists than female artists. One of the dangers of machine learning is that as listeners begin to interact with that catalog. Those biases are magnified and this creates what is called a kind of feedback loop. Spotify says its research teams assess and mitigate potential inequalities and algorithmic harms and strive for transparency about their impact.

Another criticism is that the algorithm is not optimized for new artists because there is no user data, this is known as the cold start problem. Sultan says this is where human editors play an important role in providing recommendations. They are arguably some of the best people in the world trying to understand new releases, culture and what's relevant, but Hodgson says the biggest concern is that certain metrics used in the platform's audio analysis may be culturally biased in others. parts of the world, have music systems and music cultures that are completely different, like this classic North Indian track for example, Spotify's algorithm tags its key signature. as E minor, which Hodson says is inappropriate for this musical tradition;

However, it remains true that the music emerging from South Asia is understood algorithmically in the West, rather than on the equal temperament scale. Spotify says audio analysis is a small part of the overall system, which takes many factors into account before making a recommendation, some industry experts also point out problems with the way the system understands classical music metadata For example, the metadata for a Tchaikovsky track can include not only the name of the work and the artist, but also the Opus number movement and Spotify's algorithm are not optimized for that. Apple Music, which has emerged in recent years as a competitor to Spotify, launched a new app in March that the company says is designed to solve this problem.

Spotify says no. comment on a competitor's marketing campaigns in February, the streaming service joined the recent Buzz about generative AI I am X and from this moment on I will be your personal AI DJ on Spotify, the DJ gives the algorithm a human voice and offers listeners additional context around a recommendation below I know you've been listening to summer songs lately Sultan says the company is also exploring reinforcement learning, a technique that would allow the recommendation system to automatically learn based on feedback , which will help with the diversity of your recommendation will help with long-term retention and we are trying to drive the latest in each of them, introducing new technologies, new capabilities and providing new experiences, check out, tap your card and there You will pay in less than 2 seconds.

It's faster than waiting for this and safer than this Why tap to pay has taken so long to gain traction in the US. This is the technology behind tap to pay at the heart of the tab. Nowadays there is a technology called NFC or Near Field Communication, which itself is a specific form of RFID or Radio Frequency Identification, you may be familiar with it. A way to unlock your hotel room, for example, NFC relies on those little wires that are actually antennas. The antenna looks like a race track and is kind of a coil of different wires and that's what is basically used to transmit the radio frequency on both. sides, this is Mike Mclennen, he's the general manager of hardware at Square, a company that makes card readers, and he opened one up to show us exactly how a tapto payment transaction works.

It reminds me a lot of the opening of Oysters card readers that also have those antennas. NC an antenna, so this is the magic of what makes NFC transactions work and you can see once we remove it, how thin the NFC antenna is, the card and the reader are what he calls passive and active connections, The reader is active because it is turned on and can initiate radio communication on its own and the card is passive because it has no power and needs to be near an active source to initiate that communication. His phone intervention works the same way because his phone emulates the passive C.

What is happening? now the coil here is active and it's looking for a card basically anywhere within this type of region. Up here it is doing it using a specific radio frequency for NFC devices that only works in a very short range, a maximum of about 4 cm and when the reader finds the card it requests the information necessary for the payment, that information is stored here In the chip of your card, one thing it stores is what is known as static data, that is, information that is the same every time you use your card, such as the account number and expiration date is what is sent from your card's magnetic stripe when you swipe it, but its chip not only sends that static data when you tap it, it also sends something called a cryptogram, a unique string of numbers that issuers use to verify that your card is valid. what the card does is collect information from the reader about the transaction about the reader specifically and combine it with the information on the card, using its cryptographic key to create a unique number for that transaction, that's what makes it really difficult to replicate a card fraudulently.

This is because the card needs to have a cryptographic key that allows them to generate that secret code, that information is sent through the payment card, which classifies the data it needs from the card and places it all in an encrypted package that then gets. sent to the main board, which then sends it to our servers, which then sends it to networks like Visa Mastercard MX, which then sends it to the issuer and then comes back with the approval message once it reaches this board. all information encrypted and protected and anything beyond this will have a high degree of security involved, all of that happens in just a few seconds faster than inserting your chip, so when you insert a chip card into a reader there is some back and forth. and return. strength in that communication, while in the contact list there is less back and forth, it was simply built as a more efficient, more optimized type of information and only the basic needs that are passed from one place to another, card networks and the Security experts say this is also safer. that other ways you can use your card, as opposed to swiping your static data, you are better protected during that communication and because you have no contact with the reader, you are not susceptible to malware that can affect chip insertion transactions, Rarely, this payment technology has been adopted by Tapto in recent years in part due to the push for contactless technology during the pandemic.

Square says such payments tripled between the start of 2020 and the end of 2022, while other countries like the UK have integrated contactless technology into public transport for years. It took us over a decade to get here and find a whole new way to pay. MasterCard presents the payment pass. We started issuing tapto payment cards in the 2000s. Peter Rudiger is a Wall Street Journal reporter covering payments and financial technology.Different banks thought it was great. new technology and they thought their consumers would actually flock to it didn't work just because there were few companies that were created to accept tap to pay us.

Retailers have been slow to replace payment systems in part because they are expensive to change. In 2022, about 55% of merchants said they accepted tap-to-pay and others said they were planning to implement it. Consumer behaviors are also difficult to change and the shift to contactless technology did not feel urgent; simply, it has often been described as a solution in the search. of a problem, yes, it might be convenient and yes, it might be safe, but that's usually not enough to get people to change their behavior. The technology still has room to grow and the push to integrate it into more parts of everyday life is working in its favor.

New York, for example, Subway r ERS can now tap to pay on turn styles, so it has this big push to get tap to pay on more transit systems and get people used to using it, you know it every day and if you are going to use touch. pay to get on and off the subway at least twice a day maybe you also use it for your coffee your lunch uh your drinks at least that's the idea the credit card companies are hoping will happen and other companies have also recently launched the functionality to use the NFC antenna on your phone as an active connection, essentially putting a reader in your pocket, which could make it easier for small retailers to accept card payments.

This shows you where you want it to tap and so the NFC antenna on your phone is right behind this so when I take my NFC enabled credit card and hold it there the transaction is completed here there is a wave of sound and here is its inverse, combine them and you have silence, that is the basic idea behind noise canceling headphones, but it takes a lot of technology to make that seemingly simple science possible, we have 15 millionths of a second to measure noise , run it through a little DSP processor, calculate that reverse uh signal and output it through the controller, that's how they work in a way to visualize acoustic waves which I like to use as a sneaky ruler from John has spent the last decade developing new RMS algorithms to customize the noise cancellation performance in Bose headphones, we draw them as sine waves as if something is going back and forth or up and down, but what is really happening is that these vibrations are actually They are propagating through the air, the medium has remained fixed, but you can see that pulse of energy running along it because sound is just waves, you can cancel a sound wave by introducing an equal and opposite wave, so what? what What we have is an incoming sound wave and the way destructive interference works is we say, hey, if I could generate an equal and opposite signal, could I generate the signal that is the inverse of that and if I add those two things?

I will be left with a flat line, I will be left with zero pressure and that will be interpreted by my brain as having been noise cancelled, Bose was the first company to develop noise reduction technology for headphones. Dr. Bose was returning on a flight to foreigner and became very frustrated by the fact that airplanes were very noisy, we can control all kinds of other things, why can't we control the noise? The first application that was analyzed was noise cancellation in airplanes. Pilots used their first prototypes in 1986, while the technology has become more sophisticated. The basic science remains the same.

Active noise cancellation requires a few main components. An external microphone to track incoming noise. An internal microphone that tracks sound waves. which enters inside your ear and also checks if more ambient noise is generated than expected through a digital signal processor to run some algorithms. Based on data from the microphones and speaker that cancels background noise while playing the music or sound you want. To listen, think about how fast this has to happen, sound obviously travels at the correct speed of sound, which is 343 m/s if you think about that, in terms of the thickness of an earphone, it comes from the outside of the earphone. to the inside of the earpiece in about 15 microseconds, so it's millionths of a second.

We're doing the math as fast as we can to basically cancel it out before it goes in, because everyone's ears are different shapes. Active noise cancellation requires algorithms that can be adjusted accordingly. On the user, Bose, for example, does this by playing a welcome chime and monitoring feedback from the internal microphone to customize noise cancellation filters accordingly. This requires a lot of testing to ensure that the algorithms work on a wide range of users in all types of noisy environments. But removing background noise is just the first step, the next is figuring out what noise to remove based on context.

I think it's very important to recognize that noise is not bad. Noise is essential for my brain. It is essential for my body. My brain uses it to create richer experiences to fill in what's missing, which is all the time. Poppy Crumb was previously the chief scientist at Dolby Laboratories, where she worked for more than a decade. Her LED teams focused on emerging technologies and platforms that impact sensory systems when thinking about noise cancellation. technology I like to separate it into three different categories, you know, there's passive noise cancellation or passive noise reduction, which is effectively an earplug, the second layer would be active noise reduction, it's dynamic to the system, but it's It is what we would call stationary. sounds sounds that you know are consistent, are continuous, or are predictable examples of what could be noise from a generator or an airplane.

You don't know all the airplane sounds, but General Rumble, those low frequency sounds you're trying to get rid of. They're pretty predictable for these predictable noises. Current noise cancellation algorithms can quickly identify unwanted sound waves and cancel them beyond that, things get more difficult and then the next layer, which I think is where you know the future is still evolving, is when you start to be able to really cancel out things that are what you would call non-stationary noise, things that might be a dog barking and you know those things that happen where you can't predict them and you might be able to have an expectation that they're going to happen. happen, but they are things that are not continuous or constant, but sometimes after your technology has gone through so many steps to reduce background noise, you want the airpods Pro to go back to a transparency mode where they recreate the normal sound from your surroundings, other noise-canceling headphones also offer modes to help you hear your surroundings, such as Bose's Aware Mode and Sony's Ambient Sound Mode, which preserve enough background noise to keep you alert to your surroundings when create a technology that is supposed to adapt to a human experience that you need to understand how the human system perceives the change or perceives the element that you are building and then that becomes how your algorithm behaves so that you have a continuous experience of the world, so you need labs where you can eliminate or control the noise cancellation of all other elements. could soon solve more complex problems, such as using machine learning to seamlessly and intelligently remove unwanted sounds or voices in VARIOUS contexts.

The adaptive audio coming to Apple's AirPods Pro will soon combine noise cancellation with transparency mode so you can only hear the important sounds around you. so you can stay present in your surroundings while distracting noises are automatically reduced. It's what everyone wants. They want to make certain noises disappear. That's why they still want to hear some things. They don't want to listen to others. And I can?. just give me a pair of headphones that, you know, just cancel out the things I don't want to hear, that's not something we're offering yet, but that's the kind of thing that's being worked on in July, the sphere changed. for the first time dominating the Las Vegas Skyline oh, I don't think you like it, but inside the 366t high Dome we are not outside.

It's an even more incredible immersive experience. The Dome has the largest high-definition screen in the world, 160,000 square feet. almost 20 times the size of the largest IMAX screen, but the screen is just one piece of innovative technology made for this place to create content for the screen sphere needed a new camera to film it this is the technology behind the sphere when Director Darren Nerowski set out to make Postcards from the Earth, an exclusive film featuring immersive scenes from around the world. His team began filming with this camera or rather cameras. When I first appeared, there were 11 reds welded together in Frankenstein style, but combining the views. of those 11 cameras made it difficult to shoot and edit, which led us to design the Big Sky camera system as well as the lens system to take all those different perspectives on the 11 different lenses, but making it one perspective so that sits like it's your Eyes work, this is Deanan Dilva, chief architect of the Big Sky camera, walked us through Sphere's camera lab in California.

It's amazing to have a tool that is so responsive and so versatile. It's really an amazing tool to have filming content for Sphere. You have two big challenges, you need a camera that can shoot at a very wide angle and at a super high resolution to get that super wide angle. Big Sky us uses a giant version of a fisheye lens nearly a foot in diameter, creating a linear circular image. which distorts the view to capture a wider angle in a circle and then, when light passes through the lens, it is printed on a sensor that divides it into small colored squares called pixels.

Larger pixels allow the camera to capture more light and more detail, so to capture a greater number of large pixels for a screen the size of the sphere, you need a larger sensor than a screen the size of, say, your phone, some of those pixels are wasted on a normal rectangular sensor as a circle doesn't take up the full frame, but that circle captured from that lens is projected onto the sensor in a circular format and that circle fits perfectly on a square sensor in instead of a rectangular one. The lens and sensor come together to create the massive, super-clear field of view needed.

To test this with the spherical display, we took the camera right outside the camera lab. This is how a camera with a flat lens and smaller sensor saw this area compared to how Big Sky saw it. The field of view reaches almost behind the lens. Here you can see Denan standing just to the side behind Big Sky, instead of shooting horizontally, we tilt up so we can capture your full field, which is natural in Sphere where you're sitting. The tilt helps the camera align with Horizon this line here on the screen because when that circular image is transformed into 3D, that is the place that should look most normal to the audience, the crew on set can verify that the view look natural with this virtual reality device that creates a live simulation of the footage on the sphere's screen, but to get a real preview of what the footage will look like on the sphere, the team takes the footage down the street to here , the large dome, a quarter-sized version of the sphere they use for testing. a prototype of the display panels on the sphere, it's effectively 32 LEDs on a foot and when you scale it up to the spot, that's how we get to 16k, which gives the illusion of being there, each of these lights constitutes a pixel of the screen around 170 million in total that surround and over the almost 18,000 seats in the cinema that make up the resolution of 60 16k the camera on a The image to compare your iPhone screen is only about 2500 by 1000 pixels.

You need a lot of pixels and a lot of detail everywhere because one audience member might be looking in this direction, another might be looking in this direction, so all parts of the screen are important. part of the screen and that's not the only thing that makes the sphere so immersive, these holes in the panel allow sound to pass through them since the speakers are behind the screen and different sounds can be emitted from different parts of the screen at the same time. $2.3 billion of venues also have physical effects like temperature changes, breezes and smells, as well as 10,000 seats that vibrate to respond to movie scenes, but the screenis the centerpiece and to create a movie with a huge high definition camera for a huge high definition theater.

The team had to think carefully about what they were photographing. You can't really hide behind anything when you're shooting with Big Sky because everything is sharp and if there are any errors, they're just there - this photo of a climber in the Alps, for example. it took a little bit of trial and error, you're looking at Dwayne the climber and you're looking at this kind of valley below and not seeing the depth of the mountains behind you you kind of missed the setting it didn't feel that risky Andrew Schulen is the director of photography for sphere and worked with aronowski on a postcard.

It felt riskier and more special for him to be on this kind of spire in front of this snowy landscape. Those images finally made it to the screen on the sphere. I remember we were in the editing. when I saw that and I got really excited, but we couldn't tell what it was because it was only in 4k and then we had to go through the Uprising process to be able to see it in the Big Dome sphere some elements are still being refined. of the camera is to improve its ability to photograph very small objects and make it easier to use, but the next big step is to take it to the International Space Station.

The screen is so big that the ISS can actually fit inside the screen so we can transmit. for people an experience of doing a spacewalk that is actually realistic, one of the things that we are very interested in as a sphere is using place and technology for science and those are the types of things that we are curious about. Over 23,000 songs are played on Shazam every minute in the world and we will potentially explore in the future, but how do you identify the music around you after recording just a few seconds of audio from your phone?

Shazam has to create a unique audio fingerprint of what you're listening to and then search its database for a matching fingerprint, which means simplifying sound into a bunch of data that can be processed incredibly quickly. This is the technology behind Shazam. The idea behind Shazam is to simply take the audio captured on your phone and turn it into something. computer can be easily compared to other songs on Acoustics. This is called an audio fingerprint, simplifying a song to a unique signature that you can identify even through a lot of background noise. The first step in creating Shazam was to digitize a large amount of music since there was no database.

It might help, we literally hired about 30 18 year olds to work 8 hour days and three 8 hour shifts, so 24 hours a day we put CDs into computers running custom software we built from scratch. We had 100,000 CDs, which is about 1.7 million songs. Because CDs don't contain metadata, they had to type in the name of each song, the title of each album, and each artist while building this massive database, a Once they digitized each song, they had to turn them into something that looked like this, a spectrogram. Spectrograms map frequency content over time, they also show the signal power level related to how loud something sounds, which is represented by color.

In this spectrogram of a bird's song, we have time to move to the right, as is always done in spectrograms, and then the frequency in the vertical. axis, so this is going down, you know it's not the correct Z frequency, but that's the idea. Julia Smith is professor emeritus at the Stanford Center for Computational Research in Music and Acoustics. She was also an early Shazam consultant and helped on one of the company's first projects. Patents with musical spectrograms often seem more complicated than this because you have to deal with overtones or harmonics that become softer. As you go up from the fundamental frequency of Beethoven's Fifth Symphony, the opening measure is boom BM BM and then the next whole measure is B, so this is the classic vague score and then here on the spectrogram we have the same thing, not as clearly written but clearer to hear and that's it, but if Shazam were to match spectrograms like this, it would take an incredibly long time to return a result.

The first key insight was to focus on the spikes in the spectrogram because those spikes are the main thing the brain processes. Shazam simplifies a spectrogram like this to a peak scatter plot. Many of these peaks make it through the background noise, but you can also end up with a bunch of extra meaningless peaks, so let's say we have a spectrogram of a piece of music and then we look at the spectrogram and how it changes as the noise enters. , so let's sayuh, people start walking to the bar, people start drinking a lot, people start making a lot of noise, then what you see is the original spectrogram plus a lot of new activity and the brain is very good at understanding superpositions of sounds, if people can do it, then the computer can do it well.

The core of Shazam's technology is matching scatterplots that essentially plot the most powerful signals at different frequencies over time. It also develops some important processing steps to speed things up so you can get a result quickly instead of searching through your database. For a match of all these points in the proper sequence, Shazam looks for several points at once and doesn't worry about the overall sequence until the end, for example, let's say you were in a coffee shop playing classical music, you open Shazam and record your audio . first you make a spectrogram like this, then you isolate the Peaks and make a scatterplot like this, although in reality it would probably have more points.

Shazam connects nearby Spikes to form a group of pairs, then Shazam searches for matching pairs in an organized database of millions. of songs if there are enough matches in the same song and they are aligned correctly over time Shazam can name the melody, in this case a specific rendition of snare and G major minuette after about 2 and a half years of development. Shazam was released in 2002. Before the era of smartphones, everyone carried these cell phones everywhere and you could only really do two things with them: you could make phone calls and send text messages and I guess you could also play the game Snake.

That's how Shazam worked back then. Initially, Shazam was a service that people in the UK could access by dialing 2580 after recording 15 seconds of audio, the call would be disconnected and the user would receive a text message with the name of the song while the fundamental algorithm ran. . has stayed the same Engineers have made some modifications to improve its recognition rate, it was good enough to feel like it worked most of the time and really delighted people that it would work in bars and clubs etc, we also had to then add additional settings to deal with the fact that music in the real world also adjusts in terms of speed, tempo and pitch, so for example DJs can adjust the speed of music when playing vinyl records .

Shazam was not an immediate solution. had to sell some of its patents to fund development, although it later bought them back and became unprofitable, says that until 2016 Barton was CEO for four years before resigning, although he continued to serve on the company's board of directors when I left Shazam. was going through a long period where Shazam barely survived it was a story of uh almost failing almost breaking incredible resilience for many years it didn't really reach a hockey stick of incredible user adoption until the App Store launched in 2008 , finally Appleb Shazam in 2018, is now one of the most popular free music apps on the App Store, once you pay for unlimited music streaming, what are you really going to listen to?

Because you have access to everything you want and Shazam solves it. A form of discovery: You don't just discover music when your friends tell you about a great band, you also discover it in your daily life. The technology behind Shazam has grown significantly since Apple's acquisition in 2018, when it bought Shazam. the company has access to Apple's large music library today. Shazam can identify most of Apple's music catalog of more than 100 million songs. We created an app many years before apps existed. Part of the art of a startup is the timing aspect and, fortunately, we found. a way to survive through those years