How Facebook teaches photos to talk

Facebook's News Feed is a feast for the eyes, filled with photos, videos and status updates.That's not great f...

Posted: Dec 21, 2017 3:32 PM
Updated: Dec 21, 2017 3:32 PM

Facebook's News Feed is a feast for the eyes, filled with photos, videos and status updates.

That's not great for visually impaired individuals, so Facebook has turned to artificial intelligence to improve their experience. A blind person can now hear an audio message describing a friend's photo that shows people dancing or riding bikes.

To do so, Facebook's algorithm had to be taught what it was seeing.

Artificial intelligence is the secret sauce behind making a project like this possible. It can do everything from translate languages, understand human speech and identify diseases. But AI advances aren't without flaws.

Even as artificial intelligence excels, the human element -- which includes biases and oversights by those who train the system -- surfaces in alarming ways. For example, a Microsoft bot named Tay once sparked outrage when it tweeted attacks against Jews and feminists.

Dario Garcia, an artificial intelligence engineer at Facebook, is leading the project to identify what is happening in photos and read them out loud for the blind.

"If you get it wrong, the consequences are pretty bad," said Dario Garcia, an artificial intelligence engineer at Facebook. "[Our project is] not a self-driving car, where someone will die if you get it wrong. But you can give a very misleading experience to people that most likely don't have a clear way of knowing the algorithm is wrong."

Garcia's team gathered a sample of 130,000 public images that featured people. Staffers, called annotators, wrote a single line description of each photo. The images became examples that showed the AI what a photo of a person riding a bike or a horse looked like.

The team faced tricky questions. If only part of a person's body appeared in an image, Garcia and the annotators would need to discuss how that influenced the description.

"You become almost obsessed with what the current definition of a person is," Garcia said.

The conclusions of the group impacted how billions of photos are understood.

Over time, the algorithm learned what was happening in photos and developed its own captions. After caption writing was tested, some images were relabeled to correct mistakes. The AI also learned from those corrections and strengthened its predictions in what Garcia calls a virtuous cycle.

When the system launched in April 2016, it only identified objects and humans, but it has since been updated to identify 12 distinct actions in its captions.

To use the feature, a blind person needs to access Facebook with a screen-reader -- software that helps a visually impaired reader by using a speech synthesizer or braille display -- and focus on the image.

Related: Facebook exec: We need more women in power

There's still room to improve. The National Federation of the Blind recommends Facebook users who want the blind to have access to their photos include a detailed caption due to the limitations of the service.

Matt King, a blind engineer at Facebook who contributed to the project, compares today's AI systems to machines from the 1980s that read books to the unsighted. Those machines were the size of washing machines, couldn't read fancy typefaces, and the page of the book had to be clean.

"Artificial intelligence is creating a path to a world where everyone can communicate in ways they feel are most natural and can do so without leaving anyone feeling excluded," King said.

He says he's optimistic about Facebook's progress so far.

Facebook's advancements have also been helped along by Yann LeCun, the company's director of AI Research. LeCun, who joined Facebook in 2013 and is also a professor at New York University, is one the biggest names in the AI field. He's credited with developing the convolutional neural network, a popular AI technique that has been used for years in banks and ATMs to read the numbers on checks.

Despite its advancements, LeCun knows there are still limitations with AI. LeCun's wife, who is French, cannot use voice recognition apps because they struggle to understand her accent.

"There's not a lot of people speaking English with a French accent," LeCun explained to CNN Tech. "It's not that [engineers] don't like French accented people. It's just that there's not much data."

Mississippi Coronavirus Cases

Data is updated nightly.

Confirmed Cases: 115763

Reported Deaths: 3263
CountyConfirmedDeaths
Hinds7973177
DeSoto703979
Harrison522384
Jackson457884
Rankin394086
Madison383194
Lee357380
Forrest304678
Jones292484
Washington258399
Lafayette250443
Lauderdale2478135
Lamar225538
Oktibbeha202454
Bolivar201677
Neshoba1849111
Lowndes179962
Panola170040
Leflore167187
Sunflower162349
Warren154855
Monroe150673
Pontotoc147220
Marshall143129
Lincoln140157
Pike138456
Copiah137536
Scott125429
Coahoma124937
Grenada122638
Yazoo122234
Simpson121549
Union118825
Tate116839
Leake115041
Holmes114760
Itawamba113925
Pearl River113660
Adams108544
Prentiss106120
Wayne101722
Alcorn100112
George99218
Covington97527
Marion95042
Tippah90322
Newton86627
Chickasaw85526
Tallahatchie84526
Winston84121
Hancock84028
Tishomingo81241
Attala79426
Clarke75851
Clay69321
Jasper68717
Walthall63927
Calhoun62612
Noxubee59817
Smith59416
Montgomery54923
Yalobusha54514
Claiborne53716
Tunica53517
Lawrence51814
Perry49423
Carroll49312
Greene47818
Stone47514
Humphreys43816
Amite42513
Quitman4206
Jefferson Davis41011
Webster37613
Benton3416
Wilkinson33820
Kemper32615
Sharkey28514
Jefferson27610
Franklin2423
Choctaw2086
Issaquena1074
Unassigned00

Alabama Coronavirus Cases

Confirmed Cases: 158701

Reported Deaths: 2680
CountyConfirmedDeaths
Jefferson23292377
Mobile16916315
Tuscaloosa10345140
Montgomery10250197
Madison935096
Shelby739063
Baldwin665869
Lee654665
Calhoun459961
Marshall439550
Etowah428551
Houston417034
Morgan416435
DeKalb342629
Elmore320853
St. Clair295542
Limestone287230
Walker279492
Talladega266435
Cullman248024
Lauderdale229442
Jackson215915
Autauga205931
Franklin205531
Colbert202132
Russell19493
Blount193225
Chilton188432
Dallas186627
Coffee177111
Dale176351
Covington174729
Escambia172730
Clarke135217
Chambers135044
Pike134113
Tallapoosa132987
Marion108129
Barbour10339
Marengo101922
Butler101140
Winston92913
Geneva9067
Lawrence85832
Pickens85218
Bibb84014
Randolph82716
Hale76830
Washington74912
Clay74412
Cherokee73814
Henry7176
Lowndes71328
Bullock64917
Monroe64610
Crenshaw60830
Perry5926
Fayette57713
Cleburne5698
Wilcox56812
Conecuh56113
Macon53620
Lamar4965
Sumter47221
Choctaw39212
Greene34216
Coosa2043
Out of AL00
Unassigned00
Tupelo
Overcast
59° wxIcon
Hi: 68° Lo: 55°
Feels Like: 59°
Columbus
Overcast
60° wxIcon
Hi: 73° Lo: 57°
Feels Like: 60°
Oxford
Overcast
54° wxIcon
Hi: 62° Lo: 50°
Feels Like: 54°
Starkville
Overcast
55° wxIcon
Hi: 67° Lo: 54°
Feels Like: 55°
WTVA Radar
WTVA Temperatures
WTVA Severe Weather