NYC Dining: The cost of a "B"
If you eat out in New York City, the image above should evoke some sort of visceral reaction. In July of 2010, the NYC Department of Health began rating each of the 24,000 restaurants throughout the five boroughs of the city. Each restaurant is given a grade of "A", "B", or "C" based on violations ranging from improper food temperature to sewage problems to the presence of vermin. You can browse the complete list here.
Fast forward 2 years, and the new system seems to be a win for consumers - Mayor Bloomberg credits the program to a 14% reduction in Salmonella, the lowest rate in 20 years. And according to this press release, NYC restaurant revenue is also up 9.3% since grading began. But still many restauranteurs disagree, expressing anger over these health inspections. Restaurants complain about the complexity in understanding the grading system, fighting with the city over infraction points, and spending additional money to maintain their facilities to meet the city's guidelines.
But clearly the biggest cost associated with the city's program is the fear of a "B" rating, or, even worse, an unmentionable "C" rating.
Just how costly is a "B"?
To quantify these costs, I correlated NYC restaurant inspection rating changes with their restaurant ratings on the popular review site, Yelp. Starting with the most popular 1000 restaurants in Manhattan on Yelp, I crawled each of their review pages, extracted ratings for each restaurant. NYC health inspection ratings are available via NYC's OpenData initiative, and each of these top Yelp restaurants were then correlated with their corresponding health code ratings. All code is available on GitHub under my Nyc Restaurant Inspection Project, along with a csv that contains joined Yelp restaurant reviews with their corresponding inspection ratings.
According to the Mayor's argument, Salmonella cases have gone down since restaurant inspection ratings have, on average, increased since the start of the program. The Mayor's report claims that the number of "A" ratings has increased form 65% to 72% of all restaurants since the start of the program. And within the set of top Manhattan restaurants analyzed here, trends are similar. The plot below shows average rating inspection value since July 2010 (5.0 represents "A", 4.0 "B", etc):
Looking at average Yelp reviews since 2005, we can see that the time period since August 2010 is relatively stable, hovering between 3.8 and 3.9.
To get a better sense of how ratings are impacted by inspection grades, let's look at restaurant grade changes ("A" to "B", "A" to "C") and see how their yelp ratings in the 60 days before and after changed:
| Change | Rating Before | Rating After | Delta |
|---|---|---|---|
| A -> C | 3.94 | 3.68 | -6.7% |
| B -> C | 3.86 | 3.69 | -4.6% |
| A -> B | 3.77 | 3.76 | -0.3% |
Restaurants downgraded to a "C" rating received significantly lower Yelp ratings in the month after the downgrade, but restaurants receiving a "B" rating were relatively unaffected in their review quality.
So restaurants with "C" ratings tend to have a lower review quality on Yelp, but do lower ratings deter people from dining at a restaurant in the first place? Looking at overal review counts for 60 day periods before and after rating changes:
| Change | Count Before | Count After | Delta |
|---|---|---|---|
| A -> C | 167 | 157 | -6.0% |
| B -> C | 214 | 230 | +7.5% |
| A -> B | 724 | 699 | -3.5% |
The increase in review counts in "B" to "C" downgrades is most likely due to the data being somewhat thin. Across all three downgrades, Yelp review counts as well as rating counts showed average decreases of almost 2%.
Takeaways
A recent study by Michael Luca found that increased Yelp review rating quality can lead to increased revenue. Among other things, the study also found that independently owned restaurants were much more affected by these reviews than ones with chain affiliations. Many of Manhattan's top restaurants analyzed here are independent, and the decrease in Yelp ratings here undoubtedly also corresponds to lost revenue.
An interesting question to consider is one of causation: one goal of the inspection program is to improve sanitary conditions at restaurants in NYC. When a restaurant transitions from an "A" rating to a "C" rating, the only change we can say for certain is the letter grade posted outside the front door. In the days and weeks following a downgrade, one would expect restaurants to actually clean up their sanitary conditions. So, during the time period analyzed here, sanitary conditions before the downgrade are probably worse than after.
Of course, the other goal of NYC's inspection process is to increase consumer awareness. And consumers seem to notice: when restaurants are downgraded, the costs are measurable.