Archive for the ‘api’ Category

translation of intro pages to china’s new real-time air quality reporting site

Friday, November 26th, 2010

Earlier today, I posted the fantastic news about the beginning of real-time air quality reporting in Chinese cities. Because the site is currently in Chinese only, I’ve quickly translated the first few splash pages here:

RT report 1“Enthusiastic congratulations for the successful launch of the air quality reporting system for key cities!”

RT report 2“Introduction to the China National Environmental Monitoring Center

The China National Environmental Monitoring Center is a government-affiliated supporting research institution (事业单位) directly administered by the Ministry of Environmental Protection. It includes the national environmental monitoring technology center, network center, data center, quality center, and training center. The primary responsibilities are national environmental monitoring and environmental monitoring technology development, in order to provide monitoring information, reports, and technical support to guide national environmental management and policy.”

RT report 3“Introduction to the Reporting System

The real-time reporting system for air quality in key cities automatically reports measured ambient data for sulfur dioxide, nitrogen dioxide, and inhalable particles. Hourly average data are presented for both individual monitors and entire cities. Data for individual monitors represent the average over the previous complete hour; for example, the data for 4pm represent the average value from 3pm to 4pm. Data for a city represent the arithmetic average of all stations in that city.

The data in the system are updated once per hour. Because the data transfer requires some time, there is currently a reporting lag of half an hour. For example, the data for 3pm will be reported at 3:30pm. When the monitoring equipment is undergoing calibration or encounters some communication or power error, there may be no data for that corresponding station.”

RT report 4“Introduction to the Reporting System

In accordance with the technical standard requirements of air quality monitoring, the air quality automatic monitoring data must be audited before it can be considered as part of an air quality assessment. However, in order to fulfill the public’s right to know in a timely manner, this system automatically reports the real-time data before they have been audited.

The source of data in this system is national-level ambient air quality automatic monitoring stations, not including local-level automatic monitoring stations.”

RT report 5“The relationship between real-time reported data and daily reported data

– The real-time reporting system for air quality in key cities reports the hourly average concentrations of three types of pollutants – SO2, NO2, and PM10 – for the entire city and at each monitoring station.

– The daily air quality report that has been in place since the year 2000 reports the 24-hour average air quality for key cities. In order to fulfill the need of afternoon air quality reporting by the media, the calculation for the daily report covers data from noon on the previous day to noon on the current day. After the average daily concentrations of each pollutant are calculated, they are converted to an Air Pollution Index (API), which can be easily understood by the public.

– Because the API is based on a 24-hour average of each pollutant, China, like many countries, currently does not provide an hourly API. Therefore, the real-time reporting system will only report the hourly concentration of each pollutant. The 24-hour averages will still be used for the daily API calculation.”

huge news – china begins hourly air quality reporting

Friday, November 26th, 2010

Wow, HUGE news today regarding air quality reporting in China: China’s National Environmental Monitoring Center has begun reporting hourly air quality data at over 2,000 individual monitoring stations in 113 cities in China. This represents a major development towards information transparency in China, and a key step towards providing residents here with the type of real-time air quality data they need to make healthy decisions regarding personal exposure. The release of this data will also pave the way for vast amounts of future research into the nature of air pollution in China.

In addition to providing data, the site is an extremely valuable resource for general information about China’s air quality, including air quality standards, definition of Air Pollution Index, impacts from air pollutants, main sources, etc. In many ways, the site is a like a very primitive version of the US EPA’s air quality public outreach site, www.airnow.gov. After what has seemed like a frustrating year for air quality in China, this is a wonderful development for which China deserves great credit.

The main address for the site is http://58.68.130.147/air/, with the actual data accessible at http://58.68.130.147/air/air/airtestpage.html. The site is currently exclusively in Chinese, and is developed in Silverlight, which is apparently a Microsoft version of Flash. (I had to download it from Microsoft’s website before I could view the site.) Because of the way the site is structured, it seems impossible to link to individual pages or copy/download data. This is an annoying and frustrating flaw in my opinion, but I’ll take what I can get for now.

I’m sure I will have many, many more posts about this new site and the treasure trove of data that’s available there. For now, though, let me just show a couple of examples of the type of information available:

This chart shows hourly concentrations of SO2, NO2, and PM10 from 7am this morning to noon at the Dongsi monitoring station in downtown Beijing:

dongsi real-time
This graph shows the hourly PM10 concentrations at that same individual station over the last 48 hours. The red line represents China’s daily air quality standard. The green line shows the previous day’s average.

pm10 real-time
Again, much more analysis to come, but for now my lunch break’s over and I need to get back to my day job. I’m happy to be bringing this positive report for a change, and look forward to providing more analysis and translation in the near future. In the comments section below, please feel free to ask specific questions about this new service and I’ll do my best to answer them.

“crazy bad” air quality in beijing

Wednesday, November 24th, 2010

crazy bad aqi chartI’ve taken the liberty of updating the US EPA’s AQI colors chart

A few days ago, the US embassy’s BeijingAir Twitter feed, which automatically reports Beijing’s hourly air quality, made headlines across the web by announcing that the air quality was “crazy bad.” Although the wording was quickly revised to the more politically-palatable “beyond index,” the impact was immediate. “Crazy bad” air was all the buzz of the blogosphere and at social events over the weekend; I have a feeling “crazy bad” would make it onto Beijing’s Word of the Year list, if there were such a thing. MyHealthBeijing‘s Dr. Richard St. Cyr even suggested “Crazy Bad” T-shirts.

In this post, I’ll look a little closer at the data behind the Crazy Bad incident to see what we can learn. The graph below shows hourly and daily data from the BeijingAir Twitter feed along with official air quality data from the Ministry of Environmental Protection (available here, Chinese only). For clarity, I show all data in terms of particulate matter concentration, not standardized index. (I’ve converted MEP’s reported API numbers to PM concentration; for background on the difference and methodology, see this post.) Note that MEP’s data is reported for a 24-hour period from noon to noon, which is why the daily data changes at noon each day. The break in the red corresponds to the BeijingAir Twitter feed not reporting any data for a couple of days after the Crazy Bad incident.

crazy bad

This graph reveals some really fascinating info:

1) The Crazy Bad spike on Thursday and Friday last week was both preceded and followed by gorgeous, wonderfully clean weekend days. On 11/15, the air quality was, by all accounts, “good.” By 11/18, though, the air pollution had steadily risen to what the US calls “hazardous“/”crazy bad,” and China calls “heavily polluted” (“重污染”). After the steep rise, the air quality improved just as dramatically; MEP’s reported PM10 numbers dropped 297 points – from 334 to 37 – from 11/21 to 11/22 alone. This demonstrates just how quickly the air quality can change in Beijing – both for better and for worse.

Why did it change so quickly? The start of the heating season on 11/15? Possibly, although that wouldn’t explain the sudden drop beginning 11/20. To be honest, the answer is probably less dramatic: weather. Most day-to-day pollution changes in Beijing are caused by changes in temperature and wind patterns. If there are a few days of static air, or light winds blowing from the southeast and trapping pollution against the mountains to the north and the west, the pollution builds up very quickly. I should find the time to post separately about this.

2) MEP’s air quality data tracked the embassy’s with reasonable consistency. This is actually encouraging; we should be grateful at least for some degree of accuracy and transparency with official data. There are some differences, but we shouldn’t expect them to track exactly. This is because the MEP data is an average of multiple sites across the city, while the embassy data is just a single point. Plus, they are measuring slightly different things.

In the past, we have seen situations where rapid and very short-term pollution spikes highlighted by the BeijingAir hourly readings were not reflected in daily MEP averages, but that doesn’t appear to have happened here.

(As I point this out, though, I should also note that I do not intend to make excuses for or to justify China’s current reporting mechanism. For the record, I would like to see at least three immediate changes to China’s air quality reporting: hourly release of data, more representative descriptions of health impacts, and some sort of real-time alert system for at-risk populations to avoid exposure.)

Lastly, a few data highlights for the numbers geeks out there (background on international standards here):

US EPA daily ambient air quality standard for PM2.5: 35 ug/m^3
Peak PM2.5 concentration reported by BeijingAir, 11/19: 557 ug/m^3

WHO recommended daily limit for PM10 exposure: 50 ug/m^3
China daily ambient air quality standard for PM10: 150 ug/m^3
Peak PM10 concentration reported by MEP, 11/19: 430 ug/m^3

counting grade 1 air quality days – a new metric for evaluating Beijing’s air quality?

Monday, August 30th, 2010

Earlier this month, the Economist featured a nice piece about the ineffectiveness of current efforts to improve China’s environmental quality, including some discussion of recent decreases in air quality. The authors cite the official Ministry of Environmental Protection announcement for overall air quality averages in cities across the country, but run their own numbers for Beijing specifically, arriving at this conclusion:

Residents of Beijing, who hoped the clear skies they enjoyed during the 2008 Olympic Games would persist, have also resumed their grumbling. Smog is back with a vengeance…

Official data show a diminishing share of grade 1 (“excellent”) air-quality days since the games.

The claim is supported with this chart:

201032asc031

This chart caught my attention because it’s the first time I’ve seen “percentage of Grade 1 air quality days” used as an indicator of air quality. Usually, data for Grade 1 and Grade 2 are presented together. The reason is because China’s ambient air quality standard calls for urban air quality to meet only the Grade 2 level. (“Blue Sky Days” are days which have Grade 1 (excellent) or Grade 2 (good) air quality by the Chinese standards.) Charting percentage of days meeting Grade 1 or Grade 2 yields a very different impression:

number grade 1 and 2 through july 2010

Using this metric of Grade 1 or 2, there is no apparent decrease in air quality from 2008 to 2009, the drop from 2009 to 2010 seems minimal, and the overall situation appears much less dire (the percentage of passing days floats above 70% as opposed to 10%).

Two questions come to mind here. First, is it misleading for the Economist to use this unconventional metric? Maybe. The article neither describes what Grade 1 means nor justifies why “percentage of Grade 1 days” is an appropriate / better indicator of air quality, although Grade 2 is discredited using an comparison to the BeijingAir Twitter feed:

The American embassy in Beijing, to the annoyance of local officials, issues frequent air-quality readings for its part of the city. These, based on the presence of fine particulates, mostly ranged from “moderate” to “unhealthy” in the 24 hours after midday on July 31st. But the government called that period grade 2, or “good”. Incredibly, to anyone familiar with China’s perennially grey urban landscapes, fully 91% of days in 113 big cities in the first half of this year were described as “blue sky”.

Second question: is it correct to talk about percentage of Grade 1 days? That’s tough to answer. The technical answer is no, it’s not, because any use of “number of days” or “percentage of days” meeting a single standard is statistically meaningless. (For a short and nerdy explanation of this, see this post). That having been said, though, I do believe that scrutinizing Beijing’s air quality by looking at Grade 1 – instead of Grades 1 and 2 – days provides a much more reasonable snapshot of the actual air quality in the city. For one thing, China’s air quality limits for Grade 1 are much closer to internationally-recognized standards. For another, one would assume that there is lower risk for data manipulation around the Grade 1/2 border than around the Grade 2/3 border.

I raised the idea of looking closer at Grade 1 earlier this year. Maybe it’s time for a more in depth analysis, but I’ll save that for another post.

official data shows air quality worsening in china

Wednesday, August 4th, 2010

Last week, multiple media outlets (including Xinhua, the New York Times, and the Wall Street Journal) reported on MEP’s admission of worsening air quality during the first half of 2010. Here, I will take a closer look at how this was reported, what exactly this means, and what is happening in Beijing specifically.

First of all, the original announcement from MEP is here (Chinese only). Both Xinhua* and the Wall Street Journal made major mistakes in their reporting of the announcement. Both reported that the average atmospheric particulate level rose to 0.091 mg/m^3 from 0.002 mg/m^3 last year. This is incorrect. Doesn’t anyone check original sources any more? The MEP report says “可吸入颗粒物浓度同比上升0.002毫克/立方米。” This means that the PM level is 0.002 mg/m^3 higher than last year’s, not that last year’s value was 0.002 mg/m^3. Someone pointed this out in the comments section of the Wall Street Journal article (which claims the increase is “humongous”), but it still hasn’t been corrected.

The Wall Street Journal also made the mistake of claiming that “the first half of 2010 had the worst air quality since 2005.” This is also not true. MEP’s announcement, which the WSJ story links to, says, “自2005年以来,环保重点城市空气质量优良天数比例首次出现下降,可吸入颗粒物浓度首次上升。” The correct interpretation here is that the first half of 2010 is the first period since 2005 that there has been a decrease in air quality from one year to the next. In other words, the trend of improving air quality has changed. At least Xinhua got this one right:

Economic recovery has partly caused the country’s air quality to fall in the first half of the year, the first such fall since 2005, figures from environmental authorities showed on Monday…”It was the first time for these cities to record a fall in the number of days with good air quality and a rise in the concentration of inhalable particles since 2005,” ministry spokesman Tao Detian said.

Unfortunately, the WSJ’s mistakes are already bouncing around the blogosphere.

Of course, despite the mistakes I’ve pointed out, the story is still quite significant. After years of official statistics showing improving air quality, MEP’s air quality data now shows a very slight increase in ambient particulate matter concentration. The data MEP gives are average pollution levels for 113 major cities, which is a little strange, since pollution levels vary widely across China. Although the averages are an interesting snapshot (and it is significant that MEP is reporting this bad news), these averages are not very meaningful; they say nothing about regional trends / changes or population exposure. This is an important area for more detailed research.

MEP’s announcement also gives averages in percentages of days meeting Class I and II air quality standards (so-called “Blue Sky Days“), but this metric is meaningless, as described previously on this blog.

What is more interesting is looking at data on individual cities. After all, most people spend most of their time in a single city. Plus, local and regional pollution control programs may vary from place to place. For Beijing specifically, I ran my own analysis of Beijing’s API data for the first half of the year using data I downloaded from MEP’s datacenter. After converting API to PM10 concentration using the methodology described here, I calculated the average PM10 concentration for the first half of 2010 in Beijing to be 124 ug/m^3. I also produced the following figure:

Beijing 2000-2010a

The figure shows that Beijing’s air quality, using average PM10 concentration as an indicator, has not shown improvement over the period 2008-2010. This trend is a continuation of the stagnant pollution levels I described earlier this year. Although MEP’s 113-city average (91 ug/m^3) is below China’s ambient air quality standard (100 ug/m^3), Beijing’s air quality remains well above China’s own standard, which is well, well above the WHO’s.

For those of you keeping track of Blue Sky Days, the Beijing EPB announced that there were 140 Blue Sky Days in the first half of the year, which, despite the increase in air pollution, somehow means Beijing is still on track to meet its Blue Sky Day goal. But this is a subject for another post.

Anyone who read this blog over the period 2008 to 2009 will know that I often repeated a two-part mantra: Beijing’s air quality is getting better, but we have a long way to go. I always felt that acknowledging the first part was critical to making progress working with China, and I was happy to see the New York Times article last October making this exact claim (headline: “Beijing’s Air is Cleaner, but Far from Clean;” my analysis here) It is frustrating and sad that this is no longer the case, even by China’s official data.

*To be fair to Xinhua, their mistake was only one word. Their report states, “The amount of inhalable particles, a major air pollution index, was also 0.091 milligrams per cubic meter in these cities, rising from 0.002 milligrams per cubic meter over the same period last year, the ministry reported.” The problem in this sentence is the one word “from,” which significantly changes the meaning (from incorrect to correct) if removed.

beijing’s 2010 blue sky day target announced as 266

Wednesday, January 13th, 2010

A China Daily report today mentions Beijing’s 2010 Blue Sky Day target as 266. This is the first time I’ve seen the target referenced; I’m not sure how/when they set or publicize them. Perhaps they are in Beijing’s Five Year Plan. Does anyone know?

This China Daily report also mentions Beijing’s 2009 Blue Sky Day total, 285, representing the first time I’ve seen it mentioned in English-language domestic media (although Xinhua covered the accomplishment in Chinese on 1/1.)

Recent targets and totals are shown below. Note that the 2010 target is below the number of Blue Sky Days achieved in both 2008 and 2009.

blue sky day targets and totals

why number of blue sky days is a terrible metric

Sunday, January 3rd, 2010

In my post summarizing Beijing’s 2009 air quality, I noted that although the number of Blue Sky Days increased from 2008 to 2009, there was a very minimal reduction in annual average particulate matter concentration (the reduction has now been confirmed by the Beijing EPB to be just 0.8%). I then called the “number of Blue Sky Days” metric scientifically meaningless.

Here is a simple example showing clearly how and why the use of number of Blue Sky Days can distort the reality of air quality:

Suppose you take two sets of two days, and you wish to evaluate which period had better air quality. Here are the data you have:

Day 1 and Day 2: API* is 100 on both days.
Day 3: API is 101.
Day 4: API is 1.

Over the period Day 1-2, we have an average API of 100, and a total of 2 Blue Sky Days.
Over the period Day 3-4, we have an average API of 51, and a total of 1 Blue Sky Day.

If you are judging air quality by “number of Blue Sky Days,” you would conclude that the air quality was better on Days 1-2. On the other hand, if you are judging air quality by average concentration of pollutants people are exposed to, you would judge that Days 3-4 were much better. Actually, because the normalization from pollutant concentration to API is non-linear, in this example, the average pollution level of Days 1-2 could be up to three times higher than the pollution on Days 3-4, and yet this period is judged as being better, because it has more Blue Sky Days.

This is what I mean by a scientifically meaningless metric.

*Reminder: API (air pollution index) is a 0-500 normalized measure of the pollution people are exposed to; a Blue Sky Days is a day with API of 100 or below.

summary of beijing’s 2009 air quality

Thursday, December 31st, 2009

Happy New Year (a few hours early)!!! For my last post of the year, here is some overall analysis of Beijing’s air quality over the past 12 months.

First, the numbers. As usual, I’m working from air quality data downloaded from the Ministry of Environmental Protection’s datacenter. (Direct link to Beijing’s data is here.) There is a bug in the system that prevents the query function from working properly, but you can still scroll through the data and download it manually through copy and paste.

The datacenter reports 363 data points for 2009; 2/18 and 11/14 are mysteriously missing. Assuming those two days were Blue Sky Days yields a Blue Sky Day count of 285 and a calculated average PM10 concentration of 120 ug/m3.


Update 1/14/11: The Beijing EPB’s 2009 annual report states a PM10 concentration of 121 ug/m3.

For anyone who is new to this blog or who needs a refresher, a Blue Sky Day is a day with an Air Pollution Index (API) of 100 or below, which means that the air quality meets China’s National Grade II air quality standard – theoretically “excellent” or “good” air quality. China tallies annual number of Blue Sky Days as a consumer-friendly and easily-understood measure of air quality progress, although the metric is prone to gaming and scientifically meaningless.

PM10 means particulate matter of size 10 microns or smaller, also called “inhalable particles.” In 2009, PM10 was the reported dominant pollutant on 97% of days with an API above 50. (No dominant pollutant is reported on days with an API 50 or below.) I back-calculated PM10 from API using formulas and assumptions described here.

Trends of Blue Sky Days and PM10 concentration since 2000 are shown here:

beijing 2000-2009 air quality trends

The Good News: The good news isn’t really that good, but in the interest of balance here are some positive perspectives. First, Beijing’s count of 285 Blue Sky Days in 2009 is well above that of 2008 (274), and also well above the goal of 259. This will keep the trending up in official reporting and for boosting public morale; I imagine this fact will be spun as success / progress in the Chinese media in the days to come. I suppose another bit of good news is that Beijing has achieved what appears to be an improvement over 2008 without the boost of two months of extreme traffic restrictions and factory closures implemented last year during the Olympic period.

The Bad News: My calculated average PM10 concentration in Beijing for 2009 is 120.2 ug/m3, which less than a 1% reduction from 2008’s average. In other words, from the perspective of average particulate matter concentration, there was no improvement in air quality in Beijing from 2008 to 2009. This shows the danger of using “number of Blue Sky Days” as a metric for air quality evaluation – just because the number of Blue Sky Days increases doesn’t mean that air quality has improved.

Interestingly, as a thought experiment, let’s suppose that China’s air quality goal was that all cities should meet the Grade I air quality standard, meaning that a Blue Sky Day would have an API of 50 or below as opposed to an API of 100 or below. If we count these days (shown in the graph above in light blue), we actually see that 2009 was worse than 2008; 2008 had 62 Grade I days, while 2009 had only 47.

Most importantly, regardless of how you spin the data, Beijing’s 2009 level of inhalable particulate matter was still 20% higher than China’s own air quality standard and six times higher than the WHO’s recommended annual PM10 standard. What this means, of course, is that our work continues to be cut out for us moving into 2010 and beyond.

Related post:  Summary of Beijing’s 2008 air quality

mep website redesign and datacenter problems

Monday, November 9th, 2009

The Ministry of Environmental Protection unveiled a new Chinese website about a week or so ago.  The English version appears unchanged for now. This news would probably not be blog-worthy except that the datacenter now seems a little buggy, causing two problems:

1) The Greenpeace Beijing API widget that I have on the top right of this blog isn’t fetching daily data anymore. (I’ve been in touch with the developers and I know they are working on it.)

2) The datacenter’s query function for past air quality data doesn’t quite work right; data queries return past data different from the date range entered. I hope and presume this will be fixed soon.

After the redesign, the presentation of API on the MEP homepage is now done in the datacenter analysis format, as opposed to the old list format (still used on the English site).

new mep page

air in beijing hazardous again

Friday, November 6th, 2009

11 6 09 twitter

The pollution in Beijing right now is bad. Really bad. So bad that you can feel the heaviness, the denseness of the air as you breathe it in. It is stifling.

So how bad it is? Well, the closest thing we’ve got to real-time data, the US Embassy’s BeijingAir Twitter feed, confirms our worst fears; it has been reporting hazardous air for over 24 hours now, with over half of the hourly data points maxed out at an AQI of 500. MEP’s air quality data released this afternoon, on the other hand, shows an API of just 186, “lightly polluted” (“轻度污染”).

I wrote a lot about this discrepancy back in June when Beijing experienced a similar pollution spike. (See these posts: 6/18: air in Beijing is hazardous; 6/19: more info on Beijing’s 6/18 air quality; 6/22: US Embassy outed as source of BeijingAir Twitter feed.) In June, I posited that the hazardous pollution was not reflected in the MEP data because the MEP data is a 24-hour average, while the pollution events then only lasted for a few hours each. In this case, though, BeijingAir has been continuously reporting hazardous air since around 4pm yesterday, so it’s hard to imagine how the MEP data is still so “low.” (Low is in quotation marks because an API of 186 still represents very polluted air.)

I’m afraid I don’t have time to post more right now; I’ll try to get some graphs up this weekend. In the meantime, avoid strenuous exercise and stay inside if you can. With the Embassy-reported AQI maxed out at 500, the air is, in theory, worse than hazardous:

11 6 09 aqi