Tieming Ji / Stat+Plot

A little visualization practice of pharmaceutical companies world wide

2011-03-15T10:21:00.000-07:00

I do not know if you have had similar questions with me when you get your medicine from the pharmacies. I have questions such as "Where are my medicine developed?" "What other alternative choices do I have from a different medicine company?", or "Is USA better than other countries in the world for medicine development?" "How the people in other countries deal with their illness?" etc.

I downloaded data from wikipedia. The following visualization practice does not solve all the questions in depth, we clearly need more data and more sophisticated statistical analysis for that, but these visualizations for this small amount of data could give us a first look at the questions.

I extracted the top 50 large companies' data online, sort them by countries. The following figure is a histogram showing how many large pharmaceutical companies in different countries. USA has more than 20 companies in the top 50 company list. Following US, Japan has 10 companies among the top 50 list. In the second figure, I sum up the revenue of companies in different countries (in million dollars). Clearly, US based companies (in sum) have more total revenue than other countries. Of course, there will be a currency problem, for example, 1 US dollars = 5.5 RMB. Well, it might be more fair if we only compare US with European countries, it looks like US have more companies, more people, more investment on health care research and development.

What People Complain about New York

2011-01-16T12:10:00.000-08:00

Beautiful visualization tells the story of New York.

Netflix Rental Patterns

2011-01-16T10:05:00.000-08:00

By Matthew Bloch, Amanda Cox, Jo Craven McGinty and Kevin Quealy/The New York Times

FireFox Visualization Competition

2010-12-17T09:03:00.000-08:00

The official competition website is here. In this competition, I did some analysis of user type and their web activity pattern.

I categorize users into 3 major groups based on their main reason to use the web. The users who do coding work have the highest computer skill. The users who do not do coding work but do non-coding work or school work have the intermediate computer skill. And the users who use web only for entertainment (networking, socialization, etc.) have the lowest computer skill. I call these three different groups of users coding type, non-coding-school type, and entertainment-only type users respectively.

In the first exploration, we want to examine how much time different types of users spend on web. The data are collected from the entry of their main reason to use the web in the survey table. After removing empty entries and free text entries, we totally get 3,788 complete user input with unique IDs and their gender, age, time spend on web, self-evaluated computer skill, main reason on web, and most frequently visited websites. The following analysis is based on these data.

According to the definition of the three types of users in the summary, there are 1,360 coding type users, 1,605 non-coding-school type users, and 823 entertainment-only users.

Based on the entry of time they spend on the web each day in the survey table, we plot the approximate distribution of time on web for each user type. The data are displayed in Figure 1. The figure shows that the entertainment-only users more often spend 2 to 6 hours each day online than the other two type users. While more percentage of coding users spend more than 6 hours online daily. And the majority of non-coding-school type users spend around 2 to 8 hours.

Based on the first exploration, we want to know how do different types of users spend their time online, what do they do when they are online, and whether there is any difference among them.

We use the entry of their most frequently visited websites in the survey table.
We count the number of users in each user type for the listed 13 activities.
We plot the data in Figure 2. We find that there is a substantially larger percentage (>15%) of users in the coding group than the entertainment-only group claiming that they frequently visit search pages, news pages, and their mail boxes. There is also a moderately larger percentage (>10%) of users in the coding type than the entertainment-only type claiming that they frequently visit forums, banking and online word processing pages. And a slightly larger percentage (6%) of users in the coding type than the entertainment-only type claim that they frequently visit shopping websites. The percentages of non-coding-school type users on the above activities are in the middle between the other two types. There is not much difference among these three type users for other activities, such as network, download, video, adult page, games, and gambling.

This result suggests that coding type users do not only use online resource more frequently for their work, such as search, forums, mails, etc., they also visit websites for their life more frequently, such as news pages, banking, shopping, etc.

Another point is that the most popular and useful web applications are search applications, news pages, mails, and network applications.

Optimization with constraints in R and MatLab

2010-12-12T10:46:00.000-08:00

Often we need to compute the maximum (or minimum) of a function with parameter(s) in a constraint interval. For example, we want to get the maximum output with limited budget, or maximum product with limited time, etc.

There are several R functions for computing optimal solutions for optimizing a goal function with constraints. In my research and projects, I often use optim(), optimize() or nlminb().

optim() supports high dimensional optimization with box constraints. While, optimize() is a special case of optim() that optimize() only supports one-dimensional optimization. optimize() adapts golden search algorithm and successive interpolation which does not require any additional information, such as gradient function etc. And in many cases, one-dimension optimization is sufficient for our problems. nlminb() supports high dimension optimization with box constraints using a Newton-Type algorithm. It does not require gradient (1st order derivative) or hessian matrix (2nd order derivative) if it is a one-dimension problem (see PORT routines by at&t bell labs), but they are required if it is a high dimension problem. The derivatives may behave unstable at the edge of box constraints. In this case, nlminb() does not give a very sensible solution. Thus it is always a good check to look at extreme parameter values. optimize() and nlminb() give similar results up to 5 digits after decimal point when the parameter values are not the edge point (or close to edge).

In MatLab, I use fmincon() function for one-dimension optimization with box constraints.

Sweave

2010-12-10T10:58:00.000-08:00

Copying and pasting of a huge amount of data from R output to Latex report is labourious. This motivates me to learn sweave. Imagine that I have a table of simulation results which contains 5 rows and 10 columns, and I run 3 times of simulations with different parameter values, without sweave, I need to manually copy and paste 150 times! Here is the link for the tutorial I found. Hope it is helpful for the people who have the same need with me.

http://www.cepe.ethz.ch/education/NPecoHS2010/Sartori-Sweave.pdf

Statistics on Tipping at Germany

2009-07-23T09:37:00.000-07:00

I was in a ggplot2 workshop. The following figures are examples in that workshop. In Figure 1, we were trying to explore the relationship of total bills and the amount of tips that females (F) and males (M) would give when they drink more than one cup of alcohol (Yes) or not (No). The straight line is a first-order regression line, and the curve line is a lowess fit. One obvious observation from Figure 1 is that when people drink alcohol, there is more variation in their tip. Some times, they tip up to around 20% of the total bill, and other times as low as 3%. Secondly, females do not spend as much money as males do in restaurant. The total bill for women on average are less than males. There are more observations from men which may indicate that either males pay for females more or males go out to eat more frequently. In addition, when people drink alcohol, the total bills for them are usually more than the ones who do not drink. Maybe people who do not drink alcohol have more sense of saving (or not wasting) money. At last, comparing the tipping rate at Germany with that in America, American people are more generous in tipping. We usually give 15% to 20% of the total bill for tips, don't we?

In Figure 2, we try to explore tipping patterns for groups. The shaded area is 95% confidence interval. It looks like a group of 5 and 6 exhibit dramatically more variation in tipping than other groups.

Are You Ready for Statistics

2008-10-16T15:54:00.000-07:00

A famous saying goes as "The world does not lack beauty but lacks the eyes to discover the beauty". Perhaps, both the eyes to find beauty and the mind to sense the random world need to be trained.

Dr. Bradley Efron at Stanford University came to Iowa State three days ago for a biostatistics conference as well as Lawrence Baker lectures at our university. He gave a talk titled as "Learning the experiences from others" in which he provided several examples of empirical Bayesian ideas utilized in practical problems. His talk motivated me to find more of his works including this article.

This article points out that previously people believe science should be exact. Such as how exact time goes by; how sure the light speed is, etc. However, years ago, science began to unveil another side of its face - the science of randomness, and it is played nearly everywhere. Statistics repeats its principles all the time, such as the rainfall example at Palo Alto given in the article in the first paragraph. Thus, statistics study should not be restricted to classrooms or examples in books. It should be widely noticed and studied. People who study statistics should also open their mind.

False Discovery Rate (FDR) Reliability Evaluation

2008-05-26T14:56:00.000-07:00

In analysis involving multiple tests, we can control false discovery rate (FDR) by several approaches, such as Benjamini-Hochberg adjusted p value (BH adj.p value), Storey-Tibshirani q value (q value), as well as BH adj.p value and q value based on moderated t-test after empirical bayes adjustment.

The question of interest is which method gives us a better control of FDR. That is, when we want to control FDR at alpha level, which gives us a more reliable control such that the FDR will not deviate far from alpha.

To study this problem, we simulated 14,118 genes with expression under two treatments where the first n (n<14,118) were designed to be truly differentially expressed. Then, we applied three methods for identifying differentially expressed genes - (1) q value based on usual t-test (2) BH adj.p value based on usual t-test (3) q value based on moderated t-test. The purpose is to examine the precision and accuracy of controlling FDR by these different approaches. One result of several obtained results looks like this: (simulate 100 times of 14,118 gene expression under 2 treatment with first 3000 genes truly differently expressed. Set alpha level at 0.05)
Summary statistics are listed as follows:

(1) q value based on usual t statistics

mean.q :0.04565834

var.q :6.964762e-05

(2) BH adj.p value based on usual t statistics

mean.bh.p:0.03763231

var.bh.p:6.955762e-05

(3) q value based on moderated t statistics

mean.ebayes.q:0.04702102

var.ebayes.q:3.80829e-05

The above statistics tell us that the q value based on the empirical Bayes moderated t test gives the best precision and accuracy. Similar tests can be run many times to compare and evaluate these FDR control approaches.

Occurance Competing problem

2008-04-20T14:27:00.000-07:00

Problem: Suppose we flip a coin. Given two patternss of the same length composed of "Head" and "Tail", compute the probability of seeing the first string before the second one.

Solution: First-step Analysis.

Suppose the two strings are: HHTH, HHHT.

You have graph with states and transition probabilities in the Figure.

Define PH: The probability of seeing pattern HHTH before HHHT if we are currently at state H.
Similary, we can define PT, PHH, PHHH, PHHT, PHHHT, PHHTH.
Define P(H) and P(T) is the probability of emiting an H or a T. For a fair coin, we have P(H) = P(T) = 0.5.

Thus, we have the system of equations as follows:
PH = P(H)*PHH + P(T)*PT ---------(1)
(explanation: When you stand at state H, you have probability P(H) to go to the state HH, and probability P(T) to go to the state T)
PT = P(H)*PH + P(T)*PT ----------(2)
PHH = P(H)*PHHH + P(T)*PHHT -----------(3)
PHHH = P(H)*PHHH+P(T)*PHHHT -----------(4)
PHHT = P(H)*PHHTH+P(T)*PT ------------(5)
PHHHT = 0 ---------(6)
PHHTH = 1 ---------(7)

After plugging the equations (6) and (7) in the first 5 equations, you get five unknowns and 5 equations. Now you should be able to solve the system equations to get the value of these five unknowns. They are the probabilities of seeing the first pattern before the second one given the current state.

Thus, the probability of seeing pattern HHTH before the pattern HHHT is:
Porb{HHTH occur before HHHT} = P(H)*PH + P(T)*PT.

A Tutorial for Probability

2008-03-28T18:47:00.000-07:00

http://www.math.uah.edu/stat/foundations/index.xhtml

Nugget

2008-03-18T16:22:00.000-07:00

Interdisciplinary study is not a way to avoid hard work and deep understanding of one specific field. In contrast, people who want to be experts in interdisciplinary study need to do a lot more work than others.

I am currently a graduate student at Bioinformatics and Computational Biology program. Every year our program recruits around 10 students from hundreds of applicants all over the world. After the first two-year training, no matter what your college background is, one is supposed to stand at the graduate level of computer science, biology, and statistics, and be able to talk about any of them with sufficient understanding.

This is hard, challenging, and perhaps having increasing need in future. Bioinformatics here only serves as an example.

When I worked with biologists on statistical analysis, though I have had several graduate level biology courses, I still need to sit down and ask some questions about biology. Sometimes I would wonder what if I do not know so much biology as well as computer science. What if I am only a statistics person, would I be able to do as much as I can do now? Would I be able to collabrate with people and communicate with them efficiently? Would I be able to read math biology papers and computational biology papers with ease? I think the answers to all of them are no.

In our program, not many of professors are well rounded in all these three fields. And it is very hard to find qualified TAs to correct homeworks since this field has just started.

Shuffle Compositions of a Sequence to Remain Di- and Mono- letter Frequencies

2008-02-23T14:58:00.000-08:00

There is a very interesting application of graph in solving a bioinformatics problem raised in our bioinformatics class.

Given a sequence constructed by the alphabet A, T, G, C, how to find all possible sequences which keep both the frequencies of di-nucleotide and mono-nucleotide.

Suppose we have sequence CGTGAGC. Our aim is to remain the occurring number of all possible 16 di-nucleotide frequencies, 4 mono-nucleotide frequencies, as well as the length of the sequence.

This could be viewed as a topology question if we convert it to a graph such that each vertex is a node A, T, G or C, and the occurance of each di-nucleotide is a directed bridge from the first letter to the second one. This idea is displayed in the above figure.

To find a sequence which keeps the di-letter frequencies should be the one starting from one vertex and traversing all edges. If any of the node in the graph has odd number of edges, this node should be the starting point or the ending point (Euler's seven bridge puzzle). While, here, every vertex has even number of edges. Starting from any vertex and traversing all edges would give us a sequence which keeps the di-nucleotide frequencies. However, if and only if one begins with C and ends with C can it give us a sequences which also remains the mono-letter frequencies. We can prove this by contradiction. Suppose we can find a sequence satisfying the di-letter frequencies and not begun with C, then, a C must appear in the middle. With respect to this example, CG must in the middle. Then, another di-letter which ends with C must be right ahead of it in order to connect with CG. So we have GC right ahead of CG, and form GCG. Since in the original sequence, the beginning C and the ending C are counted separately twice, and only once in the new sequence GCG, the mono-letter frequency of C is automatically deducted once. Thus, we could keep the mono-nucleotide frequency if and only if we begin and end the with the same letters of the original one(s).

Interesting Combinatoric Puzzles

2008-02-01T12:27:00.001-08:00

http://www.mathpages.com/home/icombina.htm