\u201cAre men more likely to become high-income professionals than women, i.e., to receive an income of over $50,000 per annum?\u201d<\/em><\/p>\n4.1 \u00a0Getting acquainted with data<\/h3>\n
First, let us read the data:<\/p>\n
file = open('files\/adult.data', 'r')<\/pre>\nChecking the data, we obtain:<\/p>\n
def chr_int(a):\n if a.isdigit():\n return int(a)\n else:\n return 0\n \ndata=[]\nfor line in file:\n data1=line.split(', ')\n if len(data1)==15:\n data.append([chr_int(data1[0]),data1[1],chr_int(data1[2]),data1[3],chr_int(data1[4]),data1[5],data1[6],\\\n data1[7],data1[8],data1[9],chr_int(data1[10]),chr_int(data1[11]),chr_int(data1[12]),data1[13],\\\n data1[14]])<\/pre>\nprint (data[1:2])<\/pre>\n\n- What is the obtained result? What did you ask for in the previous command? Explain.<\/strong><\/li>\n<\/ol>\n
One of the easiest ways to manage data in Python is by using the DataFramestructure, defined in the Pandas <\/em>library, which is a two-dimensional, size-mutable, potentially heterogeneous tabular data structure with labeled axes:<\/p>\n%matplotlib inline<\/span><\/strong>\nimport pandas as pd\n\ndf = pd.DataFrame(data) # Two-dimensional size-mutable, potentially heterogeneous tabular data structure with labeled axes \n\ndf.columns = ['age', 'type_employer', 'fnlwgt', 'education', \n \"education_num\",\"marital\", \"occupation\", \"relationship\", \"race\",\"sex\",\n \"capital_gain\", \"capital_loss\", \"hr_per_week\",\"country\",\"income\"]\ndf.head()<\/pre>\n\n- Describe an explain the result.<\/strong><\/li>\n<\/ol>\n
df.tail()<\/pre>\n\n- Describe and explain the result. Compare with the previous one.<\/strong><\/li>\n<\/ol>\n
The command shape gives exactly the number of data samples (in rows, in this case) and features (in columns):<\/p>\n
df.shape<\/pre>\n\n- Describe an explain the result.<\/strong><\/li>\n<\/ol>\n
counts = df.groupby('country').size()\n\nprint (counts) \n<\/pre>\n5. How many items are there for USA? and for Mexico?<\/strong><\/p>\ncounts = df.groupby('age').size() # grouping by age\nprint (counts)<\/pre>\n6. What is the age of the most represented people?<\/strong><\/p>\nThe first row shows the number of samples with unknown country, followed by the number of samples corresponding to the first countries in the dataset.<\/p>\n
Let us split people according to their gender into two groups: men and women.<\/p>\n
ml = df[(df.sex == 'Male')] # grouping by sex\nml.shape\nml1 = df[(df.sex == 'Male')&(df.income=='>50K\\n')]\nml1.shape<\/pre>\nIf we focus on high-income professionals separated by sex, we can do:<\/p>\n
fm =df[(df.sex == 'Female')]\nfm.shape<\/pre>\nfm1 =df[(df.sex == 'Female')&(df.income=='>50K\\n')]\nfm1.shape\n<\/pre>\ndf1=df[(df.income=='>50K\\n')]\n\nprint ('The rate of people with high income is: ', int(len(df1)\/float(len(df))*100), '%.' )\nprint ('The rate of men with high income is: ', int(len(ml1)\/float(len(ml))*100), '%.' )\nprint ('The rate of women with high income is: ', int(len(fm1)\/float(len(fm))*100), '%.' )<\/pre>\n4.2 Explanatory Data Analysis<\/h2>\n
The data that come from performing a particular measurement on all the subjects in a sample represent our observations for a single characteristic like country, age, education, etc.<\/p>\n
Objective: to visualize and summarize the sample distribution, thereby allowing us to make tentative assumptions about the population distribution.<\/p>\n
4.2.1 Summarizing the Data<\/h3>\n
The data in can be categorical<\/em>for which a simple tabulation of the frequency of each category is the best non-graphical exploration for data analysis. For example, we can ask what is the proportion of high- income professionals in our database:<\/em><\/p>\ndf1=df[(df.income=='>50K\\n')]\n\nprint ('The rate of people with high income is: ', int(len(df1)\/float(len(df))*100), '%.' )\nprint ('The rate of men with high income is: ', int(len(ml1)\/float(len(ml))*100), '%.' )\nprint ('The rate of women with high income is: ', int(len(fm1)\/float(len(fm))*100), '%.' )<\/pre>\n7. Describe an explain the result.<\/strong><\/p>\n\u00a0<\/em>Quantitative<\/em>: exploratory data analysis is a way to make preliminary assessments about the population distribution of the variable. The characteristics of the population distribution of a quantitative variable are its mean<\/em>, deviation<\/em>, histograms<\/em>,outliers<\/em>, etc.<\/p>\n4.2.1.1 Mean<\/h4>\n
In our case, we can consider what the average age of men and women samples in our dataset would be in terms of their mean:<\/p>\n
print ('The average age of men is: ', ml['age'].mean(), '.' )\nprint ('The average age of women is: ', fm['age'].mean(), '.')<\/pre>\nprint ('The average age of high-income men is: ', ml1['age'].mean(), '.' )\nprint ('The average age of high-income women is: ', fm1['age'].mean(), '.')<\/pre>\n8. Describe an explain the result.<\/strong><\/p>\nComment:<\/em>Later, we will work with both concepts: the population mean and the sample mean. We should not confuse them! The first is the mean of samples taken from the population; the second, the mean of the whole population.<\/p>\n4.2.1.2 Sample Variance<\/h4>\n
Usually, mean is not a sufficient descriptor of the data, we can do a little better with two numbers: mean and\u00a0variance<\/strong>:<\/p>\n![\"\"](\"http:\/\/vargas-solar.com\/data-centric-smart-everything\/wp-content\/uploads\/sites\/42\/2018\/09\/Sans-titre.png\")
<\/p>\n
Variance<\/strong>\u00a0\u03c32\u00a0describes the\u00a0spread<\/em>\u00a0of data. The term\u00a0(xi\u2212\u03bc)\u00a0is called the\u00a0deviation from the mean<\/em>, so variance is the mean squared deviation.<\/p>\nThe square root of variance,\u00a0\u03c3, is called the\u00a0standard deviation<\/strong>. We define standard deviation because variance is hard to interpret (in the case the units are grams, the variance is in grams squared).\u00a0Let us compute the mean and the variance of hours per week men and women in our dataset work:<\/p>\nml_mu = ml['age'].mean()\nfm_mu = fm['age'].mean()\nml_var = ml['age'].var()\nfm_var = fm['age'].var()\nml_std = ml['age'].std()\nfm_std = fm['age'].std()\n\nprint ('Statistics of age for men: mu:', ml_mu, 'var:', ml_var, 'std:', ml_std)\nprint ('Statistics of age for women: mu:', fm_mu, 'var:', fm_var, 'std:', fm_std)<\/pre>\nml_mu_hr = ml['hr_per_week'].mean()\nfm_mu_hr = fm['hr_per_week'].mean()\nml_var_hr = ml['hr_per_week'].var()\nfm_var_hr = fm['hr_per_week'].var()\nml_std_hr = ml['hr_per_week'].std()\nfm_std_hr = fm['hr_per_week'].std()\n\nprint ('Statistics of hours per week for men: mu:', ml_mu_hr, 'var:', ml_var_hr, 'std:', ml_std_hr)\nprint ('Statistics\u00a0of hours per week for women: mu:', fm_mu_hr, 'var:', fm_var_hr, 'std:', fm_std_hr)<\/pre>\n9. Describe an explain the result.<\/strong><\/p>\n4.2.1.3 Sample Median<\/h4>\n
What will happen if in the sample set there is an error with a value very different from the rest? For example, considering hours worked per week, it would normally be in a range between 20 and 80; but what would happen if by mistake there was a value of 1000? An item of data that is significantly different from the rest of the data is called an outlier<\/em>. In this case, the mean, \u03bc, will be drastically changed towards the outlier. One solution to this drawback is offered by the statistical median<\/em>, \u03bc12, which is an order statistic giving the middle value of a sample. In this case, all the values are ordered by their magnitude and the median is defined as the value that is in the middle of the ordered list. Hence, it is a value that is much more robust in the face of outliers.<\/p>\nLet us see, the median age of working men and women in our dataset and the median age of high-income men and women:<\/p>\n
ml_median= ml['age'].median()\nfm_median= fm['age'].median()\n\nprint (\"Median age per men and women: \", ml_median, fm_median)<\/pre>\nml_median_age= ml1['age'].median()\nfm_median_age= fm1['age'].median()\n\nprint (\"Median age per men and women with high-income: \", ml_median_age, fm_median_age)<\/pre>\nml_median_hr= ml['hr_per_week'].median()\nfm_median_hr= fm['hr_per_week'].median()\nprint (\"Median hours per week per men and women: \", ml_median_hr, fm_median_hr)<\/pre>\n10. Describe an explain the result.<\/strong><\/p>\n4.2.1.4 Quantiles and Percentiles<\/h4>\n
Order the sample\u00a0{xi<\/sub>}, then find\u00a0xp<\/sub>\u00a0so that it divides the data into two parts where:<\/p>\n\n- a fraction\u00a0p\u00a0of the data values are less than or equal to\u00a0xp<\/sub>\u00a0and<\/li>\n
- the remaining fraction\u00a0(1\u2212p)\u00a0are greater than\u00a0xp<\/sub>.<\/li>\n<\/ul>\n
That value\u00a0xp<\/sub>\u00a0is the pth-quantile, or 100\u00d7pth percentile.<\/p>\n5-number summary<\/strong>:\u00a0xmin<\/sub>,Q1<\/sub>,Q2<\/sub>,Q3<\/sub>,xmax<\/sub>, where\u00a0Q1<\/sub>\u00a0is the 25\u00d7pth percentile,\u00a0Q2<\/sub>\u00a0is the 50\u00d7pth percentile and\u00a0Q3<\/sub>\u00a0is the 75\u00d7pth percentile.<\/p>\n4.2.1.5 Data distributions<\/h4>\n
We can have a look at the data distribution, which describes how often each value appears (i.e., what is its frequency).<\/p>\n
The most common representation of a distribution is a histogram<\/em>, which is a graph that shows the frequency of each value. Let us show the age of working men and women separately.<\/p>\nimport matplotlib.pyplot as plt\nml_age=ml['age']\nml_age.hist(histtype='stepfilled', bins=20)<\/pre>\n10. Show the graphics and an explain the result.<\/strong><\/p>\nfm_age=fm['age']\nfm_age.hist(histtype='stepfilled', bins=10)\nplt.xlabel('Age',fontsize=15)\nplt.ylabel('Female samples',fontsize=15)\nplt.show()<\/pre>\n11. Show the graphics and an explain the result.<\/strong><\/p>\nWe can normalize the frequencies of the histogram by dividing\/normalizing by n<\/em>, the number of samples. The normalized histogram is called the Probability Mass Function <\/em>(PMF).<\/p>\nimport seaborn as sns\nfm_age.hist(histtype='stepfilled', alpha=.5, bins=20) # default number of bins = 10\nml_age.hist(histtype='stepfilled', alpha=.5, color=sns.desaturate(\"indianred\", .75), bins=10)\nplt.xlabel('Age',fontsize=15)\nplt.ylabel('Samples',fontsize=15)\nplt.show()<\/pre>\n12. Show the graphics and an explain the result.<\/strong><\/p>\nThe Cumulative Distribution Function <\/em>(CDF), or just distribution function, describes the probability that a real-valued random variable X <\/em>with a given proba- bility distribution will be found to have a value less than or equal to x<\/em>. Let us show the CDF of age distribution for both men and women.<\/p>\nfm_age.hist(histtype='stepfilled', alpha=.5, bins=20) # default number of bins = 10\nml_age.hist(histtype='stepfilled', alpha=.5, color=sns.desaturate(\"indianred\", .75), bins=10)\nplt.xlabel('Age',fontsize=15)\nplt.ylabel('PMF',fontsize=15)\nplt.show()<\/pre>\n13. Show the graphics and an explain the result.<\/strong><\/p>\n4.2.2 Data distributions<\/h3>\n
Summarizing can be dangerous: very different data can be described by the same statistics. It must be validated by inspecting the data.<\/p>\n
We can look at the\u00a0data distribution<\/strong>, which describes how often (frequency) each value appears.<\/p>\nWe can normalize the frequencies of the histogram by dividing\/normalizing by\u00a0n, the number of samples. The normalized histogram is called\u00a0Probability Mass Function (PMF)<\/strong>.<\/p>\nLet’s visualize and compare the MPF of male and female age in our example:<\/p>\n
ml_age.hist(histtype='stepfilled', bins=20)\n\nplt.xlabel('Age',fontsize=15)\nplt.ylabel('Probability',fontsize=15)\nplt.show()\n<\/pre>\n14. Show the graphics and an explain the result.<\/strong><\/p>\nfm_age.hist(histtype='stepfilled', bins=20)\n\nplt.xlabel('Age',fontsize=15)\nplt.ylabel('Probability',fontsize=15)\nplt.show()<\/pre>\n15. Show the graphics and an explain the result.<\/strong><\/p>\nThe\u00a0cumulative distribution function (CDF)<\/strong>, or just distribution function, describes the probability that a real-valued random variable X with a given probability distribution will be found to have a value less than or equal to x. For our example, the CDFs will be:<\/p>\nml_age.hist(histtype='step', cumulative=True, linewidth=3.5, bins=20)\n\nplt.xlabel('Age',fontsize=15)\nplt.ylabel('CDF',fontsize=15)\nplt.show()<\/pre>\n16. Show the graphics and an explain the result.<\/strong><\/p>\nfm_age.hist(histtype='step', cumulative=True, linewidth=3.5, bins=20)\n\nplt.xlabel('Age',fontsize=15)\nplt.ylabel('CDF',fontsize=15)\nplt.show()<\/pre>\n17. Show the graphics and an explain the result.<\/strong><\/p>\nml_age.hist(bins=10, histtype='stepfilled', alpha=.5) # default number of bins = 10\nfm_age.hist(bins=10, histtype='stepfilled', alpha=.5, color=sns.desaturate(\"indianred\", .75))\nplt.xlabel('Age',fontsize=15)\nplt.ylabel('Probability',fontsize=15)\nplt.show()<\/pre>\n18. Show the graphics and an explain the result.<\/strong><\/p>\nml_age.hist(histtype='step', cumulative=True, linewidth=3.5, bins=20)\nfm_age.hist(histtype='step', cumulative=True, linewidth=3.5, bins=20, color=sns.desaturate(\"indianred\", .75))\nplt.xlabel('Age',fontsize=15)\nplt.ylabel('CDF',fontsize=15)\nplt.show()<\/pre>\n19. Show the graphics and an explain the result.<\/strong><\/p>\nprint (\"The mean sample difference is \", ml_age.mean() - fm_age.mean())<\/pre>\n20. Explain the result.<\/strong><\/p>\n4.2.3 Outliers<\/h3>\n
Ouliers<\/strong>\u00a0are data samples with a value that is far from the central tendency.<\/p>\nWe can find outliers by:<\/p>\n
\n- Computing samples that are\u00a0far<\/em>\u00a0from the median.<\/li>\n
- Computing samples whose value\u00a0exceeds the mean<\/em>\u00a0by 2 or 3 standard deviations.<\/li>\n<\/ul>\n
This expression will return a series of boolean values that you can then index the series by:<\/p>\n
df['age'].median()<\/pre>\n4.2.3.1 Outliner Treatment<\/h4>\n
For example, in our case, we are interested in the age statistics of men versus women with high incomes and we can see that in our dataset, the minimum age is 17 years and the maximum is 90 years. We can consider that some of these samples are due to errors or are not representable. Applying the domain knowledge, we focus on the median age (37, in our case) up to 72 and down to 22 years old, and we consider the rest as outliers.<\/p>\n
Let’s see how many outliers we can detect in our example:<\/p>\n
len(df[(df.income == '>50K\\n') & (df['age'] < df['age'].median() - 15)])<\/pre>\nlen(df[(df.income == '>50K\\n') & (df['age'] > df['age'].median() + 35)])<\/pre>\nIf we think that outliers correspond to errors, an option is to trim the data by discarting the highest and lowest values.<\/p>\n
df2 = df.drop(df.index[(df.income=='>50K\\n') & (df['age']>df['age'].median() +35) & (df['age'] > df['age'].median()-15)])\n\ndf2.shape<\/pre>\nml1_age=ml1['age']\nfm1_age=fm1['age']<\/pre>\nml2_age = ml1_age.drop(ml1_age.index[(ml1_age >df['age'].median()+35) & (ml1_age>df['age'].median() - 15)])\n\nfm2_age = fm1_age.drop(fm1_age.index[(fm1_age > df['age'].median()+35) & (fm1_age > df['age'].median()- 15)])<\/pre>\nmu2ml = ml2_age.mean()\nstd2ml = ml2_age.std()\nmd2ml = ml2_age.median()\n\n# Computing the mean, std, median, min and max for the high-income male population\n\nprint (\"Men statistics: Mean:\", mu2ml, \"Std:\", std2ml, \"Median:\", md2ml, \"Min:\", ml2_age.min(), \"Max:\",ml2_age.max())<\/pre>\nmu3ml = fm2_age.mean()\nstd3ml = fm2_age.std()\nmd3ml = fm2_age.median()\n\n# Computing the mean, std, median, min and max for the high-income female population\nprint (\"Women statistics: Mean:\", mu2ml, \"Std:\", std2ml, \"Median:\", md2ml, \"Min:\", fm2_age.min(), \"Max:\",fm2_age.max())<\/pre>\nprint ('The mean difference with outliers is: %4.2f.'% (ml_age.mean() - fm_age.mean()))\nprint (\"The mean difference without outliers is: %4.2f.\"% (ml2_age.mean() - fm2_age.mean()))<\/pre>\nLet us compare visually the age distributions before and after removing the outliers:<\/p>\n
plt.figure(figsize=(13.4,5))\n\ndf.age[(df.income == '>50K\\n')].plot(alpha=.25, color='blue')\ndf2.age[(df2.income == '>50K\\n')].plot(alpha=.45,color='red')\n\nplt.ylabel('Age')\nplt.xlabel('Samples')<\/pre>\nLet us see what is happening near the mode:<\/p>\n
import <\/strong>numpy as <\/strong>np\n\ncountx,divisionx = np.histogram(ml2_age)\ncounty,divisiony = np.histogram(fm2_age)<\/pre>\nimport <\/strong>matplotlib.pyplot as <\/strong>plt\n\nval = [(divisionx[i]+divisionx[i+1])\/2 for <\/strong>i in <\/strong>range(len(divisionx)-1)]\n\nplt.plot(val, countx-county,'o-')\nplt.title('Differences in promoting men vs. women')\nplt.xlabel('Age',fontsize=15)\nplt.ylabel('Differences',fontsize=15)\nplt.show()<\/pre>\nThere is still some evidence for our hypothesis!<\/p>\n
print (\"Remember:\\n We have the following mean values for men, women and the difference:\\nOriginally: \", ml_age.mean(), fm_age.mean(), ml_age.mean()- fm_age.mean()) # The difference between the mean values of male and female populations.)\nprint (\"For high-income: \", ml1_age.mean(), fm1_age.mean(), ml1_age.mean()- fm1_age.mean()) # The difference between the mean values of male and female populations.)\nprint (\"After cleaning: \", ml2_age.mean(), fm2_age.mean(), ml2_age.mean()- fm2_age.mean()) # The difference between the mean values of male and female populations.)\n\nprint (\"\\nThe same for the median:\")\nprint (ml_age.median(), fm_age.median(), ml_age.median()- fm_age.median()) # The difference between the mean values of male and female populations.)\nprint (ml1_age.median(), fm1_age.median(), ml1_age.median()- fm1_age.median()) # The difference between the mean values of male and female populations.)\nprint (ml2_age.median(), fm2_age.median(), ml2_age.median()- fm2_age.median()), # The difference between the mean values of male and female populations.)<\/pre>\n4.2.1.7 Measuring Asymmetry<\/h4>\n
For univariate data, the formula for skewness <\/em>is a statistic that measures the asymmetry of the set of n <\/em>data samples, xi <\/em>:<\/p>\n![\"\"](\"http:\/\/vargas-solar.com\/data-centric-smart-everything\/wp-content\/uploads\/sites\/42\/2018\/09\/standardDeviation.png\")
<\/p>\n
where \u03bcis the mean, \u03c3is the standard deviation, and n<\/em>is the number of data points. The numerator is the mean squared deviation (or variance) and the denominator the mean cubed deviation.<\/p>\nNegative deviation indicates that the distribution \u201cskews left\u201d (it extends further to the left than to the right). One can easily see that the skewness for a normal distribution is zero, and any symmetric data must have a skewness of zero. Note that skewness can be affected by outliers! A simpler alternative is to look at the relationship between the mean \u03bcand the median \u03bc1\/2<\/sub>.<\/p>\ndef skewness(x):\n res=0\n m=x.mean()\n s=x.std()\n for i in x:\n res+=(i-m)*(i-m)*(i-m)\n res\/=(len(x)*s*s*s)\n return res\n\nprint (\"The skewness of the male population is:\", skewness(ml2_age))\nprint (\"The skewness of the female population is:\", skewness(fm2_age))<\/pre>\n\n- Explain the result<\/strong><\/li>\n<\/ol>\n
The Pearson\u2019s median skewness coefficient <\/strong>is a more robust alternative to the skewness coefficient and is defined as follows:<\/p>\n![\"\"](\"http:\/\/vargas-solar.com\/data-centric-smart-everything\/wp-content\/uploads\/sites\/42\/2018\/09\/skewness.png\")
<\/p>\n
There are many other definitions for skewness that will not be discussed here. In our case, if we check the Pearson\u2019s skewness coefficient for both men and women, we can see that the difference between them actually increases:<\/p>\n
def pearson(x):\n return 3*(x.mean()-x.median())\/x.std()\n\nprint (\"The Pearson's coefficient of the male population is:\", pearson(ml2_age))\nprint (\"The Pearson's coefficient of the female population is:\", pearson(fm2_age))<\/pre>\n4.2.1.8 Relative Risk<\/h4>\n
Let’s say that a person is “early” promoted if he\/she is promoted before the age of 41, “on time” if he\/she is promoted of age 41, 42, 43 or 44, and “late” promoted if he\/she is ascended to get income bigger than 50K after being 44 years old. Let us compute the probability of being early, on time and late promoted for men and women:<\/p>\n
ml1 = df[(df.sex == 'Male')&(df.income=='>50K\\n')]\n\nml2 = ml1.drop(ml1.index[(ml1['age']>df['age'].median() +35)&(ml1['age']> df['age'].median()- 15)])\n\nfm2 = fm1.drop(fm1.index[(fm1['age']> df['age'].median() + 35)& (fm1['age']> df['age'].median() - 15)])\n\nprint (ml2.shape, fm2.shape)<\/pre>\nprint (\"Men grouped in 3 categories:\")\nprint (\"Young:\",int(round(100*len(ml2_age[ml2_age<41])\/float(len(ml2_age.index)))),\"%.\")\nprint (\"Elder:\", int(round(100*len(ml2_age[ml2_age >44])\/float(len(ml2_age.index)))),\"%.\")\nprint (\"Average age:\", int(round(100*len(ml2_age[(ml2_age>40) & (ml2_age< 45)])\/float(len(ml2_age.index)))),\"%.\")<\/pre>\nprint (\"Women grouped in 3 categories:\")\nprint (\"Young:\",int(round(100*len(fm2_age[fm2_age <41])\/float(len(fm2_age.index)))),\"%.\")\nprint (\"Elder:\", int(round(100*len(fm2_age[fm2_age >44])\/float(len(fm2_age.index)))),\"%.\")\nprint (\"Average age:\", int(round(100*len(fm2_age[(fm2_age>40) & (fm2_age< 45)])\/float(len(fm2_age.index)))),\"%.\")<\/pre>\nThe\u00a0relative risk<\/strong>\u00a0is the ratio of two probabilities. In order to get the relative risk \u00a0of early promotion, we need to consider the fraction of both probabilities.<\/p>\nprint (\"The male mean:\", ml2_age.mean())\nprint (\"The female mean:\", fm2_age.mean())<\/pre>\nml2_young = len(ml2_age[(ml2_age<41)])\/float(len(ml2_age.index))\nfm2_young = len(fm2_age[(fm2_age<41)])\/float(len(fm2_age.index))\nprint (\"The relative risk of female early promotion is: \", 100*(1-ml2_young\/fm2_young))<\/pre>\nml2_elder = len(ml2_age[(ml2_age>44)])\/float(len(ml2_age.index))\nfm2_elder = len(fm2_age[(fm2_age>44)])\/float(len(fm2_age.index))\nprint (\"The relative risk of male late promotion is: \", 100*ml2_elder\/fm2_elder)<\/pre>\nDiscussion<\/h4>\n
After exploring the data, we obtained some apparent effects that support our initial assumptions. For example, the mean age for men in our dataset is 39.4 years; while for women, is 36.8 years. When analyzing the high-income salaries, the mean age for men increased to 44.6 years; while for women, increased to 42.1 years. When the data were cleaned from outliers, we obtained mean age for high-income men: 44.3, and for women: 41.8. Moreover, histograms and other statistics show the skewness of the data and the fact that women used to be promoted a little bit earlier than men, in general.<\/p>\n
4.2.5 Continuous Distribution<\/h3>\n
The distributions we have considered up to now are based on empirical observations and thus are called empirical distributions<\/em>. As an alternative, we may be interested in considering distributions that are defined by a continuous function and are called continuous distributions<\/em>.<\/p>\n4.2.5.1 Exponential distribution<\/h4>\n
The CDF of the exponential distribution is:<\/p>\n
CDF(x)=1\u2212exp\u2212<\/sup>\u03bb<\/sup>x<\/sup><\/p>\nAnd its PDF is:<\/p>\n
PDF(x)=\u03bbexp\u2212<\/sup>\u03bb<\/sup>x<\/sup><\/p>\nThe parameter\u00a0\u03bb\u00a0determines the shape of the distribution, the mean of the distribution is\u00a01\/\u03bb\u00a0and its variance is\u00a01\/\u03bb2<\/sup>. The median is\u00a0ln(2)\/\u03bb.<\/p>\nl = 3\nx=np.arange(0,2.5,0.1)\ny= 1- np.exp(-l*x)\n\nplt.plot(x,y,'-')\nplt.title('Exponential CDF: $\\lambda$ =%.2f'% l ,fontsize=15)\nplt.xlabel('x',fontsize=15)\nplt.ylabel('CDF',fontsize=15)\nplt.show()<\/pre>\nfrom __future__ import division\nimport scipy.stats as stats\n\nl = 3\nx=np.arange(0,2.5,0.1)\ny= l * np.exp(-l*x)\n\nplt.plot(x,y,'-')\nplt.title('Exponential PDF: $\\lambda$ =%.2f'% l, fontsize=15)\nplt.xlabel('x', fontsize=15)\nplt.ylabel('PDF', fontsize=15)\nplt.show()<\/pre>\n\u00a0<\/strong>There are a lot of real world events that can be described with this distribution.<\/p>\n\n- The time until a radioactive particle decays,<\/li>\n
- The time it takes before your next telephone call,<\/li>\n
- The time until default (on payment to company debt holders) in reduced form credit risk modeling.<\/li>\n<\/ul>\n
The random variable\u00a0X\u00a0of the life lengths of some batteries is associated with a probability density function of the form:<\/p>\n
![\"\"](\"http:\/\/vargas-solar.com\/data-centric-smart-everything\/wp-content\/uploads\/sites\/42\/2018\/09\/PDF.png\")
<\/p>\n
l = 0.25\n\nx=np.arange(0,25,0.1)\ny= l * np.exp(-l*x)\n\nplt.plot(x,y,'-')\nplt.title('Exponential: $\\lambda$ =%.2f' %l ,fontsize=15)\nplt.xlabel('x',fontsize=15)\nplt.ylabel('PDF',fontsize=15)\nplt.show()<\/pre>\n4.2.5.2 Normal distribution<\/span><\/h4>\nThe\u00a0<\/span>normal, or Gaussian distribution<\/strong>\u00a0<\/span>is the most used one because it describes a lot of phenomena and because it is amenable for analysis.\u00a0<\/span><\/span><\/p>\nIts CDF has no closed-form expression and its more common representation is the PDF:<\/span><\/p>\n![\"\"](\"http:\/\/vargas-solar.com\/data-centric-smart-everything\/wp-content\/uploads\/sites\/42\/2018\/09\/gaussian.png\")
<\/p>\n
u=6 # mean\ns=2 # standard deviation\n\nx=np.arange(0,15,0.1)\n\ny=(1\/(np.sqrt(2*np.pi*s*s)))*np.exp(-(((x-u)**2)\/(2*s*s)))\n\nplt.plot(x,y,'-')\nplt.title('Gaussian PDF: $\\mu$=%.1f, $\\sigma$=%.1f'%(u,s),fontsize=15)\nplt.xlabel('x',fontsize=15)\nplt.ylabel('Probability density',fontsize=15)\nplt.show()<\/pre>\nExamples:<\/p>\n
\n- Measures of size of living tissue (length, height, skin area, weight);<\/li>\n
- The length of inert appendages (hair, claws, nails, teeth) of biological specimens, in the direction of growth; presumably the thickness of tree bark also falls under this category;<\/li>\n
- Certain physiological measurements, such as blood pressure of adult humans.<\/li>\n<\/ul>\n
4.2.5.3 Central limit theorem<\/h4>\n
The normal distribution is also important, because it is involved in the Central Limit Theorem:<\/p>\n
Take the mean of\u00a0n\u00a0random samples from ANY arbitrary distribution with a\u00a0well-defined\u00a0standard deviation\u00a0\u03c3\u00a0and mean\u00a0\u03bc. As\u00a0n\u00a0gets bigger the\u00a0distribution of the sample mean<\/strong>\u00a0will always converge to a Gaussian (normal) distribution with mean\u00a0\u03bc\u00a0and standard deviation\u00a0\u03c3n\u221a.<\/p>\nColloquially speaking, the theorem states the distribution of an average tends to be normal, even when the distribution from which the average is computed is decidedly non-normal. This explains the ubiquity of the Gaussian distribution in science and statistics.<\/p>\n
Example: Uniform Distribution<\/h4>\n
The uniform distribution is obviously non-normal. Let us call it the\u00a0parent\u00a0distribution.<\/p>\n
To compute an average, two samples are drawn (n=2), at random, from the parent distribution and averaged. Then another sample of two is drawn and another value of the average computed. This process is repeated, over and over, and averages of two are computed.<\/p>\n
Repeatedly taking more elements (n=3,4…) from the parent distribution, and computing the averages, produces a normal probability density.<\/p>\n
fig, ax = plt.subplots(1, 4, sharey=True, squeeze=True, figsize=(14, 5))\nx = np.linspace(0, 1, 100)\nfor i in range(4):\n f = np.mean(np.random.random((10000, i+1)), 1)\n m, s = np.mean(f), np.std(f, ddof=1)\n fn = (1\/(s*np.sqrt(2*np.pi)))*np.exp(-(x-m)**2\/(2*s**2)) # normal pdf \n ax[i].hist(f, 40, color=[0, 0.2, .8, .6]) \n ax[i].set_title('n=%d' %(i+1))\n ax[i].plot(x, fn, color=[1, 0, 0, .6], linewidth=5)\nplt.suptitle('Demonstration of the central limit theorem for a uniform distribution', y=1.05)\nplt.show()<\/pre>\n4.2.6\u00a0\u00a0\u00a0\u00a0\u00a0 Kernel Density<\/h3>\n
In many real problems, we may not be interested in the parameters of a particular distribution of data, but just a continuous representation of the data. In this case, we should estimate the distribution non-parametrically (i.e., making no assumptions about the form of the underlying distribution) using kernel density estimation.<\/p>\n
from scipy.stats.distributions import norm\n\n# Some random data\ny = np.random.random(15) * 10\nx = np.linspace(0, 10, 100)\n\nx1 = np.random.normal(-1, 2, 15) # parameters: (loc=0.0, scale=1.0, size=None)\nx2 = np.random.normal(6, 3, 10)\ny = np.r_[x1, x2] # r_ Translates slice objects to concatenation along the first axis.\nx = np.linspace(min(y), max(y), 100)\n\n# Smoothing parameter\ns = 0.4\n\n# Calculate the kernels\nkernels = np.transpose([norm.pdf(x, yi, s) for yi in y])\n\nplt.plot(x, kernels, 'k:')\nplt.plot(x, kernels.sum(1), 'r')\nplt.plot(y, np.zeros(len(y)), 'go', ms=10)<\/pre>\nLet us imagine that we have a set of data measurements without knowing their distribution and we need to estimate the continuous representation of their distribution. In this case, we can consider a Gaussian kernel to generate the density around the data. Let us consider a set of random data generated by a bimodal normal distribution. If we consider a Gaussian kernel around the data, the sum of those kernels can give us a continuous function that when normalized would approximate the density of the distribution:<\/p>\n
from <\/strong>scipy.stats import <\/strong>kde\n\nx1 = np.random.normal(-1, 0.5, 15)\n\n# parameters: (loc=0.0, scale=1.0, size=None)\n\nx2 = np.random.normal(6, 1, 10)\ny = np.r_[x1, x2]\n\n# r_ Translates slice objects to concatenation along the first axis.\n\nx = np.linspace(min(y), max(y), 100)\ns = 0.4 # Smoothing parameter\n\nkernels = np.transpose([norm.pdf(x, yi, s) for yi in y])\n\n# Calculate the kernels\ndensity = kde.gaussian_kde(y)\n\nplt.plot(x, kernels, 'k:')\nplt.plot(x, kernels.sum(1), 'r')\nplt.plot(y, np.zeros(len(y)), 'bo', ms=10)<\/pre>\n\n- What does the figure shows?<\/strong><\/li>\n<\/ol>\n
xgrid = np.linspace(x.min(), x.max(), 200)\nplt.hist(y, bins=28)\nplt.plot(xgrid, density(xgrid), 'r-')<\/pre>\nSciPy implements a Gaussian KDE that automatically chooses an appropriate bandwidth. Let’s create a bi-modal distribution of data that is not easily summarized by a parametric distribution:<\/p>\n
# Create a bi-modal distribution with a mixture of Normals.\n\nx1 = np.random.normal(-1, 2, 15) # parameters: (loc=0.0, scale=1.0, size=None)\nx2 = np.random.normal(6, 3, 10)\n\n# Append by row\nx = np.r_[x1, x2]\n\n# r_ Translates slice objects to concatenation along the first axis.\nplt.hist(x, bins=18)<\/pre>\nIn fact, the library SciPy3 implements a Gaussian kernel density estimation that automatically chooses the appropriate bandwidth parameter for the kernel. Thus, the final construction of the density estimate will be obtained by:<\/p>\n
density = kde.gaussian_kde(x)\nxgrid = np.linspace(x.min(), x.max(), 200)\nplt.hist(x, bins=18)\nplt.plot(xgrid, density(xgrid), 'r-')<\/pre>\n4.2\u00a0\u00a0\u00a0\u00a0 Estimation<\/h3>\n
An important aspect when working with statistical data is being able to use estimates to approximate the values of unknown parameters of the dataset. In this section, we will review different kinds of estimators (estimated mean, variance, standard score, etc.).<\/p>\n
x = np.random.normal(0.0, 1.0, 10000)\na = plt.hist(x,50)<\/pre>\nDefinition:<\/strong>\u00a0Estimation<\/em>\u00a0is the process of inferring the parameters (e.g. mean) of a distribution from a statistic of samples drown from a population.<\/p>\nFor example: What is the estimated mean\u00a0\u03bc\u0302\u00a0of the following normal data?<\/p>\n
We can use our definition of empirical mean:<\/p>\n
print ('The empirical mean of the sample is ', x.mean())<\/pre>\n4.3.1 Sample and Estimated Mean, Variance and Standard Scores<\/h3>\n
In continuation, we will deal with point estimators that are single numerical estimates of parameters of a population.<\/p>\n
4.3.1.1 Mean<\/h4>\n
Let us assume that we know that our data are coming from a normal distribution and the random samples drawn are as follows:<\/p>\n
{0.33, \u22121.76, 2.34, 0.56, 0.89}.<\/p>\n
The question is can we guess the mean \u03bc of the distribution? One approximation is given by the sample mean, x \u0304. This process is called estimation and the statistic (e.g., the sample mean) is called an estimator. In our case, the sample mean is 0.472, and it seems a logical choice to represent the mean of the distribution. It is not so evident if we add a sample with a value of \u2212465. In this case, the sample mean will be \u221277.11, which does not look like the mean of the distribution. The reason is due to the fact that the last value seems to be an outlier compared to the rest of the sample. In order to avoid this effect, we can try first to remove outliers and then to estimate the mean; or we can use the sample median as an estimator of the mean of the distribution. If there are no outliers, the sample mean x \u0304 minimizes the following mean squared error:<\/p>\n
![\"\"](\"http:\/\/vargas-solar.com\/data-centric-smart-everything\/wp-content\/uploads\/sites\/42\/2018\/09\/MSE.png\")
<\/p>\n
where n is the number of times we estimate the mean. Let us compute the MSE of a set of random data:<\/p>\n
NTs=200\nmu=0.0\nvar=1.0\nerr = 0.0\nNPs=1000\nfor i in range(NTs):\n x = np.random.normal(mu, var, NPs)\n err += (x.mean()-mu)**2\n\nprint ('MSE: ', err\/NTs)<\/pre>\n\n- What do you obtained as result?<\/strong><\/li>\n<\/ol>\n
4.3.1.2 Variance<\/h4>\n
We can also estimate the variance with:<\/p>\n
\u03c3\u0302\u00a0