Programming for Data Analysis and Visualization

Programming for Data Analysis and Visualization

CA 2

Submit one single R file for the solution of the following questions as: Firstname.Surename.R
Q1.
The dataset “Power Plant” records variables which the company’s engineers believe are
important factors in the operation of the plant. The company is interested in maximising net
hourly electrical energy output (recorded as PE in the dataset). For each hour of energy
output recorded, other variable “Temperature” (AT) in the range 1.81°C and 37.11°C is
recorded.
Steps:

  1. Run a linear regression model for PE over AT. Record the value for the slope
    and take it as the actual population parameter .
  2. For 1000 iterations:
    a. Take 50 random samples from the dataset. Run the regression model and
    using the expression for CI for , that we found in the lecture, find a 95%
    CI for .
    b. Find what percentage of the CIs generated in step 2 would contain the
    that you got in step 1.

Q2.
If and are independent random samples from the Uniform distribution U(0,1), by
generating random samples find | −
| <

.

Q3.
If
are independent random samples from the Beta distribution (1, 1 + ), by
generating random samples for 3 different values for find

∑ ln (1 −
)

∑ ln (1 −
)

and show that the result is independent from .
b. Using the distribution in Q2, show that the result is even independent from the
distribution of
.

Python data mining

Your project proposal must be typed and should be approximately one page long. The purpose of the proposal is to help you sort and summarize your project ideas, and select your most interested data mining topic for project. We will review your project proposal and make sure you are on the right track. After submitting the project proposal, you will need to discuss with me to confirm and finalize your project topic and directions at Office Hours. We will give you project feedback comments so you can complete a high-quality data mining project.In your proposal you should cover the following items:• Tentative title of the project.• Abstract for your project topic. It should be one paragraph long, and should provide a high level summary of your project and outline your main goals. What is the major data mining problem and why it is meaningful to perform data mining on this data or topic?  • Brief description of project plan.1. What data sets do you plan to use? Describe the data briefly and provide the information of the data sources. We do not require significant effort on data collection and processing in this project. You can use data sets from UCI, Kaggle, or other public datasets on your interested topics, such as healthcare, energy, manufacturing, etc.  2. If you need do significant work to process raw data and convert it into the proper format for data mining. You can describe the expected data processing step.3. What programming languages do you plan to use (Matlab/Python/R)? What other machine learning tools do you also plan to use (e.g., WEKA, Tableau, SAS, etc. This is optional.)   4. How do you formulate the data mining problem? E.g., is it a classification task for discrete class labels, or a regression/prediction task for continuous response variables? You can also do both classification and regression on one dataset. For example, you can discretize continuous response variable into multiple categories (such as low, medium, high), then we can convert the problem into a classification problem, and implement classification models. 5. Note describe what exactly are you trying to predict or classify. It is critical that your problem is well-defined. 6. What data mining methods tentatively to be implemented for the project? (e.g., decision trees, KNN, Bayesian decision rules, LDA, neural networks, SVM, Neural Networks etc.) We would like you to practice different classification/prediction models on your project, and compare the performance of different models. This is just a draft plan, and you can add more models later when you make more progress on your project.  7. Indicate what types of projects you are going to do. Research project or application-based project. 
Types of ProjectsThere are two main types of projects. Research Project: you can decide to do a research project, where you look at a research issue. This could be original research, but could also be something straightforward—such as an empirical evaluation of data mining methods or strategies for improving performance (e.g., a study about strategies for removing missing values, evaluate different feature selection algorithms using simulated and real-world datasets, explore recent machine learning and deep learning methods on some research data). If you would like to do research project, we could provide some research dataset for you to explore. And also provide some new data mining ideas to explore. This option mainly applies to PhD students and senior MS students with good programming skills.Application Based Project: this is the most common project format and many of you will select application-based project to explore some real-world data sets using learned data mining models and methods. You can select something interesting for data mining, practice essential data mining steps, including data preprocessing, data visualization, variable selection (optional), classification/prediction modeling, model parameter tuning, and model performance evaluation. You should make sure that your analysis is not trivial, and explore some meaning data mining tasks. For example, running a data set through WEKA and spending an hour on the analysis and then doing a quick write-up would be considered trivial. You should study the dataset, determine the issues, address any preprocessing issues, try multiple modeling techniques, and perhaps take some creative steps to try to improve the classification or predictive performance.
Project ReportEach team will complete a data mining report at the end of the semester. It is very important for everyone to learn scientific writing for technical report. This is an important skill for your future work. The project report need be well organized and clearly written. The following report sections can be taken as a reasonable template for your project report writing.• Abstract: summarizes the project and the goals of the data mining work (required)• Introduction: Introduces the project and what you are trying to do. Also include relevant information to introduce the data mining problems and why it is a meaningful topic. What are the motivations people do data mining on this topic. • Background: you may want a separate background section to provide domain information for the topic that you are studying. You can describe with citations to relevant papers, documents, or web recourses. For public datasets on an interesting topic, you can always find a lot of related work. Assume you are writing a technical paper to public readers, you can introduce the domain knowledge and problem background information clearly to help readers understand the problem and the filed. You can also combine background and introduction into one section (with sub-sections).• Dataset Description: Describes the experiments and the experimental setup for data collection based on the documents from data recourses. Will describe the explored data sets in details. • Data Mining Experiments: in this section, describe data mining experiments you have done, such as data processing, feature extraction, feature selection, data mining models and tools, data mining strategies you explored, the evaluation metrics, and any other work related to the data mining experiments.• Experimental Results: summarize the experiment results of different models and methods/ideas. A discussion of the results may be included. • Conclusion: Provide your conclusion. For example, comment on the quality of your results. You may also want to include some material on future work, whether or not you intend to do such work. A high quality data mining project may generate a conference or journal paper after the class. • References: you may cite some papers and documents/website in the sections above. Make a reference list with clear index.