week3-data-exploration-and-analysis

Type: lecturenote

Lecture Notes:

Key Ethical Questions:
- Consent: Should we always ask permission?
- Privacy: how do we protect anonymity?
- Representation: Whose voices are included/excluded?
- Benefit: Who gains from this research?
- Cultural Resources: What about indigenous language/discourse?

Risk Assessment	Requirement
Lawfulness	Processed lawfully, fairly, and in a transparent manner
Data Minimization	Collected for specified explicit and legitimate purposes
Accuracy	Accurate and where necessary, kept up to date
Purpose Limitations	Adequate, relevant and limited to what is necessary
Storage Limitation	Retained only for a time that is necessary
Confidentiality	Processed in an appropriate manner to maintain security
Accountability	Supported by the further principle of accountability to customers and employees

A corpus is simply a collection of texts stored digitally.
- Plus metadata — information about the texts:
  - Who wrote/translated it
  - When it was created
  - What language it’s in

Symbolic Methods	Statistical Learning
Formal logic & AI	Probability theory & statistics
Handcrafted parsers	Frequency analysis
Ontologies (WordNet)	Collocation detection
Mostly disappeared	Machine learning
Still used in semantics	Neural networks
	Dominates the field

Refers to words that frequently occur together in natural language. These are word combinations that appear more often than we would expect by chance.
- Strong tea ↔ Powerful tea
- Make a decision ↔ Do a decision
- Heavy rain ↔ Strong rain
Usefulness:
1. Natural language use: Native speakers use certain word combinations automatically.
2. Language learning: Helps learners sound more natural.
3. Translation: Different languages have different collocational patterns.
4. Meaning: Words take on specific meanings when paired together.
Collocation analysis: How do we know if words appear together by chance or as meaningful patterns?