### A Pluto.jl notebook ###
# v0.15.1

using Markdown
using InteractiveUtils

# ╔═╡ 449faf07-3214-47b6-bb02-824cf900bc07
begin
	#using Markdown
	using PlutoUI
	using Plots
end

# ╔═╡ 2dd2d4e3-14a0-43f1-b207-969b98ee5346
PlutoUI.TableOfContents(aside=true)

# ╔═╡ 89f1748e-593f-473f-907d-b3a64073e197
md"""
## Physics and math concepts

- Free fall
- Air friction
- Terminal speed
- Ordinary differential equations (ODEs)

"""

# ╔═╡ f878dcc5-06db-4dd4-8df8-c4666cf8f605
md"""
## Julia and numerical concepts

- Array comprehension
- Broadcasting
- Plotting, using `Plots` package
- Function arguments and keywords
- Euler and midpoint methods for ODEs
"""

# ╔═╡ f5e86f2a-076d-11ec-1080-c341c92d3fd1
md"""
# Free Fall
"""

# ╔═╡ 26f6893e-17ab-48e9-88d0-98a79ec0a1b0
md"""
## Assumption

Suppose we drop a ball from some height, and suppose we do it in a vacuum where air resistance is not an issue. Realistically, this would have to be a contrived experiment in a lab. It would not be case if we just went outside and dropped a ball off the roof.

And so let's be honest and say we will **neglect air resistance or friction** in our study. The statement **neglect _______** is used often, and can mean a couple of things:

1. The thing being neglected produces an effect so small you will not notice it, or

2. The effect may not be tiny and it may have a big effect on things, but it's difficult to add and we want to get an **approximate** picture of what's happening before we go ahead, do a bunch of work, and then add it.

The last option is the case here. It forms an important **assumption** in our work that follows.
"""

# ╔═╡ ecb886b8-cf77-4d82-865b-d2af63f22a2e
md"""
## Acceleration

**Free fall** refers to an object falling due to gravity, without air resistance. In this case, it **accelerates**, or gains a set amount of speed in a set amount of time, without bound in principle. Near the surface of Earth, it is about 9.8 m/s every second. This **gravitational acceleration** is denoted by the symbol ``g``. Hence,

```math
g = 9.8 \, \text{m}/\text{s}^2 .
```

We do not need to consider time increments of one second. They can be anything. If it is ``\Delta t``, then the speed at that increment later is ``g\, \Delta t``.

Expressed as an equation, if the speed of the ball at time ``t`` is ``v(t)``, then its speed at a later time ``t + \Delta t$ will be given by
```math
v(t+ \Delta t) = v(t) + g \, \Delta t .
```

Rounding ``g`` up to a convenient 10 m/s``^2`` and dropping the ball from rest,

|   time [s] | speed [m/s] |
|:----------:|:---------:|
|   0        |     0     |
|   0.5      |     5     |
|   1.0      |     10    |
|   2.0      |     20    |
|   4.0      |     40    |
|   ...      |  ...      |

"""

# ╔═╡ a3e1ba79-2f26-4ceb-b32d-457d9eef5ef9
md"""
Now, what about the **distance** fallen? The distance covered is speed times time. For a constant speed we have for any time increment
```math
z(t+ \Delta t) = z(t) + v \, \Delta t .
```
But now the speed is continually changing. How to deal with that?

A couple of options come to mind:
1. Take the speed to be constant and equal to that at the *start* of the time increment, or
2. Take the speed to be constant and equal to that in the *middle* of the time increment, or
3. what else?
"""

# ╔═╡ fe7c9f07-10c7-4aa3-b303-211da17ec695
md"""
## Computer experiments
"""

# ╔═╡ 6e215ec5-dafb-4eb0-a159-b9028d6de08d
md"""
Let's now apply some computational thinking to explore this physical situation.
"""

# ╔═╡ fcba2327-2306-4fc3-bb70-14e94e5d55c5
md"""
Finding the speed increment is not an issue. But the distance increment poses a problem, since the speed *changes* during the increment. We can think immediately of two approaches:
- use the average speed at the start of the increment
- use the initial speed
"""

# ╔═╡ 809e639a-8492-4b0a-bc98-2226c2404478
function Δv_grav(Δt, a=10.0)
	Δt*a
end

# ╔═╡ dd4a6136-75a7-460d-9bbc-97a374e23a4f
function Δz(v₀, Δt, Δv::Function)
	v_ave = ( v₀ + (v₀ + Δv(Δt)) )/2
	v_ave*Δt
end

# ╔═╡ c90cceb1-5f04-46e5-8879-da29516ea81f
function Δz(v₀, Δt)
	v_init = v₀
	v_init*Δt
end

# ╔═╡ 52671566-39e8-4849-b27d-669f6bdb49db
md"""
>Question: What magic is going on with the definition of Δz?
>
>Answer: This is an example of **multiple dispatch**, a centerpiece of Julia.
"""

# ╔═╡ 453ad11f-7bed-49f7-8255-6b4e7a86ae57
md"""
Our strategy will be **discretize** time. This means we will calculate quantities only at set time values. This is a common thing to do in computation, since we cannot find quantities at infinitely many points infinitely close together. And this is likely not useful even we could.

It's natural to start things at zero time. We can then find the **time grid** given the time interval ``\Delta t`` and the number of time points we want to consider.
"""

# ╔═╡ 0e324d7d-932a-41a5-b4e8-924286ed6878
begin
	Δt = 0.2
	t = [n*Δt for n in 0:10]
end

# ╔═╡ bf58c0d6-1c31-4fcc-b9e6-72058c79d79a
md"""
Speed and distance fallen will be determined at these time grid points, and they will thus fill arrays of the same size. Set the **speed and distance grids**. It doesn't matter what values we fill them with initially, since they'll be overwritten by those we calculate.
"""

# ╔═╡ cc9737b7-33ee-48c4-9cbd-dde3f9b28afc
begin
	v = similar(t)
	z = similar(t)
	v[1] = 0.0    # initial speed
	z[1] = 0.0    # initial location

	z_ave = similar(z)
	z_init = similar(z)
end

# ╔═╡ de13ad92-d15f-450a-9388-4e765c29ea66
md"""
Step through the time grid and get the speed and the distance fallen up to that time (using the two methods). We know what they are initially, so start at second grip point.
"""

# ╔═╡ e59e1646-d949-456d-8e28-a4f9e1f55a8a
for i in 2:length(v)
	v[i] = v[i-1] + Δv_grav(Δt)
	z_init[i] = z_init[i-1] + Δz(v[i-1], Δt)
	z_ave[i] = z_ave[i-1] + Δz(v[i-1], Δt, Δv_grav)
end

# ╔═╡ 421f19c1-38eb-4cff-b8d4-7a6ae4f438d0
md"""
"Sanity check" of the results... show the arrays.
"""

# ╔═╡ a87dd44d-89ca-41cc-b077-d444ea37c48d
v

# ╔═╡ becfff3a-b130-44e8-8f9b-f63d693fdc0d
z_init

# ╔═╡ 20e80196-d302-44ad-af58-c2c6211bcc95
z_ave

# ╔═╡ 202922e8-9d1a-4a23-afdc-f91deceb899e
md"""
>Question: Anything strange here?
>
>Answer:
"""

# ╔═╡ d98ad456-1f29-4047-8724-78a8cb35187a
md"""
Plot speed as a function of time.
"""

# ╔═╡ bafa445a-f3c8-4e83-bfa7-2b907bd33319
begin
	plot(t,v,label="v")
	plot!(title="speed vs time",xlabel="time [s]",ylabel="speed [m/s]")
end

# ╔═╡ 1d0dab51-947d-40f6-a38b-6bf54b1595cf
md"""
Now look at distance, and compare our two approaches. We also know the exact result

```math
z(t) = z_0 + v_0 t + \frac{1}{2} g t^2
```

and show that for comparison.
"""

# ╔═╡ 2f305768-1b2b-4939-926e-dee480da23f1
function z_exact(z₀, v₀, t, a=10.0)
		z₀ + v₀*t + a*t^2 / 2
end

# ╔═╡ 6afbfcf5-d0ed-484a-85df-9629f9d731f4
begin
	plot(t, z_init, label="init")
	plot!(t, z_ave, label="ave")
	plot!(t, z_exact.(z[1],v[1],t),label="exact")
	plot!(title="distance vs time",xlabel="time [s]",ylabel="distance [m]")
end

# ╔═╡ 2bfc8a93-4dc6-4c44-8df9-41866e52a681
md"""
>Question: What magic is happening in the plot command for the exact results?
>
>Answer: This is called **broadcasting**.
"""

# ╔═╡ 15bf0e04-9996-4c03-ae80-2e1df0151c85
z_exact.(z[1],v[1],t)

# ╔═╡ a9409040-77ea-442f-89c5-a5f7636b3e56
md"""
>Question: Which approach for calculating the distance is the best? Experiment with decreasing ``\Delta t`` (and increasing the number of grid points to keep the final time the same). What happens?
>
>Answer:
"""

# ╔═╡ 16b85f25-b033-47f0-a9a9-e92ae0ad97de
md"""
# Air friction
"""

# ╔═╡ 90e92459-1bfa-41b1-a0e1-e9499ef0d3b2
md"""
## A model
"""

# ╔═╡ b64d55a2-2272-4fb4-b71c-bc0e4ad0bfff
md"""
Air resistance or friction is complicated, and accurate approximate treatments require state-of-the-art computational fluid dynamics.

One model for the friction force that is generally decent is

```math
f = \frac{1}{2} C_d\, \rho\, A\, v^2 ,
```
where
- ``C_d`` is the **coefficient of dynamic friction**, and takes into account all the complicated effects of air moving around the object,
- ``\rho`` is the density of air,
- ``A`` is the cross-sectional area of the object,
- ``v`` is its speed, and
- ``m`` is its mass (used later).

The friction force is always directed *opposite* to the direction of motion.

It's effect is to decrease the object speed by
```math
\Delta v = -\frac{f}{m} \Delta t,
```
if ``\Delta t \rightarrow 0``.

Our time intervals are not **infinitesimally small**, so this becomes another approximation. And unfortunately, ``\Delta v`` depends on speed, so it is *not constant* over a time interval.
"""

# ╔═╡ c72fe0c7-c08a-4447-a727-612717c67e11
md"""
Learning from the distance study above, it looks reasonable to find the change in speed in an internal using the speed in the *middle* of the interval. So what we are doing is
1. **Stepping forward** a half interval, and then using that to
2. Step forward the whole interval.
"""

# ╔═╡ 930a4dff-f3ca-4817-9bd7-ef5c1d476431
md"""
## Computer experiments
"""

# ╔═╡ ea8c0bb9-c6bf-4be5-a582-f637de2bfa15
md"""
We need to consider a concrete example. Let's use a baseball.

Parameters:

- ``m = 0.145`` kg
- ``C_d = 0.346``
- ``\rho = 1.225`` kg/m``^3``
- ``A = 0.0043`` m``^2``
"""

# ╔═╡ 7a7b5329-582a-467b-a797-711e63e11966
function Δv_friction(v_i, Δt, a=10.0; mass=0.145, C_d=0.346, ρ=1.225, A=0.0043)
	f = C_d*ρ*A*v_i^2 / 2
	-(f/mass)*Δt
end

# ╔═╡ 4656fd48-7908-4c12-9117-4bf8a9e40a9a
function v_update!(v, Δt, Δv_grav::Function, Δv_friction::Function)
	for i in 2:length(v)
		v_halfstep = v[i-1] + ( Δv_grav(Δt/2) + Δv_friction(v[i-1], Δt/2) )
		v[i] = v[i-1] + ( Δv_grav(Δt) + Δv_friction(v_halfstep, Δt) )
	end
end

# ╔═╡ e60f5c16-f960-4c33-b966-4aa96b62ea84
begin
	v_update!(v, Δt, Δv_grav, Δv_friction0)
	v_friction = copy(v)
end

# ╔═╡ c356c512-565b-4da3-9d77-a12da63d349c
begin
	plot(t,v_friction,label="v")
	plot!(title="speed vs time",xlabel="time [s]",ylabel="speed [m/s]")
end

# ╔═╡ d8331098-d52c-4cd1-90d0-2756d6b6589e
md"""
Experiment with longer fall times, say out to 20 s. Notice that the speed behaves differently than with no friction. It approaches a limiting value, known as the **terminal speed**. This is different for every object.

Also, by updating according to the approach in `v_update!`, we have essentially solved a **differential equation** for the speed using a numerical technique called the **midpoint method** or **second-order Runge-Kutta**.

"""

# ╔═╡ 01d90dbf-b9ea-4daa-8141-022e974bdbed
md"""
>Question: Experiment with changing ``\Delta t`` (and increasing the number of grid points to keep the final time the same). What happens to the value of the terminal speed?
>
>Answer:
"""

# ╔═╡ 00000000-0000-0000-0000-000000000001
PLUTO_PROJECT_TOML_CONTENTS = """
[deps]
Plots = "91a5bcdd-55d7-5caf-9e0b-520d859cae80"
PlutoUI = "7f904dfe-b85e-4ff6-b463-dae2292396a8"

[compat]
Plots = "~1.21.2"
PlutoUI = "~0.7.9"
"""

# ╔═╡ 00000000-0000-0000-0000-000000000002
PLUTO_MANIFEST_TOML_CONTENTS = """
# This file is machine-generated - editing it directly is not advised

[[Adapt]]
deps = ["LinearAlgebra"]
git-tree-sha1 = "84918055d15b3114ede17ac6a7182f68870c16f7"
uuid = "79e6a3ab-5dfb-504d-930d-738a2a938a0e"
version = "3.3.1"

[[ArgTools]]
uuid = "0dad84c5-d112-42e6-8d28-ef12dabb789f"

[[Artifacts]]
uuid = "56f22d72-fd6d-98f1-02f0-08ddc0907c33"

[[Base64]]
uuid = "2a0f44e3-6c83-55bd-87e4-b1978d98bd5f"

[[Bzip2_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "c3598e525718abcc440f69cc6d5f60dda0a1b61e"
uuid = "6e34b625-4abd-537c-b88f-471c36dfa7a0"
version = "1.0.6+5"

[[Cairo_jll]]
deps = ["Artifacts", "Bzip2_jll", "Fontconfig_jll", "FreeType2_jll", "Glib_jll", "JLLWrappers", "LZO_jll", "Libdl", "Pixman_jll", "Pkg", "Xorg_libXext_jll", "Xorg_libXrender_jll", "Zlib_jll", "libpng_jll"]
git-tree-sha1 = "e2f47f6d8337369411569fd45ae5753ca10394c6"
uuid = "83423d85-b0ee-5818-9007-b63ccbeb887a"
version = "1.16.0+6"

[[ColorSchemes]]
deps = ["ColorTypes", "Colors", "FixedPointNumbers", "Random"]
git-tree-sha1 = "9995eb3977fbf67b86d0a0a0508e83017ded03f2"
uuid = "35d6a980-a343-548e-a6ea-1d62b119f2f4"
version = "3.14.0"

[[ColorTypes]]
deps = ["FixedPointNumbers", "Random"]
git-tree-sha1 = "024fe24d83e4a5bf5fc80501a314ce0d1aa35597"
uuid = "3da002f7-5984-5a60-b8a6-cbb66c0b333f"
version = "0.11.0"

[[Colors]]
deps = ["ColorTypes", "FixedPointNumbers", "Reexport"]
git-tree-sha1 = "417b0ed7b8b838aa6ca0a87aadf1bb9eb111ce40"
uuid = "5ae59095-9a9b-59fe-a467-6f913c188581"
version = "0.12.8"

[[Compat]]
deps = ["Base64", "Dates", "DelimitedFiles", "Distributed", "InteractiveUtils", "LibGit2", "Libdl", "LinearAlgebra", "Markdown", "Mmap", "Pkg", "Printf", "REPL", "Random", "SHA", "Serialization", "SharedArrays", "Sockets", "SparseArrays", "Statistics", "Test", "UUIDs", "Unicode"]
git-tree-sha1 = "727e463cfebd0c7b999bbf3e9e7e16f254b94193"
uuid = "34da2185-b29b-5c13-b0c7-acf172513d20"
version = "3.34.0"

[[CompilerSupportLibraries_jll]]
deps = ["Artifacts", "Libdl"]
uuid = "e66e0078-7015-5450-92f7-15fbd957f2ae"

[[Contour]]
deps = ["StaticArrays"]
git-tree-sha1 = "9f02045d934dc030edad45944ea80dbd1f0ebea7"
uuid = "d38c429a-6771-53c6-b99e-75d170b6e991"
version = "0.5.7"

[[DataAPI]]
git-tree-sha1 = "ee400abb2298bd13bfc3df1c412ed228061a2385"
uuid = "9a962f9c-6df0-11e9-0e5d-c546b8b5ee8a"
version = "1.7.0"

[[DataStructures]]
deps = ["Compat", "InteractiveUtils", "OrderedCollections"]
git-tree-sha1 = "7d9d316f04214f7efdbb6398d545446e246eff02"
uuid = "864edb3b-99cc-5e75-8d2d-829cb0a9cfe8"
version = "0.18.10"

[[DataValueInterfaces]]
git-tree-sha1 = "bfc1187b79289637fa0ef6d4436ebdfe6905cbd6"
uuid = "e2d170a0-9d28-54be-80f0-106bbe20a464"
version = "1.0.0"

[[Dates]]
deps = ["Printf"]
uuid = "ade2ca70-3891-5945-98fb-dc099432e06a"

[[DelimitedFiles]]
deps = ["Mmap"]
uuid = "8bb1440f-4735-579b-a4ab-409b98df4dab"

[[Distributed]]
deps = ["Random", "Serialization", "Sockets"]
uuid = "8ba89e20-285c-5b6f-9357-94700520ee1b"

[[Downloads]]
deps = ["ArgTools", "LibCURL", "NetworkOptions"]
uuid = "f43a241f-c20a-4ad4-852c-f6b1247861c6"

[[EarCut_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "92d8f9f208637e8d2d28c664051a00569c01493d"
uuid = "5ae413db-bbd1-5e63-b57d-d24a61df00f5"
version = "2.1.5+1"

[[Expat_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "b3bfd02e98aedfa5cf885665493c5598c350cd2f"
uuid = "2e619515-83b5-522b-bb60-26c02a35a201"
version = "2.2.10+0"

[[FFMPEG]]
deps = ["FFMPEG_jll"]
git-tree-sha1 = "b57e3acbe22f8484b4b5ff66a7499717fe1a9cc8"
uuid = "c87230d0-a227-11e9-1b43-d7ebe4e7570a"
version = "0.4.1"

[[FFMPEG_jll]]
deps = ["Artifacts", "Bzip2_jll", "FreeType2_jll", "FriBidi_jll", "JLLWrappers", "LAME_jll", "LibVPX_jll", "Libdl", "Ogg_jll", "OpenSSL_jll", "Opus_jll", "Pkg", "Zlib_jll", "libass_jll", "libfdk_aac_jll", "libvorbis_jll", "x264_jll", "x265_jll"]
git-tree-sha1 = "3cc57ad0a213808473eafef4845a74766242e05f"
uuid = "b22a6f82-2f65-5046-a5b2-351ab43fb4e5"
version = "4.3.1+4"

[[FixedPointNumbers]]
deps = ["Statistics"]
git-tree-sha1 = "335bfdceacc84c5cdf16aadc768aa5ddfc5383cc"
uuid = "53c48c17-4a7d-5ca2-90c5-79b7896eea93"
version = "0.8.4"

[[Fontconfig_jll]]
deps = ["Artifacts", "Bzip2_jll", "Expat_jll", "FreeType2_jll", "JLLWrappers", "Libdl", "Libuuid_jll", "Pkg", "Zlib_jll"]
git-tree-sha1 = "35895cf184ceaab11fd778b4590144034a167a2f"
uuid = "a3f928ae-7b40-5064-980b-68af3947d34b"
version = "2.13.1+14"

[[Formatting]]
deps = ["Printf"]
git-tree-sha1 = "8339d61043228fdd3eb658d86c926cb282ae72a8"
uuid = "59287772-0a20-5a39-b81b-1366585eb4c0"
version = "0.4.2"

[[FreeType2_jll]]
deps = ["Artifacts", "Bzip2_jll", "JLLWrappers", "Libdl", "Pkg", "Zlib_jll"]
git-tree-sha1 = "cbd58c9deb1d304f5a245a0b7eb841a2560cfec6"
uuid = "d7e528f0-a631-5988-bf34-fe36492bcfd7"
version = "2.10.1+5"

[[FriBidi_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "aa31987c2ba8704e23c6c8ba8a4f769d5d7e4f91"
uuid = "559328eb-81f9-559d-9380-de523a88c83c"
version = "1.0.10+0"

[[GLFW_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Libglvnd_jll", "Pkg", "Xorg_libXcursor_jll", "Xorg_libXi_jll", "Xorg_libXinerama_jll", "Xorg_libXrandr_jll"]
git-tree-sha1 = "dba1e8614e98949abfa60480b13653813d8f0157"
uuid = "0656b61e-2033-5cc2-a64a-77c0f6c09b89"
version = "3.3.5+0"

[[GR]]
deps = ["Base64", "DelimitedFiles", "GR_jll", "HTTP", "JSON", "Libdl", "LinearAlgebra", "Pkg", "Printf", "Random", "Serialization", "Sockets", "Test", "UUIDs"]
git-tree-sha1 = "182da592436e287758ded5be6e32c406de3a2e47"
uuid = "28b8d3ca-fb5f-59d9-8090-bfdbd6d07a71"
version = "0.58.1"

[[GR_jll]]
deps = ["Artifacts", "Bzip2_jll", "Cairo_jll", "FFMPEG_jll", "Fontconfig_jll", "GLFW_jll", "JLLWrappers", "JpegTurbo_jll", "Libdl", "Libtiff_jll", "Pixman_jll", "Pkg", "Qt5Base_jll", "Zlib_jll", "libpng_jll"]
git-tree-sha1 = "d59e8320c2747553788e4fc42231489cc602fa50"
uuid = "d2c73de3-f751-5644-a686-071e5b155ba9"
version = "0.58.1+0"

[[GeometryBasics]]
deps = ["EarCut_jll", "IterTools", "LinearAlgebra", "StaticArrays", "StructArrays", "Tables"]
git-tree-sha1 = "58bcdf5ebc057b085e58d95c138725628dd7453c"
uuid = "5c1252a2-5f33-56bf-86c9-59e7332b4326"
version = "0.4.1"

[[Gettext_jll]]
deps = ["Artifacts", "CompilerSupportLibraries_jll", "JLLWrappers", "Libdl", "Libiconv_jll", "Pkg", "XML2_jll"]
git-tree-sha1 = "9b02998aba7bf074d14de89f9d37ca24a1a0b046"
uuid = "78b55507-aeef-58d4-861c-77aaff3498b1"
version = "0.21.0+0"

[[Glib_jll]]
deps = ["Artifacts", "Gettext_jll", "JLLWrappers", "Libdl", "Libffi_jll", "Libiconv_jll", "Libmount_jll", "PCRE_jll", "Pkg", "Zlib_jll"]
git-tree-sha1 = "7bf67e9a481712b3dbe9cb3dac852dc4b1162e02"
uuid = "7746bdde-850d-59dc-9ae8-88ece973131d"
version = "2.68.3+0"

[[Grisu]]
git-tree-sha1 = "53bb909d1151e57e2484c3d1b53e19552b887fb2"
uuid = "42e2da0e-8278-4e71-bc24-59509adca0fe"
version = "1.0.2"

[[HTTP]]
deps = ["Base64", "Dates", "IniFile", "Logging", "MbedTLS", "NetworkOptions", "Sockets", "URIs"]
git-tree-sha1 = "44e3b40da000eab4ccb1aecdc4801c040026aeb5"
uuid = "cd3eb016-35fb-5094-929b-558a96fad6f3"
version = "0.9.13"

[[IniFile]]
deps = ["Test"]
git-tree-sha1 = "098e4d2c533924c921f9f9847274f2ad89e018b8"
uuid = "83e8ac13-25f8-5344-8a64-a9f2b223428f"
version = "0.5.0"

[[InteractiveUtils]]
deps = ["Markdown"]
uuid = "b77e0a4c-d291-57a0-90e8-8db25a27a240"

[[IterTools]]
git-tree-sha1 = "05110a2ab1fc5f932622ffea2a003221f4782c18"
uuid = "c8e1da08-722c-5040-9ed9-7db0dc04731e"
version = "1.3.0"

[[IteratorInterfaceExtensions]]
git-tree-sha1 = "a3f24677c21f5bbe9d2a714f95dcd58337fb2856"
uuid = "82899510-4779-5014-852e-03e436cf321d"
version = "1.0.0"

[[JLLWrappers]]
deps = ["Preferences"]
git-tree-sha1 = "642a199af8b68253517b80bd3bfd17eb4e84df6e"
uuid = "692b3bcd-3c85-4b1f-b108-f13ce0eb3210"
version = "1.3.0"

[[JSON]]
deps = ["Dates", "Mmap", "Parsers", "Unicode"]
git-tree-sha1 = "8076680b162ada2a031f707ac7b4953e30667a37"
uuid = "682c06a0-de6a-54ab-a142-c8b1cf79cde6"
version = "0.21.2"

[[JpegTurbo_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "d735490ac75c5cb9f1b00d8b5509c11984dc6943"
uuid = "aacddb02-875f-59d6-b918-886e6ef4fbf8"
version = "2.1.0+0"

[[LAME_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "f6250b16881adf048549549fba48b1161acdac8c"
uuid = "c1c5ebd0-6772-5130-a774-d5fcae4a789d"
version = "3.100.1+0"

[[LZO_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "e5b909bcf985c5e2605737d2ce278ed791b89be6"
uuid = "dd4b983a-f0e5-5f8d-a1b7-129d4a5fb1ac"
version = "2.10.1+0"

[[LaTeXStrings]]
git-tree-sha1 = "c7f1c695e06c01b95a67f0cd1d34994f3e7db104"
uuid = "b964fa9f-0449-5b57-a5c2-d3ea65f4040f"
version = "1.2.1"

[[Latexify]]
deps = ["Formatting", "InteractiveUtils", "LaTeXStrings", "MacroTools", "Markdown", "Printf", "Requires"]
git-tree-sha1 = "a4b12a1bd2ebade87891ab7e36fdbce582301a92"
uuid = "23fbe1c1-3f47-55db-b15f-69d7ec21a316"
version = "0.15.6"

[[LibCURL]]
deps = ["LibCURL_jll", "MozillaCACerts_jll"]
uuid = "b27032c2-a3e7-50c8-80cd-2d36dbcbfd21"

[[LibCURL_jll]]
deps = ["Artifacts", "LibSSH2_jll", "Libdl", "MbedTLS_jll", "Zlib_jll", "nghttp2_jll"]
uuid = "deac9b47-8bc7-5906-a0fe-35ac56dc84c0"

[[LibGit2]]
deps = ["Base64", "NetworkOptions", "Printf", "SHA"]
uuid = "76f85450-5226-5b5a-8eaa-529ad045b433"

[[LibSSH2_jll]]
deps = ["Artifacts", "Libdl", "MbedTLS_jll"]
uuid = "29816b5a-b9ab-546f-933c-edad1886dfa8"

[[LibVPX_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "12ee7e23fa4d18361e7c2cde8f8337d4c3101bc7"
uuid = "dd192d2f-8180-539f-9fb4-cc70b1dcf69a"
version = "1.10.0+0"

[[Libdl]]
uuid = "8f399da3-3557-5675-b5ff-fb832c97cbdb"

[[Libffi_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "761a393aeccd6aa92ec3515e428c26bf99575b3b"
uuid = "e9f186c6-92d2-5b65-8a66-fee21dc1b490"
version = "3.2.2+0"

[[Libgcrypt_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Libgpg_error_jll", "Pkg"]
git-tree-sha1 = "64613c82a59c120435c067c2b809fc61cf5166ae"
uuid = "d4300ac3-e22c-5743-9152-c294e39db1e4"
version = "1.8.7+0"

[[Libglvnd_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Xorg_libX11_jll", "Xorg_libXext_jll"]
git-tree-sha1 = "7739f837d6447403596a75d19ed01fd08d6f56bf"
uuid = "7e76a0d4-f3c7-5321-8279-8d96eeed0f29"
version = "1.3.0+3"

[[Libgpg_error_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "c333716e46366857753e273ce6a69ee0945a6db9"
uuid = "7add5ba3-2f88-524e-9cd5-f83b8a55f7b8"
version = "1.42.0+0"

[[Libiconv_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "42b62845d70a619f063a7da093d995ec8e15e778"
uuid = "94ce4f54-9a6c-5748-9c1c-f9c7231a4531"
version = "1.16.1+1"

[[Libmount_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "9c30530bf0effd46e15e0fdcf2b8636e78cbbd73"
uuid = "4b2f31a3-9ecc-558c-b454-b3730dcb73e9"
version = "2.35.0+0"

[[Libtiff_jll]]
deps = ["Artifacts", "JLLWrappers", "JpegTurbo_jll", "Libdl", "Pkg", "Zlib_jll", "Zstd_jll"]
git-tree-sha1 = "340e257aada13f95f98ee352d316c3bed37c8ab9"
uuid = "89763e89-9b03-5906-acba-b20f662cd828"
version = "4.3.0+0"

[[Libuuid_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "7f3efec06033682db852f8b3bc3c1d2b0a0ab066"
uuid = "38a345b3-de98-5d2b-a5d3-14cd9215e700"
version = "2.36.0+0"

[[LinearAlgebra]]
deps = ["Libdl"]
uuid = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e"

[[Logging]]
uuid = "56ddb016-857b-54e1-b83d-db4d58db5568"

[[MacroTools]]
deps = ["Markdown", "Random"]
git-tree-sha1 = "0fb723cd8c45858c22169b2e42269e53271a6df7"
uuid = "1914dd2f-81c6-5fcd-8719-6d5c9610ff09"
version = "0.5.7"

[[Markdown]]
deps = ["Base64"]
uuid = "d6f4376e-aef5-505a-96c1-9c027394607a"

[[MbedTLS]]
deps = ["Dates", "MbedTLS_jll", "Random", "Sockets"]
git-tree-sha1 = "1c38e51c3d08ef2278062ebceade0e46cefc96fe"
uuid = "739be429-bea8-5141-9913-cc70e7f3736d"
version = "1.0.3"

[[MbedTLS_jll]]
deps = ["Artifacts", "Libdl"]
uuid = "c8ffd9c3-330d-5841-b78e-0817d7145fa1"

[[Measures]]
git-tree-sha1 = "e498ddeee6f9fdb4551ce855a46f54dbd900245f"
uuid = "442fdcdd-2543-5da2-b0f3-8c86c306513e"
version = "0.3.1"

[[Missings]]
deps = ["DataAPI"]
git-tree-sha1 = "2ca267b08821e86c5ef4376cffed98a46c2cb205"
uuid = "e1d29d7a-bbdc-5cf2-9ac0-f12de2c33e28"
version = "1.0.1"

[[Mmap]]
uuid = "a63ad114-7e13-5084-954f-fe012c677804"

[[MozillaCACerts_jll]]
uuid = "14a3606d-f60d-562e-9121-12d972cd8159"

[[NaNMath]]
git-tree-sha1 = "bfe47e760d60b82b66b61d2d44128b62e3a369fb"
uuid = "77ba4419-2d1f-58cd-9bb1-8ffee604a2e3"
version = "0.3.5"

[[NetworkOptions]]
uuid = "ca575930-c2e3-43a9-ace4-1e988b2c1908"

[[Ogg_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "7937eda4681660b4d6aeeecc2f7e1c81c8ee4e2f"
uuid = "e7412a2a-1a6e-54c0-be00-318e2571c051"
version = "1.3.5+0"

[[OpenSSL_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "15003dcb7d8db3c6c857fda14891a539a8f2705a"
uuid = "458c3c95-2e84-50aa-8efc-19380b2a3a95"
version = "1.1.10+0"

[[Opus_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "51a08fb14ec28da2ec7a927c4337e4332c2a4720"
uuid = "91d4177d-7536-5919-b921-800302f37372"
version = "1.3.2+0"

[[OrderedCollections]]
git-tree-sha1 = "85f8e6578bf1f9ee0d11e7bb1b1456435479d47c"
uuid = "bac558e1-5e72-5ebc-8fee-abe8a469f55d"
version = "1.4.1"

[[PCRE_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "b2a7af664e098055a7529ad1a900ded962bca488"
uuid = "2f80f16e-611a-54ab-bc61-aa92de5b98fc"
version = "8.44.0+0"

[[Parsers]]
deps = ["Dates"]
git-tree-sha1 = "438d35d2d95ae2c5e8780b330592b6de8494e779"
uuid = "69de0a69-1ddd-5017-9359-2bf0b02dc9f0"
version = "2.0.3"

[[Pixman_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "b4f5d02549a10e20780a24fce72bea96b6329e29"
uuid = "30392449-352a-5448-841d-b1acce4e97dc"
version = "0.40.1+0"

[[Pkg]]
deps = ["Artifacts", "Dates", "Downloads", "LibGit2", "Libdl", "Logging", "Markdown", "Printf", "REPL", "Random", "SHA", "Serialization", "TOML", "Tar", "UUIDs", "p7zip_jll"]
uuid = "44cfe95a-1eb2-52ea-b672-e2afdf69b78f"

[[PlotThemes]]
deps = ["PlotUtils", "Requires", "Statistics"]
git-tree-sha1 = "a3a964ce9dc7898193536002a6dd892b1b5a6f1d"
uuid = "ccf2f8ad-2431-5c83-bf29-c5338b663b6a"
version = "2.0.1"

[[PlotUtils]]
deps = ["ColorSchemes", "Colors", "Dates", "Printf", "Random", "Reexport", "Statistics"]
git-tree-sha1 = "9ff1c70190c1c30aebca35dc489f7411b256cd23"
uuid = "995b91a9-d308-5afd-9ec6-746e21dbc043"
version = "1.0.13"

[[Plots]]
deps = ["Base64", "Contour", "Dates", "Downloads", "FFMPEG", "FixedPointNumbers", "GR", "GeometryBasics", "JSON", "Latexify", "LinearAlgebra", "Measures", "NaNMath", "PlotThemes", "PlotUtils", "Printf", "REPL", "Random", "RecipesBase", "RecipesPipeline", "Reexport", "Requires", "Scratch", "Showoff", "SparseArrays", "Statistics", "StatsBase", "UUIDs"]
git-tree-sha1 = "9e1a400fb1f27b4146fe35dc1a22de6c793b8f20"
uuid = "91a5bcdd-55d7-5caf-9e0b-520d859cae80"
version = "1.21.2"

[[PlutoUI]]
deps = ["Base64", "Dates", "InteractiveUtils", "JSON", "Logging", "Markdown", "Random", "Reexport", "Suppressor"]
git-tree-sha1 = "44e225d5837e2a2345e69a1d1e01ac2443ff9fcb"
uuid = "7f904dfe-b85e-4ff6-b463-dae2292396a8"
version = "0.7.9"

[[Preferences]]
deps = ["TOML"]
git-tree-sha1 = "00cfd92944ca9c760982747e9a1d0d5d86ab1e5a"
uuid = "21216c6a-2e73-6563-6e65-726566657250"
version = "1.2.2"

[[Printf]]
deps = ["Unicode"]
uuid = "de0858da-6303-5e67-8744-51eddeeeb8d7"

[[Qt5Base_jll]]
deps = ["Artifacts", "CompilerSupportLibraries_jll", "Fontconfig_jll", "Glib_jll", "JLLWrappers", "Libdl", "Libglvnd_jll", "OpenSSL_jll", "Pkg", "Xorg_libXext_jll", "Xorg_libxcb_jll", "Xorg_xcb_util_image_jll", "Xorg_xcb_util_keysyms_jll", "Xorg_xcb_util_renderutil_jll", "Xorg_xcb_util_wm_jll", "Zlib_jll", "xkbcommon_jll"]
git-tree-sha1 = "ad368663a5e20dbb8d6dc2fddeefe4dae0781ae8"
uuid = "ea2cea3b-5b76-57ae-a6ef-0a8af62496e1"
version = "5.15.3+0"

[[REPL]]
deps = ["InteractiveUtils", "Markdown", "Sockets", "Unicode"]
uuid = "3fa0cd96-eef1-5676-8a61-b3b8758bbffb"

[[Random]]
deps = ["Serialization"]
uuid = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"

[[RecipesBase]]
git-tree-sha1 = "44a75aa7a527910ee3d1751d1f0e4148698add9e"
uuid = "3cdcf5f2-1ef4-517c-9805-6587b60abb01"
version = "1.1.2"

[[RecipesPipeline]]
deps = ["Dates", "NaNMath", "PlotUtils", "RecipesBase"]
git-tree-sha1 = "d4491becdc53580c6dadb0f6249f90caae888554"
uuid = "01d81517-befc-4cb6-b9ec-a95719d0359c"
version = "0.4.0"

[[Reexport]]
git-tree-sha1 = "45e428421666073eab6f2da5c9d310d99bb12f9b"
uuid = "189a3867-3050-52da-a836-e630ba90ab69"
version = "1.2.2"

[[Requires]]
deps = ["UUIDs"]
git-tree-sha1 = "4036a3bd08ac7e968e27c203d45f5fff15020621"
uuid = "ae029012-a4dd-5104-9daa-d747884805df"
version = "1.1.3"

[[SHA]]
uuid = "ea8e919c-243c-51af-8825-aaa63cd721ce"

[[Scratch]]
deps = ["Dates"]
git-tree-sha1 = "0b4b7f1393cff97c33891da2a0bf69c6ed241fda"
uuid = "6c6a2e73-6563-6170-7368-637461726353"
version = "1.1.0"

[[Serialization]]
uuid = "9e88b42a-f829-5b0c-bbe9-9e923198166b"

[[SharedArrays]]
deps = ["Distributed", "Mmap", "Random", "Serialization"]
uuid = "1a1011a3-84de-559e-8e89-a11a2f7dc383"

[[Showoff]]
deps = ["Dates", "Grisu"]
git-tree-sha1 = "91eddf657aca81df9ae6ceb20b959ae5653ad1de"
uuid = "992d4aef-0814-514b-bc4d-f2e9a6c4116f"
version = "1.0.3"

[[Sockets]]
uuid = "6462fe0b-24de-5631-8697-dd941f90decc"

[[SortingAlgorithms]]
deps = ["DataStructures"]
git-tree-sha1 = "b3363d7460f7d098ca0912c69b082f75625d7508"
uuid = "a2af1166-a08f-5f64-846c-94a0d3cef48c"
version = "1.0.1"

[[SparseArrays]]
deps = ["LinearAlgebra", "Random"]
uuid = "2f01184e-e22b-5df5-ae63-d93ebab69eaf"

[[StaticArrays]]
deps = ["LinearAlgebra", "Random", "Statistics"]
git-tree-sha1 = "3240808c6d463ac46f1c1cd7638375cd22abbccb"
uuid = "90137ffa-7385-5640-81b9-e52037218182"
version = "1.2.12"

[[Statistics]]
deps = ["LinearAlgebra", "SparseArrays"]
uuid = "10745b16-79ce-11e8-11f9-7d13ad32a3b2"

[[StatsAPI]]
git-tree-sha1 = "1958272568dc176a1d881acb797beb909c785510"
uuid = "82ae8749-77ed-4fe6-ae5f-f523153014b0"
version = "1.0.0"

[[StatsBase]]
deps = ["DataAPI", "DataStructures", "LinearAlgebra", "Missings", "Printf", "Random", "SortingAlgorithms", "SparseArrays", "Statistics", "StatsAPI"]
git-tree-sha1 = "8cbbc098554648c84f79a463c9ff0fd277144b6c"
uuid = "2913bbd2-ae8a-5f71-8c99-4fb6c76f3a91"
version = "0.33.10"

[[StructArrays]]
deps = ["Adapt", "DataAPI", "StaticArrays", "Tables"]
git-tree-sha1 = "1700b86ad59348c0f9f68ddc95117071f947072d"
uuid = "09ab397b-f2b6-538f-b94a-2f83cf4a842a"
version = "0.6.1"

[[Suppressor]]
git-tree-sha1 = "a819d77f31f83e5792a76081eee1ea6342ab8787"
uuid = "fd094767-a336-5f1f-9728-57cf17d0bbfb"
version = "0.2.0"

[[TOML]]
deps = ["Dates"]
uuid = "fa267f1f-6049-4f14-aa54-33bafae1ed76"

[[TableTraits]]
deps = ["IteratorInterfaceExtensions"]
git-tree-sha1 = "c06b2f539df1c6efa794486abfb6ed2022561a39"
uuid = "3783bdb8-4a98-5b6b-af9a-565f29a5fe9c"
version = "1.0.1"

[[Tables]]
deps = ["DataAPI", "DataValueInterfaces", "IteratorInterfaceExtensions", "LinearAlgebra", "TableTraits", "Test"]
git-tree-sha1 = "d0c690d37c73aeb5ca063056283fde5585a41710"
uuid = "bd369af6-aec1-5ad0-b16a-f7cc5008161c"
version = "1.5.0"

[[Tar]]
deps = ["ArgTools", "SHA"]
uuid = "a4e569a6-e804-4fa4-b0f3-eef7a1d5b13e"

[[Test]]
deps = ["InteractiveUtils", "Logging", "Random", "Serialization"]
uuid = "8dfed614-e22c-5e08-85e1-65c5234f0b40"

[[URIs]]
git-tree-sha1 = "97bbe755a53fe859669cd907f2d96aee8d2c1355"
uuid = "5c2747f8-b7ea-4ff2-ba2e-563bfd36b1d4"
version = "1.3.0"

[[UUIDs]]
deps = ["Random", "SHA"]
uuid = "cf7118a7-6976-5b1a-9a39-7adc72f591a4"

[[Unicode]]
uuid = "4ec0a83e-493e-50e2-b9ac-8f72acf5a8f5"

[[Wayland_jll]]
deps = ["Artifacts", "Expat_jll", "JLLWrappers", "Libdl", "Libffi_jll", "Pkg", "XML2_jll"]
git-tree-sha1 = "3e61f0b86f90dacb0bc0e73a0c5a83f6a8636e23"
uuid = "a2964d1f-97da-50d4-b82a-358c7fce9d89"
version = "1.19.0+0"

[[Wayland_protocols_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Wayland_jll"]
git-tree-sha1 = "2839f1c1296940218e35df0bbb220f2a79686670"
uuid = "2381bf8a-dfd0-557d-9999-79630e7b1b91"
version = "1.18.0+4"

[[XML2_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Libiconv_jll", "Pkg", "Zlib_jll"]
git-tree-sha1 = "1acf5bdf07aa0907e0a37d3718bb88d4b687b74a"
uuid = "02c8fc9c-b97f-50b9-bbe4-9be30ff0a78a"
version = "2.9.12+0"

[[XSLT_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Libgcrypt_jll", "Libgpg_error_jll", "Libiconv_jll", "Pkg", "XML2_jll", "Zlib_jll"]
git-tree-sha1 = "91844873c4085240b95e795f692c4cec4d805f8a"
uuid = "aed1982a-8fda-507f-9586-7b0439959a61"
version = "1.1.34+0"

[[Xorg_libX11_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Xorg_libxcb_jll", "Xorg_xtrans_jll"]
git-tree-sha1 = "5be649d550f3f4b95308bf0183b82e2582876527"
uuid = "4f6342f7-b3d2-589e-9d20-edeb45f2b2bc"
version = "1.6.9+4"

[[Xorg_libXau_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "4e490d5c960c314f33885790ed410ff3a94ce67e"
uuid = "0c0b7dd1-d40b-584c-a123-a41640f87eec"
version = "1.0.9+4"

[[Xorg_libXcursor_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Xorg_libXfixes_jll", "Xorg_libXrender_jll"]
git-tree-sha1 = "12e0eb3bc634fa2080c1c37fccf56f7c22989afd"
uuid = "935fb764-8cf2-53bf-bb30-45bb1f8bf724"
version = "1.2.0+4"

[[Xorg_libXdmcp_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "4fe47bd2247248125c428978740e18a681372dd4"
uuid = "a3789734-cfe1-5b06-b2d0-1dd0d9d62d05"
version = "1.1.3+4"

[[Xorg_libXext_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Xorg_libX11_jll"]
git-tree-sha1 = "b7c0aa8c376b31e4852b360222848637f481f8c3"
uuid = "1082639a-0dae-5f34-9b06-72781eeb8cb3"
version = "1.3.4+4"

[[Xorg_libXfixes_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Xorg_libX11_jll"]
git-tree-sha1 = "0e0dc7431e7a0587559f9294aeec269471c991a4"
uuid = "d091e8ba-531a-589c-9de9-94069b037ed8"
version = "5.0.3+4"

[[Xorg_libXi_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Xorg_libXext_jll", "Xorg_libXfixes_jll"]
git-tree-sha1 = "89b52bc2160aadc84d707093930ef0bffa641246"
uuid = "a51aa0fd-4e3c-5386-b890-e753decda492"
version = "1.7.10+4"

[[Xorg_libXinerama_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Xorg_libXext_jll"]
git-tree-sha1 = "26be8b1c342929259317d8b9f7b53bf2bb73b123"
uuid = "d1454406-59df-5ea1-beac-c340f2130bc3"
version = "1.1.4+4"

[[Xorg_libXrandr_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Xorg_libXext_jll", "Xorg_libXrender_jll"]
git-tree-sha1 = "34cea83cb726fb58f325887bf0612c6b3fb17631"
uuid = "ec84b674-ba8e-5d96-8ba1-2a689ba10484"
version = "1.5.2+4"

[[Xorg_libXrender_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Xorg_libX11_jll"]
git-tree-sha1 = "19560f30fd49f4d4efbe7002a1037f8c43d43b96"
uuid = "ea2f1a96-1ddc-540d-b46f-429655e07cfa"
version = "0.9.10+4"

[[Xorg_libpthread_stubs_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "6783737e45d3c59a4a4c4091f5f88cdcf0908cbb"
uuid = "14d82f49-176c-5ed1-bb49-ad3f5cbd8c74"
version = "0.1.0+3"

[[Xorg_libxcb_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "XSLT_jll", "Xorg_libXau_jll", "Xorg_libXdmcp_jll", "Xorg_libpthread_stubs_jll"]
git-tree-sha1 = "daf17f441228e7a3833846cd048892861cff16d6"
uuid = "c7cfdc94-dc32-55de-ac96-5a1b8d977c5b"
version = "1.13.0+3"

[[Xorg_libxkbfile_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Xorg_libX11_jll"]
git-tree-sha1 = "926af861744212db0eb001d9e40b5d16292080b2"
uuid = "cc61e674-0454-545c-8b26-ed2c68acab7a"
version = "1.1.0+4"

[[Xorg_xcb_util_image_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Xorg_xcb_util_jll"]
git-tree-sha1 = "0fab0a40349ba1cba2c1da699243396ff8e94b97"
uuid = "12413925-8142-5f55-bb0e-6d7ca50bb09b"
version = "0.4.0+1"

[[Xorg_xcb_util_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Xorg_libxcb_jll"]
git-tree-sha1 = "e7fd7b2881fa2eaa72717420894d3938177862d1"
uuid = "2def613f-5ad1-5310-b15b-b15d46f528f5"
version = "0.4.0+1"

[[Xorg_xcb_util_keysyms_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Xorg_xcb_util_jll"]
git-tree-sha1 = "d1151e2c45a544f32441a567d1690e701ec89b00"
uuid = "975044d2-76e6-5fbe-bf08-97ce7c6574c7"
version = "0.4.0+1"

[[Xorg_xcb_util_renderutil_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Xorg_xcb_util_jll"]
git-tree-sha1 = "dfd7a8f38d4613b6a575253b3174dd991ca6183e"
uuid = "0d47668e-0667-5a69-a72c-f761630bfb7e"
version = "0.3.9+1"

[[Xorg_xcb_util_wm_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Xorg_xcb_util_jll"]
git-tree-sha1 = "e78d10aab01a4a154142c5006ed44fd9e8e31b67"
uuid = "c22f9ab0-d5fe-5066-847c-f4bb1cd4e361"
version = "0.4.1+1"

[[Xorg_xkbcomp_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Xorg_libxkbfile_jll"]
git-tree-sha1 = "4bcbf660f6c2e714f87e960a171b119d06ee163b"
uuid = "35661453-b289-5fab-8a00-3d9160c6a3a4"
version = "1.4.2+4"

[[Xorg_xkeyboard_config_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Xorg_xkbcomp_jll"]
git-tree-sha1 = "5c8424f8a67c3f2209646d4425f3d415fee5931d"
uuid = "33bec58e-1273-512f-9401-5d533626f822"
version = "2.27.0+4"

[[Xorg_xtrans_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "79c31e7844f6ecf779705fbc12146eb190b7d845"
uuid = "c5fb5394-a638-5e4d-96e5-b29de1b5cf10"
version = "1.4.0+3"

[[Zlib_jll]]
deps = ["Libdl"]
uuid = "83775a58-1f1d-513f-b197-d71354ab007a"

[[Zstd_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "cc4bf3fdde8b7e3e9fa0351bdeedba1cf3b7f6e6"
uuid = "3161d3a3-bdf6-5164-811a-617609db77b4"
version = "1.5.0+0"

[[libass_jll]]
deps = ["Artifacts", "Bzip2_jll", "FreeType2_jll", "FriBidi_jll", "JLLWrappers", "Libdl", "Pkg", "Zlib_jll"]
git-tree-sha1 = "acc685bcf777b2202a904cdcb49ad34c2fa1880c"
uuid = "0ac62f75-1d6f-5e53-bd7c-93b484bb37c0"
version = "0.14.0+4"

[[libfdk_aac_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "7a5780a0d9c6864184b3a2eeeb833a0c871f00ab"
uuid = "f638f0a6-7fb0-5443-88ba-1cc74229b280"
version = "0.1.6+4"

[[libpng_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Zlib_jll"]
git-tree-sha1 = "94d180a6d2b5e55e447e2d27a29ed04fe79eb30c"
uuid = "b53b4c65-9356-5827-b1ea-8c7a1a84506f"
version = "1.6.38+0"

[[libvorbis_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Ogg_jll", "Pkg"]
git-tree-sha1 = "c45f4e40e7aafe9d086379e5578947ec8b95a8fb"
uuid = "f27f6e37-5d2b-51aa-960f-b287f2bc3b7a"
version = "1.3.7+0"

[[nghttp2_jll]]
deps = ["Artifacts", "Libdl"]
uuid = "8e850ede-7688-5339-a07c-302acd2aaf8d"

[[p7zip_jll]]
deps = ["Artifacts", "Libdl"]
uuid = "3f19e933-33d8-53b3-aaab-bd5110c3b7a0"

[[x264_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "d713c1ce4deac133e3334ee12f4adff07f81778f"
uuid = "1270edf5-f2f9-52d2-97e9-ab00b5d0237a"
version = "2020.7.14+2"

[[x265_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
git-tree-sha1 = "487da2f8f2f0c8ee0e83f39d13037d6bbf0a45ab"
uuid = "dfaa095f-4041-5dcd-9319-2fabd8486b76"
version = "3.0.0+3"

[[xkbcommon_jll]]
deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg", "Wayland_jll", "Wayland_protocols_jll", "Xorg_libxcb_jll", "Xorg_xkeyboard_config_jll"]
git-tree-sha1 = "ece2350174195bb31de1a63bea3a41ae1aa593b6"
uuid = "d8fb68d0-12a3-5cfd-a85a-d49703b185fd"
version = "0.9.1+5"
"""

# ╔═╡ Cell order:
# ╠═449faf07-3214-47b6-bb02-824cf900bc07
# ╠═2dd2d4e3-14a0-43f1-b207-969b98ee5346
# ╟─89f1748e-593f-473f-907d-b3a64073e197
# ╟─f878dcc5-06db-4dd4-8df8-c4666cf8f605
# ╟─f5e86f2a-076d-11ec-1080-c341c92d3fd1
# ╟─26f6893e-17ab-48e9-88d0-98a79ec0a1b0
# ╟─ecb886b8-cf77-4d82-865b-d2af63f22a2e
# ╟─a3e1ba79-2f26-4ceb-b32d-457d9eef5ef9
# ╟─fe7c9f07-10c7-4aa3-b303-211da17ec695
# ╟─6e215ec5-dafb-4eb0-a159-b9028d6de08d
# ╟─fcba2327-2306-4fc3-bb70-14e94e5d55c5
# ╠═809e639a-8492-4b0a-bc98-2226c2404478
# ╠═dd4a6136-75a7-460d-9bbc-97a374e23a4f
# ╠═c90cceb1-5f04-46e5-8879-da29516ea81f
# ╟─52671566-39e8-4849-b27d-669f6bdb49db
# ╟─453ad11f-7bed-49f7-8255-6b4e7a86ae57
# ╠═0e324d7d-932a-41a5-b4e8-924286ed6878
# ╟─bf58c0d6-1c31-4fcc-b9e6-72058c79d79a
# ╠═cc9737b7-33ee-48c4-9cbd-dde3f9b28afc
# ╟─de13ad92-d15f-450a-9388-4e765c29ea66
# ╠═e59e1646-d949-456d-8e28-a4f9e1f55a8a
# ╟─421f19c1-38eb-4cff-b8d4-7a6ae4f438d0
# ╠═a87dd44d-89ca-41cc-b077-d444ea37c48d
# ╠═becfff3a-b130-44e8-8f9b-f63d693fdc0d
# ╠═20e80196-d302-44ad-af58-c2c6211bcc95
# ╟─202922e8-9d1a-4a23-afdc-f91deceb899e
# ╟─d98ad456-1f29-4047-8724-78a8cb35187a
# ╠═bafa445a-f3c8-4e83-bfa7-2b907bd33319
# ╟─1d0dab51-947d-40f6-a38b-6bf54b1595cf
# ╠═2f305768-1b2b-4939-926e-dee480da23f1
# ╠═6afbfcf5-d0ed-484a-85df-9629f9d731f4
# ╟─2bfc8a93-4dc6-4c44-8df9-41866e52a681
# ╠═15bf0e04-9996-4c03-ae80-2e1df0151c85
# ╟─a9409040-77ea-442f-89c5-a5f7636b3e56
# ╟─16b85f25-b033-47f0-a9a9-e92ae0ad97de
# ╟─90e92459-1bfa-41b1-a0e1-e9499ef0d3b2
# ╟─b64d55a2-2272-4fb4-b71c-bc0e4ad0bfff
# ╟─c72fe0c7-c08a-4447-a727-612717c67e11
# ╟─930a4dff-f3ca-4817-9bd7-ef5c1d476431
# ╟─ea8c0bb9-c6bf-4be5-a582-f637de2bfa15
# ╠═7a7b5329-582a-467b-a797-711e63e11966
# ╠═4656fd48-7908-4c12-9117-4bf8a9e40a9a
# ╠═e60f5c16-f960-4c33-b966-4aa96b62ea84
# ╠═c356c512-565b-4da3-9d77-a12da63d349c
# ╟─d8331098-d52c-4cd1-90d0-2756d6b6589e
# ╟─01d90dbf-b9ea-4daa-8141-022e974bdbed
# ╟─00000000-0000-0000-0000-000000000001
# ╟─00000000-0000-0000-0000-000000000002


# # Multinomial logistic regression

import MaximumLikelihoodProblems

import Distributions
import ForwardDiff
import LogDensityProblems
import NNlib
import TransformVariables

struct MultinomialLogisticRegression{Ty, TX}
    y::Ty
    X::TX
end

function (problem::MultinomialLogisticRegression)(θ)
    y = problem.y
    X = problem.X

    β = θ.β

    num_rows = size(X, 1)
    num_covariates = size(β, 1) # `size(β, 1)` is equal to `size(X, 2)`
    num_classes = size(β, 2) + 1

    ## the first column of all zeros corresponds to the base class
    ## i.e. the coefficient β₀ for the base class is always fixed to be zero
    β_with_base_class = hcat(zeros(num_covariates), β)
    η = X * β_with_base_class

    μ = NNlib.softmax(η; dims=2)
    log_likelihood = sum([Distributions.logpdf(Distributions.Multinomial(1, μ[i, :]), y[i, :]) for i = 1:num_rows])
    return log_likelihood
end

N = 10_000

## the first column (the column of all ones) is the intercept
X = hcat(ones(N), randn(N))

size_β = (2, 3)
β_true = [1.0 2.0 3.0; 4.0 5.0 6.0]
num_covariates = size(β_true, 1)
β_true_with_base_class = hcat(zeros(num_covariates), β_true)
η_true = X * β_true_with_base_class
μ_true = NNlib.softmax(η_true; dims=2)
y = vcat([rand(Distributions.Multinomial(1, μ_true[i,:]))' for i in 1:N]...)

problem = MultinomialLogisticRegression(y, X)
transformation = TransformVariables.as((β = TransformVariables.as(Array, size_β), ))
transformed_problem = LogDensityProblems.TransformedLogDensity(transformation,
                                                               problem)
transformed_gradient_problem = LogDensityProblems.ADgradient(:ForwardDiff,
                                                             transformed_problem)

β_hat_initial_guess = zeros(size_β)
θ_hat_initial = (; β = β_hat_initial_guess)

# θ_hat:
# (I recommend changing the value of `show_progress_meter` to `true` so that
# you can see the progress.)

θ_hat = MaximumLikelihoodProblems.fit(transformed_gradient_problem,
                                      θ_hat_initial;
                                      show_progress_meter = false) # change this `false` to `true`

# β_hat:

β_hat = θ_hat[:β]

# β\_hat\_with\_base\_class:

num_covariates = size(β_hat, 1)
β_hat_with_base_class = hcat(zeros(num_covariates), β_hat)

# Value of the log likelihood function evaluated at θ_hat:

MaximumLikelihoodProblems.loglikelihood(transformed_gradient_problem, θ_hat)


### A Pluto.jl notebook ###
# v0.12.20

using Markdown
using InteractiveUtils

# ╔═╡ 5bd9e6aa-6820-11eb-2b20-3f17c0045841
begin
	using Test
	using BenchmarkTools
	using PlutoUI
end

# ╔═╡ 19046b20-6820-11eb-3fac-e993b8847dc0
md"""
### Sum of Intervals

Write a function called `sum_intervals()` that accepts an array of intervals, and returns the sum of all the interval lengths. Overlapping intervals should only be counted once.

**Intervals**:

Intervals are represented by a pair of integers in the form of a tuple. The first value of the interval will always be less than the second value. Interval example: `(1, 5)` is an interval from 1 to 5. The length of this interval is 4.

**Overlapping Intervals**:

List containing overlapping intervals:

```julia
[
  (1, 4),
  (7, 10),
  (3, 5)
]
```

The sum of the lengths of these intervals is 7. Since `(1, 4)` and `(3, 5)` overlap, we can treat the interval as `(1, 5)`, which has a length of 4.

**Examples**:

```julia
sum_intervals([
   (1, 2),
   (6, 10),
   (11, 15)
]); # => 9

sum_intervals([
   (1, 4),
   (7, 10),
   (3, 5)
]); # => 7

sum_intervals([
   (1, 5),
   (10, 20),
   (1, 6),
   (16, 19),
   (5, 11)
]); # => 19
```
"""

# ╔═╡ 698441e4-6820-11eb-2ed2-e90fbf53d7f0
md"""
#### Defining a parametric type
"""

# ╔═╡ 7da89592-6820-11eb-0ad7-699807d1d952
struct Interval{T <: Integer}
    lb::T  # lower bound
    ub::T  # upper bound

    function Interval{T}(lb, ub) where {T <: Integer}
		(lb, ub) = lb > ub ? (ub, lb) : (lb, ub)
		@assert 0 ≤ lb ≤ ub
		new(lb, ub)
	end
end

# ╔═╡ 9e07bbc4-6820-11eb-07ca-77f29e8a3030
IntervInt64 = Interval{Int64}

# ╔═╡ b9125d20-6820-11eb-3773-07dd4728988d
interv0 = IntervInt64(3, 1)

# ╔═╡ f6264b68-6820-11eb-1fc4-afcb22381c09
md"""
##### Length of an Interval
"""

# ╔═╡ 1640dbfe-6821-11eb-30bb-37b1f76609a3
function interv_len(interv::Interval{T})::T where T <: Integer
    interv.ub - interv.lb
end

# ╔═╡ 3cdd4d22-6821-11eb-3c20-a7997b70c44d
begin
	interv1 = IntervInt64(3, 1)
	@test interv_len(interv1) == 2

	interv2 = IntervInt64(4, 7)
	@test interv_len(interv2) == 3

	interv3 = IntervInt64(1, 1)
	@test interv_len(interv3) == 0

	interv4 = IntervInt64(5, 1)
	@test interv_len(interv4) == 4
end

# ╔═╡ 46c62dea-6821-11eb-0b18-771fd704ee84
md"""
##### Compute if two intervals overlap
"""

# ╔═╡ 91d63a66-6821-11eb-07a4-97c4a000b482
begin

function interv_inter(i₁::Interval{T}, i₂::Interval{T})::Bool where T <: Integer
	i₁ ⊆ i₂ || i₂ ⊆ i₁ || i₁ ⊆ i₂ || i₁ ∩ i₂ || i₂ ∩ i₁
end

function ⊆(i₁::Interval{T}, i₂::Interval{T})::Bool where T <: Integer
	lb1, ub1 = i₁.lb, i₁.ub
  	lb2, ub2 = i₂.lb, i₂.ub

	lb2 ≤ lb1 ≤ ub1 ≤ ub2
end

# \cap ≡ ∩
function ∩(i₁::Interval{T}, i₂::Interval{T})::Bool where T <: Integer
	lb1, ub1 = i₁.lb, i₁.ub
  	lb2, ub2 = i₂.lb, i₂.ub

	lb1 ≤ lb2 ≤ ub1 ≤ ub2
end

end

# ╔═╡ 14f1261a-6823-11eb-3d78-ebc8784d5ef8
begin
	@test interv_inter(IntervInt64(1, 10), IntervInt64(3, 7))
	@test interv_inter(IntervInt64(3, 7), IntervInt64(1, 8))
	@test interv_inter(IntervInt64(1, 6), IntervInt64(3, 9))
	@test interv_inter(IntervInt64(1, 5), IntervInt64(3, 4))
	@test interv_inter(IntervInt64(1, 5), IntervInt64(5, 6))
	@test interv_inter(IntervInt64(6, 10), IntervInt64(5, 8))
	@test !interv_inter(IntervInt64(1, 5), IntervInt64(6, 8))
	@test !interv_inter(IntervInt64(1, 5), IntervInt64(10, 20))
end

# ╔═╡ aac494f8-6823-11eb-138e-59b1641e416b
md"""
##### Adding two intervals length
"""

# ╔═╡ c2440f3a-6823-11eb-1fac-07057f45e3e3
function Base.:+(i₁::Interval{T}, i₂::Interval{T})::T where T <: Integer
	lb1, ub1 = i₁.lb, i₁.ub
	lb2, ub2 = i₂.lb, i₂.ub

	i₁ ⊆ i₂ && return ub2 - lb2
	i₂ ⊆ i₁ && return ub1 - lb1

   	s_inter = interv_len(i₁) + interv_len(i₂)

	# There might be some overlap: i₁.lb ---- i₂.lb ----- i₁.ub ---- i₂.ub
   	i₁ ∩ i₂ && return s_inter - (ub1 - lb2)
   	i₂ ∩ i₁ && return s_inter - (ub2 - lb1) # symmetry

   	s_inter
end

# ╔═╡ a68980ee-6824-11eb-06e2-b5a9527b24b5
begin
	@test IntervInt64(1, 10) + IntervInt64(3, 7) == 9
	@test IntervInt64(3, 7) + IntervInt64(1, 8) == 7
	@test IntervInt64(1, 6) + IntervInt64(3, 9) == 8
	@test IntervInt64(6, 10) + IntervInt64(5, 8) == 5
	@test IntervInt64(1, 5) + IntervInt64(6, 8) == 6
end

# ╔═╡ dda42a42-6825-11eb-1896-c7c0c2f6577d
md"""
##### Adding interval (length) and a length
"""

# ╔═╡ eb0fe112-6825-11eb-1d1c-d5c2fc96e381
begin

function Base.:+(i::Interval{T}, len::T)::T where T <: Integer
   interv_len(i) + len
end

function Base.:+(len::T, i::Interval{T})::T where T <: Integer
   interv_len(i) + len
end

end

# ╔═╡ 2be6f644-6826-11eb-168f-31248b1f7b57
begin
	@test IntervInt64(1, 10) + 10 == 19
	@test 6 + IntervInt64(2, 10) == 14
end

# ╔═╡ 647479b4-6826-11eb-0e26-8ffb421b8aba
md"""
##### Sorting an array of Intervals

We will sort on ascending order given the lower bound.
"""

# ╔═╡ 73948ede-6826-11eb-3414-e184635bb6b4
begin
	ary = [
    	IntervInt64(2, 4), IntervInt64(8, 10), IntervInt64(1, 5),
    	IntervInt64(1, 3), IntervInt64(4, 8), IntervInt64(3, 7),
    	IntervInt64(5, 11)
	]

	with_terminal() do
		sort!(ary, by = interv -> interv.lb, rev=false)
		for e ∈ ary
			println(e)
		end
	end
end

# ╔═╡ f204da94-6826-11eb-3007-c1a3d607e835
md"""
#### Merging overlapping intervals (in linear time)
"""

# ╔═╡ 30bf00fc-6827-11eb-14ea-8bf3ef119dfe
begin

const VT{T <: Integer} = Vector{Interval{T}}

function merge_interv(il::VT{T})::VT{T} where T <: Integer
	"""
	Merge all the intervals from given vector
	"""

	size(il, 1) == 1 && return il   ## identity

	n = size(il, 1)
	n_il = VT{T}(undef, n)  ## at most the size of given vector
	i₁, jx = il[1], 1

	for ix ∈ 2:n
		i₂ = il[ix]
		if interv_inter(i₁, i₂)
			i₁ = Interval{T}(min(i₁.lb, i₂.lb), max(i₁.ub, i₂.ub))
		else
			n_il[jx] = i₁
			jx += 1
			i₁ = i₂
		end
	end

	n_il[jx] = i₁
	view(n_il, 1:jx)
end

end

# ╔═╡ 344e9cda-6829-11eb-3c5d-61e4908b38a0
begin
	ary1 = [
    	IntervInt64(2, 4), IntervInt64(8, 10), IntervInt64(1, 5),
    	IntervInt64(1, 3), IntervInt64(4, 8), IntervInt64(3, 7),
    	IntervInt64(5, 11)
	]

	sort!(ary1, by = interv -> interv.lb)

	ary2 = [
    	IntervInt64(2, 4), IntervInt64(8, 10), IntervInt64(1, 5),
    	IntervInt64(17, 20), IntervInt64(11, 15), IntervInt64(3, 7),
	]

	sort!(ary2, by = interv -> interv.lb)

	with_terminal() do
		for e ∈ ary1
			println(e)
		end
		println()

		for e ∈ ary2
			println(e)
		end
	end
end

# ╔═╡ 88e56bca-6829-11eb-3da4-d9b2299f2801
@test merge_interv(ary1) == [IntervInt64(1, 11)]

# ╔═╡ fc900436-6829-11eb-1783-b1e82c817e8d
@test merge_interv(ary2) == [IntervInt64(1, 7),
	IntervInt64(8, 10),
    IntervInt64(11, 15),
	IntervInt64(17, 20)
]

# ╔═╡ 158a8cb8-682a-11eb-1d50-d922aa7274a2
begin
	ary3 = [IntervInt64(1, 2), IntervInt64(11, 15)]
	sort!(ary3, by = interv -> interv.lb)

	@test merge_interv(ary3) == ary3 # Identity
end

# ╔═╡ 27a6a1ac-682a-11eb-39f5-9b9159b03088
with_terminal() do
	@btime merge_interv(ary1)
end

# ╔═╡ 33bdfd96-682a-11eb-3f55-93f22f7b1fdf
with_terminal() do
	@btime merge_interv(ary2)
end

# ╔═╡ 8ff62b88-682a-11eb-355e-03ea9b57ab05
# import Base: convert #promote_rule

# overload + already done

Base.convert(::Type{T}, i::Interval{T}) where T <: Integer = i.ub - i.lb
# promote_rule(::Interval{T}, ::Interval{<:Integer}) = T

# ╔═╡ c7b6c8fc-682a-11eb-2cc9-5706878cf4a3
md"""
##### Compute Sum of Interval array

###### Take 1
"""

# ╔═╡ d359aff8-682a-11eb-1061-89a949359934
function sum_intervals(il::VT{T}; merger_fn=merge_interv)::T where T <: Integer
    size(il, 1) == 1 && return interv_len(il[1])

	foldl((s, i) -> s += interv_len(i),
		merger_fn(sort(il, by=interv -> interv.lb));
		init=0)
end

# ╔═╡ 697050a0-682b-11eb-0ec0-650a0a68b11a
begin
	s_ary11 = [
    	IntervInt64(2, 4), IntervInt64(8, 10), IntervInt64(1, 5),
    	IntervInt64(1, 3), IntervInt64(4, 8), IntervInt64(3, 7),
    	IntervInt64(5, 11)
	]

	s_ary12 = [
    	IntervInt64(2, 4), IntervInt64(8, 10), IntervInt64(1, 5),
    	IntervInt64(1, 3), IntervInt64(4, 8), IntervInt64(3, 7),
    	IntervInt64(5, 11)
	]

	@test sum_intervals(s_ary11) == sum_intervals(s_ary12)
end

# ╔═╡ 9cc8c4b4-682b-11eb-046f-b9f2e3e82908
begin
	_ary11 = [IntervInt64(1, 2), IntervInt64(11, 15)]
	@test sum_intervals(_ary11) == 5

	_ary12 = [IntervInt64(1, 2), IntervInt64(11, 15), IntervInt64(6, 10)]
	@test sum_intervals(_ary12) == 9

	_ary13 = [
    	IntervInt64(1, 5), IntervInt64(10, 20), IntervInt64(1, 6),
    	IntervInt64(16, 19), IntervInt64(5, 11)
	]
	@test sum_intervals(_ary13) == 19
end

# ╔═╡ ca5c153e-682b-11eb-390b-ef6bbc821d95
with_terminal() do
	@btime sum_intervals(_ary13)
end

# ╔═╡ 1ae7fe64-682c-11eb-349c-3f3919aafd76
md"""
###### Take 2
"""

# ╔═╡ 2d3db23e-682c-11eb-2baa-91d0f9caf485
function sum_intervals_2(il::VT{T}; merger_fn=merge_interv)::T where T <: Integer
 	size(il, 1) == 1 && return interv_len(il[1])

	map(iv -> interv_len(iv),
		merger_fn(sort(il, by=interv -> interv.lb))) |> sum

end

# ╔═╡ 21c48f2c-682c-11eb-2971-a34d13b77f1c
@test sum_intervals_2(s_ary11) == sum_intervals_2(s_ary12)

# ╔═╡ 9e3b563a-682c-11eb-03b6-292ba8b72db5
begin
	@test sum_intervals_2(_ary11) == 5
	@test sum_intervals_2(_ary12) == 9
	@test sum_intervals_2(_ary13) == 19
end

# ╔═╡ 134299a0-6852-11eb-17ad-f7fa7323406b
with_terminal() do
	@btime sum_intervals_2(_ary13)
end

# ╔═╡ 218593a8-682c-11eb-1eb8-579b44dfd57c
md"""
###### Take 3
"""

# ╔═╡ 04684c04-6852-11eb-3522-bf07f9428110
function sum_intervals_3(il::VT{T}; merger_fn=merge_interv)::T where T <: Integer
 	size(il, 1) == 1 && return interv_len(il[1])

    reduce(+,
		merger_fn(sort(il, by=interv -> interv.lb))
	)
end

# ╔═╡ 40cfa4dc-6853-11eb-2469-ef9c84d3a4f6
@test sum_intervals_3(s_ary11) == sum_intervals_3(s_ary12)

# ╔═╡ 39406316-6853-11eb-2783-35b77b3b62cc
begin
	@test sum_intervals_3(_ary11) == 5
	@test sum_intervals_3(_ary12) == 9
	@test sum_intervals_3(_ary13) == 19
end

# ╔═╡ 47aa8d46-6853-11eb-2505-5727d9c83eb2
with_terminal() do
	@btime sum_intervals_3(_ary13)
end

# ╔═╡ Cell order:
# ╟─19046b20-6820-11eb-3fac-e993b8847dc0
# ╠═5bd9e6aa-6820-11eb-2b20-3f17c0045841
# ╟─698441e4-6820-11eb-2ed2-e90fbf53d7f0
# ╠═7da89592-6820-11eb-0ad7-699807d1d952
# ╠═9e07bbc4-6820-11eb-07ca-77f29e8a3030
# ╠═b9125d20-6820-11eb-3773-07dd4728988d
# ╟─f6264b68-6820-11eb-1fc4-afcb22381c09
# ╠═1640dbfe-6821-11eb-30bb-37b1f76609a3
# ╠═3cdd4d22-6821-11eb-3c20-a7997b70c44d
# ╟─46c62dea-6821-11eb-0b18-771fd704ee84
# ╠═91d63a66-6821-11eb-07a4-97c4a000b482
# ╠═14f1261a-6823-11eb-3d78-ebc8784d5ef8
# ╟─aac494f8-6823-11eb-138e-59b1641e416b
# ╠═c2440f3a-6823-11eb-1fac-07057f45e3e3
# ╠═a68980ee-6824-11eb-06e2-b5a9527b24b5
# ╟─dda42a42-6825-11eb-1896-c7c0c2f6577d
# ╠═eb0fe112-6825-11eb-1d1c-d5c2fc96e381
# ╠═2be6f644-6826-11eb-168f-31248b1f7b57
# ╟─647479b4-6826-11eb-0e26-8ffb421b8aba
# ╠═73948ede-6826-11eb-3414-e184635bb6b4
# ╟─f204da94-6826-11eb-3007-c1a3d607e835
# ╠═30bf00fc-6827-11eb-14ea-8bf3ef119dfe
# ╠═344e9cda-6829-11eb-3c5d-61e4908b38a0
# ╠═88e56bca-6829-11eb-3da4-d9b2299f2801
# ╠═fc900436-6829-11eb-1783-b1e82c817e8d
# ╠═158a8cb8-682a-11eb-1d50-d922aa7274a2
# ╠═27a6a1ac-682a-11eb-39f5-9b9159b03088
# ╠═33bdfd96-682a-11eb-3f55-93f22f7b1fdf
# ╠═8ff62b88-682a-11eb-355e-03ea9b57ab05
# ╟─c7b6c8fc-682a-11eb-2cc9-5706878cf4a3
# ╠═d359aff8-682a-11eb-1061-89a949359934
# ╠═697050a0-682b-11eb-0ec0-650a0a68b11a
# ╠═9cc8c4b4-682b-11eb-046f-b9f2e3e82908
# ╠═ca5c153e-682b-11eb-390b-ef6bbc821d95
# ╟─1ae7fe64-682c-11eb-349c-3f3919aafd76
# ╠═2d3db23e-682c-11eb-2baa-91d0f9caf485
# ╠═21c48f2c-682c-11eb-2971-a34d13b77f1c
# ╠═9e3b563a-682c-11eb-03b6-292ba8b72db5
# ╠═134299a0-6852-11eb-17ad-f7fa7323406b
# ╟─218593a8-682c-11eb-1eb8-579b44dfd57c
# ╠═04684c04-6852-11eb-3522-bf07f9428110
# ╠═40cfa4dc-6853-11eb-2469-ef9c84d3a4f6
# ╠═39406316-6853-11eb-2783-35b77b3b62cc
# ╠═47aa8d46-6853-11eb-2505-5727d9c83eb2


@doc raw"""
    PolynomialKernel([a=1 [,c=1 [,d=3]]])

The polynomial kernel is a Mercer kernel given by:

```math
\kappa(\mathbf{x},\mathbf{y}) =
(a \mathbf{x}^\intercal \mathbf{y} + c)^d
\qquad \alpha > 0, \; c \geq 0, \; d \in \mathbb{Z}_{+}
```
"""
struct PolynomialKernel{T<:AbstractFloat,U<:Integer} <: MercerKernel{T}
    a::HyperParameter{T}
    c::HyperParameter{T}
    d::HyperParameter{U}
    function PolynomialKernel{T}(a::Real, c::Real, d::U) where {T<:AbstractFloat,U<:Integer}
        new{T,U}(HyperParameter(convert(T,a), interval(OpenBound(zero(T)), nothing)),
                 HyperParameter(convert(T,c), interval(ClosedBound(zero(T)), nothing)),
                 HyperParameter(d, interval(ClosedBound(one(U)), nothing)))
    end
end
function PolynomialKernel(a::T1=1.0, c::T2=one(T1), d::Integer=3) where {T1<:Real,T2<:Real}
    PolynomialKernel{floattype(T1,T2)}(a, c, d)
end

@inline eltypes(::Type{<:PolynomialKernel{T,U}}) where {T,U} = (T,U)
@inline thetafieldnames(κ::PolynomialKernel) = Symbol[:a, :c]

@inline polynomialkernel(z::T, a::T, c::T, d::U) where {T<:AbstractFloat,U<:Integer} = (a*z + c)^d

@inline basefunction(::PolynomialKernel) = ScalarProduct()
@inline function kappa(κ::PolynomialKernel{T}, z::T) where {T}
    polynomialkernel(z, getvalue(κ.a), getvalue(κ.c), getvalue(κ.d))
end


#############################################################################
#############################################################################
#
# This file implement several basic algorithms for integers.
#
#############################################################################
#############################################################################

"""
The quotient between two numbers
"""
function quo(a::T,b::T) where T <: Integer
    a < 0 && return -quo(-a,b)
    a < b && return 0
    return 1 + quo(a-b, b)
end

"""
Euclid's algorithm.
"""
function euclid_alg(a, b; rem_function = %)
    b == 0 && return a
    return euclid_alg(b, rem_function(a,b); rem_function = rem_function)
end

"""
Euclid's algorithm on multiple arguments.
"""
euclid_alg(a...) = foldl((a,b)->euclid_alg(a,b), a; init = 0)

"""
Euclid's algorithm on a vector.
"""
euclid_alg(a::Vector{T}) where T <: Integer = euclid_alg(a...)

"""
The extended Euclidean algorithm.
"""
function ext_euclid_alg(a, b, rem_function = %, div_function = ÷)
    a == 0 && return b, 0, 1
    g, t, s = ext_euclid_alg(rem_function(b,a), a, rem_function, div_function)
    s = s - div_function(b,a)*t
    @assert g == a*s + b*t
    return g, s, t
end

"""
Display the result of the extended Euclidean algorithm.
"""
pretty_print_egcd((a,b),(g,s,t)) = println("$a × $s + $b × $t = $g") #\times + [TAB]

"""
Integer inverse symmetric mod
"""
function int_inverse_mod(a::Int, m::Int)::Int
    if mod(a, m) == 0
        error("Can't find inverse of $a mod $m because $m divides $a")
    end
    return mod(ext_euclid_alg(a,m)[2],m)
end


"""
A library of functions used in the Julia Companion to
the book **Vectors, Matrices, and Least Squares**.
"""
module VMLS

using LinearAlgebra
using SparseArrays
using DelimitedFiles
export avg, rms, stdev, ang, correl_coef, eye, speye, kmeans, gram_schmidt
export diagonal, spdiagonal, vandermonde, toeplitz, linspace
export confusion_matrix, row_argmax, one_hot, mols_solve, cls_solve
export levenberg_marquardt, aug_lag_method
export house_sales_data, population_data, petroleum_consumption_data
export vehicle_miles_data, temperature_data, iris_data, ozone_data
export regularized_fit_data, portfolio_data, lq_estimation_data
export orth_dist_reg_data

"""
    avg(x)

Returns the average of the elements of the vector or matrix `x`.
"""
avg(x) = sum(x) / length(x)

"""
    rms(x)

Returns the RMS value of the elements of the vector or matrix `x`.
"""
rms(x) = norm(x) / sqrt(length(x))

"""
    stdev(x)

Returns the standard deviation of the elements of the vector or matrix `x`.
"""
stdev(x) = rms(x .- avg(x))

"""
    ang(x,y)

Returns the angle in radians between non-zero vectors `x` and `y`.
"""
ang(x,y) = acos(x'*y  / (norm(x) * norm(y)))

"""
    correl_coef(x,y)

Returns the correlation coefficient between non-constant vectors `x` and `y`.
"""
function correl_coef(x,y)
    xdm = x .- avg(x)
    ydm = y .- avg(y)
    return xdm'*ydm / (norm(xdm) * norm(ydm))
end

"""
    eye(n)

Returns an `n` times `n` identity matrix.
"""
eye(n) = Matrix(1.0I, n, n)

"""
    speye(n)

Returns an `n` times `n` sparse identity matrix.
"""
speye(n) = sparse(1.0I, n, n)

"""
    kmeans(X, k; maxiters = 100, tol = 1e-5)

Applies the k-means algorithm for `k` clusters to the vectors stored in `X`.

The argument `X` is a one-dimensional array of N n-vectors or an `n` times `N` matrix.

`kmeans` returns a tuple with two elements: `assignment` is an array of N
integers between 1 and k with the cluster assignment for the N data points.
`reps` is an array of k n-vectors with the k cluster representatives.
"""
function kmeans(X, k; maxiters = 100, tol = 1e-5)
    if ndims(X) == 2
        X = [X[:,i] for i in 1:size(X,2)]
    end;
    N = length(X)
    n = length(X[1])
    distances = zeros(N)
    reps = [zeros(n) for j=1:k]
    assignment = [ rand(1:k) for i in 1:N ]
    Jprevious = Inf
    for iter = 1:maxiters
        for j = 1:k
            group = [i for i=1:N if assignment[i] == j]
            reps[j] = sum(X[group]) / length(group);
        end;
        for i = 1:N
            (distances[i], assignment[i]) =
                findmin([norm(X[i] - reps[j]) for j = 1:k])
        end;
        J = norm(distances)^2 /  N
        println("Iteration ", iter, ": Jclust = ", J, ".")
        if iter > 1 && abs(J - Jprevious) < tol * J
            return assignment, reps
        end
        Jprevious = J
    end
end

"""
    gram_schmidt(a; tol = 1e-10)

Applies the Gram-Schmidt algorithm to the vectors stored in the array `a`
and returns the result as an array of (orthonormal) vectors.  Issues a
warning if the original vectors are linearly dependent.
"""
function gram_schmidt(a; tol = 1e-10)
    q = []
    for i = 1:length(a)
        qtilde = a[i]
        for j = 1:i-1
            qtilde -= (q[j]'*a[i]) * q[j]
        end
        if norm(qtilde) < tol
            println("Vectors are linearly dependent.")
            return q
        end
        push!(q, qtilde/norm(qtilde))
    end;
    return q
end

"""
    diagonal(x)

Returns a diagonal matrix with the entries of the vector `x` on its diagonal.
"""
diagonal(x) = diagm(0 => x)

"""
    spdiagonal(x)

Returns a sparse diagonal matrix with the entries of the vector `x` on its diagonal.
"""
spdiagonal(x) = spdiagm(0 => x)

"""
    vandermonde(t, n)

Returns the Vandermonde matrix with `n` columns and i-th column `t`^(i-1).
"""
vandermonde(t,n) = hcat( [t.^i for i = 0:n-1]... )

"""
    toeplitz(a, n)

Returns the Toeplitz matrix with `n` columns and the vector `a` in the
leading positions of the first column.
"""
function toeplitz(a,n)
    m = length(a)
    T = zeros(n+m-1,n)
    for i=1:m
        T[i : n+m : end] .= a[i]
    end
    return T
end

"""
    linspace(a,b,n)

Returns a vector with `n` equally spaced numbers between `a` and `b`.
"""
linspace(a,b,n) = Vector(range(a, stop = b, length = n))

"""
    confusion_matrix(y, yhat, K=2)

Returns the confusion matrix for a data vector `y` and the vector of predictions
`yhat`.  If `K` is 2, the vectors are Boolean.  If `K` is greater than 2, they
contain integers from 1,...,K.
"""
function confusion_matrix(y, yhat, K=2)
    if K==2
        tp = sum( (y .== true) .& (yhat .== true) )
        fn = sum( (y .== true) .& (yhat .== false) )
        fp = sum( (y .== false) .& (yhat .== true) )
        tn = sum( (y .== false) .& (yhat .== false) )
        return [tp fn; fp tn]
    else
        C = zeros(K,K)
        for i = 1:K  for j=1:K
            C[i,j] = sum( (y .== i) .& (yhat .== j) )
        end end
        return C
    end;
end;

"""
    row_argmax(A)

Returns a `size(A,1)` vector with as i-th element the column index of the
largest element in row i of `A`.
"""
row_argmax(A) = [ argmax(A[i,:]) for i = 1:size(A,1) ]

"""
    one_hot(x, K)

Returns the one-hot encoding of the vector `x`.
"""
function one_hot(y, K)
    n = length(y)
    Y = zeros(n, K)
    for j in 1:K
       Y[findall(y .== j), j] .= 1.0
    end
    return Y
end

"""
    mols_solve(As, bs, lambdas)

Returns the solution of the multi-objective least squares problem
with coefficient matrices in the array `As`, right-hand side vectors in the
array `bs`, and weights in the array `lambdas`.
"""
function mols_solve(As,bs,lambdas)
    k = length(lambdas);
    Atil = vcat([sqrt(lambdas[i])*As[i] for i=1:k]...)
    btil = vcat([sqrt(lambdas[i])*bs[i] for i=1:k]...)
    return Atil \ btil
end

"""
   cls_solve(A, b, C, d)

Returns the solution of the constrained least squares problem with coefficient
matrices `A` and `C`, and right-hand side vectors or matrices `b` and `d`.
"""
function cls_solve(A,b,C,d)
    m, n = size(A)
    p, n = size(C)
    Q, R = qr([A; C])
    Q = Matrix(Q)
    Q1 = Q[1:m,:]
    Q2 = Q[m+1:m+p,:]
    Qtil, Rtil = qr(Q2')
    Qtil = Matrix(Qtil)
    w = Rtil \ (2*Qtil'*Q1'*b - 2*(Rtil'\d))
    return R \ (Q1'*b - Q2'*w/2)
end

"""
    levenberg_marquardt(f, Df, x1, lambda1; kmax=500, tol=1e-6)

Applies the Levenberg-Marquardt algorithm to the function defined in `f` and
`Df`, with starting point `x1` and initial regularization parameter `lambda1`.
The function returns the final iterate `x` and a dictionary with the convergence
history.
"""
function levenberg_marquardt(f, Df, x1, lambda1; kmax=100, tol=1e-6)
    n = length(x1)
    x = x1
    lambda = lambda1
    objectives = zeros(0,1)
    residuals = zeros(0,1)
    for k = 1:kmax
         fk = f(x)
         Dfk = Df(x)
         objectives = [objectives; norm(fk)^2]
         residuals = [residuals; norm(2*Dfk'*fk)]
         if norm(2*Dfk'*fk) < tol
             break
         end;
         xt = x - [ Dfk; sqrt(lambda)*eye(n) ] \ [ fk; zeros(n) ]
         if norm(f(xt)) < norm(fk)
             lambda = 0.8*lambda
             x = xt
         else
             lambda = 2.0*lambda
         end
    end
    return x, Dict([ ("objectives", objectives), ("residuals", residuals) ])
end

"""
    aug_lag_method(f, Df, g, Dg, x1, lambda1; kmax=100, feas_tol=1e-4, oc_tol=1e-4)

Applies the augmented Lagrangian method to the constrained nonlinear least
squares problem defined by `f`, `Df`, `g`, `Dg`, with starting point `x1`.
The subproblems are solved using the Levenberg-Marquardt method with
initial regularization parameter `lambda1`.  Returns the final iterate `x`,
multiplier `z`, and a dictionary with the convergence history.
"""
function aug_lag_method(f, Df, g, Dg, x1, lambda1;  kmax = 100,
    feas_tol = 1e-4, oc_tol = 1e-4)
    x = x1
    z = zeros(length(g(x)))
    mu = 1.0
    feas_res = [norm(g(x))]
    oc_res = [norm(2*Df(x)'*f(x) + 2*mu*Dg(x)'*z)]
    lm_iters = zeros(Int64,0,1);
    mus = [mu]
    for k=1:kmax
        F(x) = [f(x); sqrt(mu)*(g(x) + z/(2*mu))]
        DF(x) = [Df(x); sqrt(mu)*Dg(x)]
        x, hist = levenberg_marquardt(F, DF, x, lambda1, tol = oc_tol)
        z = z + 2*mu*g(x)
        feas_res = [feas_res; norm(g(x))]
        oc_res = [oc_res; hist["residuals"][end]]
        lm_iters = [lm_iters; length(hist["residuals"])]
        if norm(g(x)) < feas_tol
            break
        end
        mu = (norm(g(x)) < 0.25*feas_res[end-1]) ? mu : 2*mu
        mus = [mus; mu]
    end
    return x, z, Dict([ ("lm_iterations", lm_iters),
         ("feas_res", feas_res), ("oc_res", oc_res), ("mus", mus) ])
end

"""
    petroleum_consumption_data()

Returns a 34-vector with the world annual petroleum consumption between
1980 and 2013, in thousand barrels/day (discussed on page 252).
"""
petroleum_consumption_data() = [
    63122, 60953, 59551, 58785, 59795, 60083, 61819, 63107, 64978, 66090,
    66541, 67186, 67396, 67619, 69006, 70258, 71880, 73597, 74274, 75975,
    76928, 77732, 78457, 80089, 83063, 84558, 85566, 86724, 86046, 84972,
    88157, 89105, 90340, 91195 ]

include("house_sales_data.jl")
include("population_data.jl")
include("vehicle_miles_data.jl")
include("temperature_data.jl")
include("iris_data.jl")
include("ozone_data.jl")
include("regularized_fit_data.jl")
include("portfolio_data.jl")
include("lq_estimation_data.jl")
include("orth_dist_reg_data.jl")

end # module


"""
    RegularizedSVDPseudoInverse(A; ϵ=1e-8)

Structure able to solve linear system `A` using a regularized SVD.

The threshold `ϵ` controls the regularization.
Let ``σ_{\\max}`` be the larger singular value.
A singular value ``σ`` such that ``σ \\leq ϵ σ_{\\max}`` is approximated by
zero when solving the least-squares problem.
"""
struct RegularizedSVDPseudoInverse
    A
    ϵ
    F
    I
    Uϵ
    iΣϵ
    Vϵ
    function RegularizedSVDPseudoInverse(A; ϵ=1e-8)
        F = svd(A)
        I = argmin(F.S .> ϵ * F.S[1]) - 1
        if I == 0 I = length(F.S) end
        iΣϵ = spdiagm(0 => 1 ./ F.S[1:I])
        return new(A, ϵ, F, I, F.U[:,1:I], iΣϵ, F.V[:,1:I])
    end
end

"""
    condition_number(B::RegularizedSVDPseudoInverse)

Computes condition number of the matrix for which `B` was constructed.

The condition number is defined as the ratio of the largest over the smallest
singular values of the matrix ``A``.
```math
κ := \\frac{σ_{\\max}}{σ_{\\min}}.
```
"""
function condition_number(B::RegularizedSVDPseudoInverse)
    return B.F.S[1] / B.F.S[end]
end

"""
    solve_via_regularizedSVD(B::RegularizedSVDPseudoInverse, b)

Solve the linear system ``Ax=b`` using the regularized SVD of ``A``.
"""
function solve_via_regularizedSVD(B::RegularizedSVDPseudoInverse, b)
    return B.Vϵ * ( B.iΣϵ * ( adjoint(B.Uϵ) * b ))
end


function GaussSeidelSolve(A::Any,B::Vector; Guess=false,ϵ=0.05, MaxItr=30, λ=1)
      #A = the matrix of coefficients 'A' must be squared
  #B = The Vector 'B' right hand side vector Vertical
    m,n =size(A)
    v = size(B,1)
    if m != n
        error("Matrix must be squared")
    elseif m != v
        error("Vector Length must be equal to Matrix")
    end

    w=1:m          #Count m range

    c=zeros(m,m-1)
    er=[1.]
    cond=false
    for i=1:m
        W=w[w.!=i]
       c[i,:]= A[i,W]
    end

    Guess==false ? X=zeros(1,m) : X=Guess'
    itr=2

    while cond==false

        X=vcat(X,X[[itr-1],:])
        #Relaxation
        X[itr,:]=λ.*X[itr,:].+(1-λ).*X[itr-1,:]

        for i=1:m
            W=w[w.!=i]
            X[itr,i]=(B[i]-(c[i,:]'*X[itr,W]))/A[i,i]
        end

        e=maximum((X[itr,:].-X[itr-1,:])./X[itr,:])
        push!(er,e)
        cond=er[itr] .<=ϵ || itr .>= MaxItr
        println("Iteration $(itr-1) Error $(er[itr]*100) %")

        itr += 1
    end #End while loop

    return X[end,:], itr
end


using FdeSolver
using Plots

## inputs
I0 = 0.001             # intial value of infected
tSpan = [0, 100]       # [intial time, final time]
y0 = [1 - I0, I0, 0]   # initial values [S0,I0,R0]
α = [1, 1, 1]          # order of derivatives
h = 0.1                # step size of computation (default=0.01)

## ODE model
par = [0.4, 0.04] # parameters [β, recovery rate]

function F(t, y, par)

    # parameters
    β = par[1]    # infection rate
    γ = par[2]    # recovery rate

    S = y[1]   # Susceptible
    I = y[2]   # Infectious
    R = y[3]   # Recovered

    # System equation
    dSdt = - β .* S .* I
    dIdt = β .* S .* I .- γ .* I
    dRdt = γ .* I

    return [dSdt, dIdt, dRdt]

end

## Jacobian of ODE system
function JacobF(t, y, par)

    # parameters
    β = par[1]     # infection rate
    γ = par[2]     # recovery rate

    S = y[1]    # Susceptible
    I = y[2]    # Infectious
    R = y[3]    # Recovered

    # System equation
    J11 = - β * I
    J12 = - β * S
    J13 =  0
    J21 =  β * I
    J22 =  β * S - γ
    J23 =  0
    J31 =  0
    J32 =  γ
    J33 =  0

    J = [J11 J12 J13
         J21 J22 J23
         J31 J32 J33]

    return J

end
## Solution
t, Yapp = FDEsolver(F, tSpan, y0, α, par, h = h)
t1, Yapp1 = FDEsolver(F, tSpan, y0, α, par, JF = JacobF, h = h)

## plotting
plot(t, Yapp, linewidth = 5, title = "Numerical solution of SIR model",
     xaxis = "Time (t)", yaxis = "SIR populations", label=["Susceptible" "Infectious" "Recovered"])
plot!(t1, Yapp1, linewidth = 5, ls = :dash, label=["S_Jacob" "I_Jacob" "R_Jacob"])


# ---
# title: 202. Happy Number
# id: problem202
# author: Tian Jun
# date: 2020-10-31
# difficulty: Easy
# categories: Hash Table, Math
# link: <https://leetcode.com/problems/happy-number/description/>
# hidden: true
# ---
#
# Write an algorithm to determine if a number `n` is "happy".
#
# A happy number is a number defined by the following process: Starting with any
# positive integer, replace the number by the sum of the squares of its digits,
# and repeat the process until the number equals 1 (where it will stay), or it
# **loops endlessly in a cycle** which does not include 1. Those numbers for
# which this process **ends in 1** are happy numbers.
#
# Return True if `n` is a happy number, and False if not.
#
# **Example:  **
#
#
#
#     Input: 19
#     Output: true
#     Explanation: 12 + 92 = 82
#     82 + 22 = 68
#     62 + 82 = 100
#     12 + 02 + 02 = 1
#
#
#
## @lc code=start
using LeetCode

## add your code here:
## @lc code=end


# ---
# title: 202. Happy Number
# id: problem202
# author: zhwang
# date: 2022-03-22
# difficulty: Easy
# categories: Hash Table, Math
# link: <https://leetcode.com/problems/happy-number/description/>
# hidden: true
# ---
#
# Write an algorithm to determine if a number `n` is "happy".
#
# A happy number is a number defined by the following process: Starting with any
# positive integer, replace the number by the sum of the squares of its digits,
# and repeat the process until the number equals 1 (where it will stay), or it
# **loops endlessly in a cycle** which does not include 1. Those numbers for
# which this process **ends in 1** are happy numbers.
#
# Return True if `n` is a happy number, and False if not.
#
# **Example:  **
#
#
#
#     Input: 19
#     Output: true
#     Explanation: 12 + 92 = 82
#     82 + 22 = 68
#     62 + 82 = 100
#     12 + 02 + 02 = 1
#
#
#
## @lc code=start
using LeetCode
function ishappy(n::Int)::Bool
    while n != 1
        n == 4 && return false ## all loops must contain 4
        n = sum(digits(n) .^ 2)
    end
    true
end
## @lc code=end

## @lc test=start
@testset "202.happy-number.jl" begin
    @test all(ishappy(i) for i in (1, 7, 10, 13, 19, 23, 28, 31, 123456))
    @test !any(ishappy(i) for i in 2:6)
    @test !any(ishappy(i) for i in (8, 9, 11, 12, 12345))
end
## @lc test=end

## @lc info=start
# link: [solution to 202]()
## @lc info=end


# # A trivial SOS decomposition example

#md # [![](https://mybinder.org/badge_logo.svg)](@__BINDER_ROOT_URL__/generated/sos_decomposition.ipynb)
#md # [![](https://img.shields.io/badge/show-nbviewer-579ACA.svg)](@__NBVIEWER_ROOT_URL__/generated/sos_decomposition.ipynb)
# **Contributed by**: votroto
# **Adapted from**: Examples 3.25 of [BPT12]
#
# [BPT12] Blekherman, G.; Parrilo, P. A. & Thomas, R. R.
# *Semidefinite Optimization and Convex Algebraic Geometry*.
# Society for Industrial and Applied Mathematics, **2012**.

using Test #src
using DynamicPolynomials
using SumOfSquares
import CSDP

# The polynomial `p = x^2 - x*y^2 + y^4 + 1` is SOS.
# We can, for example, decompose it as
# `p = 3/4*(x - y^2)^2 + 1/4*(x + y)^2 + 1`,
# which clearly proves that `p` is SOS, and there are infinitely many other ways
# to decompose `p` into sums of squares.

# We can use SumOfSquares.jl to find such decompositions.

# First, setup the polynomial of interest.

@polyvar x y
p = x^2 - x*y^2 + y^4 + 1

# Secondly, constrain the polynomial to be nonnegative.
# SumOfSquares.jl transparently reinterprets polyonmial nonnegativity as the
# appropriate SOS certificate for polynomials nonnegative on semialgebraic sets.

model = SOSModel(CSDP.Optimizer)
@constraint(model, cref, p >= 0)

# Thirdly, optimize the feasibility problem!

optimize!(model)

# Lastly, recover a SOS decomposition.
# In general, SOS decompositions are not unique!

sos_dec = sos_decomposition(cref, 1e-4)

# Converting, rounding, and simplifying - Huzza, Back where we began!

polynomial(sos_dec, Float32)

# ## A deeper explanation and the unexplained `1e-4` parameter

# `p = x^2 - x*y^2 + y^4 + 1` can be represented in terms of its Gram matrix as

gram = gram_matrix(cref)

#-

gram.basis.monomials' * gram.Q * gram.basis.monomials

# where the matrix `gram.Q` is positive semidefinite, because `p` is SOS. If we
# could only get the decomposition `gram.Q = V' * V`, the SOS decomposition would
# simply be `||V * monomials||^2`.

# Unfortunately, we can not use Cholesky decomposition, since gram `Q` is only
# semidefinite, not definite. Hence, SumOfSquares.jl uses SVD decomposition
# instead and discards small singular values (in our case `1e-4`).


export periapsis, apoapsis, semimajoraxis, eccentricity
export meananomaly, trueanomaly, eccentricanomaly
export orbitalperiod, orbitaldistance, orbit

"""
    periapsis(a, e)

Compute the periapsis (closest approach) distance using semi-major axis and eccentricity
"""
periapsis(a, e) = a*(1 - e)

"""
    apoapsis(a, e)

Compute the apoapsis (farthest distance) distance using semi-major axis and eccentricity
"""
apoapsis(a, e) = a*(1 + e)

"""
    semimajoraxis(T, m)

Compute the [semi-major axis](https://en.wikipedia.org/wiki/Semi-major_and_semi-minor_axes) of an orbit
"""
semimajoraxis(T, m) = (𝐆*m*T^2/(4π^2))^(1/3)

"""
    eccentricity(rₚ, rₐ)

Compute [eccentricity](https://en.wikipedia.org/wiki/Orbital_eccentricity)
"""
eccentricity(rₚ, rₐ) = (rₐ - rₚ)/(rₐ + rₚ)

"""
    meananomaly(E, e)

Compute the [mean anomaly](https://en.wikipedia.org/wiki/Mean_anomaly)
"""
meananomaly(E, e) = E - e*sin(E)

"""
    trueanomaly(E, e)

Compute the [true anomaly](https://en.wikipedia.org/wiki/True_anomaly)
"""
function trueanomaly(E, e)
    f = 2*atan(sqrt((1 + e)/(1 - e))*tan(E/2))
    #use [0,2π] instead of [-π,π]
    f < 0 ? f + 2π : f
end

"""
    trueanomaly(t, a, m, e)

Compute the [true anomaly](https://en.wikipedia.org/wiki/True_anomaly)
"""
trueanomaly(t, a, m, e) = trueanomaly(eccentricanomaly(t, a, m, e), e)

"""
    eccentricanomaly(t, a, m, e)

Numerically compute the [eccentric anomaly](https://en.wikipedia.org/wiki/Eccentric_anomaly) using [Kepler's equation](https://en.wikipedia.org/wiki/Kepler%27s_equation)
"""
function eccentricanomaly(t, a, m, e)
    @assert t >= 0 "time must be positive"
    #Kepler's Third Law
    T = orbitalperiod(a, m)
    #definition of mean anomaly
    M = 2π*rem(t, T)/T
    #eccentric anomaly must be found numerically
    E = regulafalsi((E,p)->meananomaly(E, e) - M, 0, 2π)
    return E
end

"""
    orbitalperiod(a, m)

[Kepler's Third Law](https://en.wikipedia.org/wiki/Kepler%27s_laws_of_planetary_motion#Third_law) describing the orbital period of an elliptical orbit
"""
orbitalperiod(a, m) = 2π*√(a^3/(𝐆*m))

"""
    orbitaldistance(a, f, e)

Compute the distance of a planet from its host
"""
orbitaldistance(a, f, e) = a*(1 - e^2)/(1 + e*cos(f))

"""
    orbitaldistance(t, a, m, e)

Compute the distance of a planet from its host, assuming the planet is at periapsis at t=0
"""
orbitaldistance(t, a, m, e) = orbitaldistance(a, trueanomaly(t, a, m, e), e)

"""
    orbit(a, m, e, N=1000)

Create a distance time-series of `N` points for an elliptical orbit, returning vectors for time, distance, and true anomaly
"""
function orbit(a, m, e, N::Int=1000)
    T = orbitalperiod(a, m)
    t = LinRange(0, T, N+1)[1:end-1]
    f = trueanomaly.(t, a, m, e)
    r = orbitaldistance.(a, f, e)
    return t, r, f
end


using DifferentialEquations
using Plots
using Markdown

description = md"""

Solve the differential equation
``du/dt = f(u,p,t) = α u``
where t∈[0,1] and u₀= 1/2

"""
display(description)

α = 1.01
f(u, p, t) = α*u

# define IC's and time span
u₀ = 1/2
tspan = (0.0, 1.0)

prob = ODEProblem(f, u₀, tspan)
sol = solve(prob)  # type ?solve at repl to see list of solver options (methods, tolerance setting, etc...)

# visualize the solution
plot(sol, linewidth=5, title="Solution to the linear ODE", xaxis="Time (t)", yaxis="u(t) (μm)", label="Numerical Solution")
plot!(sol.t, t->0.5*exp(1.01t), lw=3, ls=:dash, label="True Solution")


"""
    gelombang(f,g,a,b,c,m,n)
berisi program untuk mencari solusi persamaan differensial parsial yaitu persamaan gelombang.
Program ini secara
default berisi 7 masukan, yaitu fungsi nilai batas `f(x)` dan `g(x)`, batas solusi `a` dan `b`,
koefisien persamaan gelombang `c`, dan jumlah grid solusi x dan t yaitu `m` dan `n`.
# Example
```jl
julia> f(x) = sin(pi*x)+sin(2*pi*x);

julia> g(x) = 0;

julia> U = gelombang(f,g,1,0.5,2,11,11)
11×11 Adjoint{Float64,Array{Float64,2}}:
 0.0   0.896802      1.53884     1.76007   …  -0.14204      -0.363271   -0.278768   0.0
 0.0   0.769421      1.32844     1.53884       1.11022e-16  -0.210404   -0.181636   0.0
 0.0   0.431636      0.769421    0.948401      0.360616      0.181636    0.0683644  0.0
 0.0  -2.22045e-16   0.0515989   0.181636      0.769421      0.639384    0.363271   0.0
 ⋮                                         ⋱                                        ⋮
 0.0  -0.363271     -0.639384   -0.769421     -0.181636     -0.0515989   0.0        0.0
 0.0  -0.0683644    -0.181636   -0.360616     -0.948401     -0.769421   -0.431636   0.0
 0.0   0.181636      0.210404    0.0          -1.53884      -1.32844    -0.769421   0.0
 0.0   0.278768      0.363271    0.14204   …  -1.76007      -1.53884    -0.896802   0.0
```
return solusi persamaan gelombang `U`.
"""
function gelombang(f,g,a,b,c,m,n)
  h = a/(m-1);
  x = 0:h:1;
  k = b/(n-1);
  r = c*k/h;
  U = zeros(m,n);
  for i = 2:m-1
    U[i,1] = f(x[i]);
    U[i,2] = (1-r^2)*f(x[i]) + k*g(x[i]) + r^2/2*(f(x[i+1])+f(x[i-1]));
  end
  for j = 3:n
    for i = 2:(m-1)
      U[i,j] = (2-2*r^2)*U[i,j-1]+r^2*(U[i-1,j-1]+U[i+1,j-1])-U[i,j-2];
    end
  end
  U = U';
end


# This file illustrates the calculation of energy spectral density and
# total energy of a complex exponential pulse.

using FFTW
using Plots

# Construct the signal
f = 20
fs = 100
l = 2^10
ts = 1 / fs
t = (0 : l -1) * ts
x = exp.(1im * 2π * f * t)

# Compute fft
xf = fft(x) * ts
ff = 0 : fs / (l - 1) : fs

# Compute energy
esd = abs.(xf).^2
Δf = fs / (l - 1)
Ef = sum(esd) * Δf              # Energy calculated in time
Et = sum(abs.(x).^2) * ts       # Energy calculated in frequency
@show Ef, Et

# Plots
plt = plot(layout=1)
plot!(ff, esd, subplot=1)
xlabel!("Frequency [Hz]")
ylabel!("Ψ(f)")
title!("Energy Spectal Density")
display(plt)


using Plots; pyplot()

xVals = [-2,3,5,6,12,14]
yVals = [7,2,9,3,12,3]
n = length(xVals)

V = [xVals[i+1]^(j) for i in 0:n-1, j in 0:n-1]
c = V \ yVals
xGrid = -5:0.01:20
f1(x) = c'*[x^i for i in 0:n-1]

beta0, beta1 = 4.58, 0.17
f2(x) = beta0 + beta1*x

plot(xGrid,f1.(xGrid), c=:blue, label="Polynomial 5th order")
plot!(xGrid,f2.(xGrid),c=:red, label="Linear model")
scatter!(xVals,yVals,
	c=:black, shape=:xcross, ms=8,
	label="Data points", xlims=(-5,20), ylims=(-50,50),
    	xlabel = "x", ylabel = "y")


"""
Program to check if a number is an [Armstrong/Narcissistic number](https://en.wikipedia.org/wiki/Narcissistic_number) in decimal system.

Armstrong number is a number that is the sum of its own digits raised to the power of the number of digits.

Contributed By:- [Ashwani Rathee](https://github.com/ashwani-rathee)

A positive integer is called an Armstrong number (of order n) if

`abcd... = a^n + b^n + c^n + d^n +....`
"""
function is_armstrong(x)
    # If the number is less than 0, then it is not a armstrong number.
    if x < 0
        return false
    elseif x == 0
        return true
    else
        # Finding the total number of digits in the number
        length = Int(floor(log10(x) + 1))
        result = 0
        temp = x
        for i in 1:length
            val = temp % 10
            # Finding each digit raised to the power total digit and add it to the total sum
            result = result + Int(trunc(val))^length
            temp = temp / 10
        end
        return x == result ? true : false
    end
end


"""
This functions takes two strings and calculates de edit distance, which is the number
of changes of two stringsn when both display different lengths:

```@example
    s1 = "ATCTCGT"
    s2 = "ACTCCTC"
    edit_distance(s1,s2)[1] = 3.0
    edit_distance(s1,s2)[2] =  [0.0 1.0 2.0 3.0 4.0 5.0 6.0 7.0;
                               1.0 0.0 1.0 2.0 3.0 4.0 5.0 6.0;
                               2.0 1.0 1.0 1.0 2.0 3.0 4.0 5.0;
                               3.0 2.0 1.0 2.0 1.0 2.0 3.0 4.0;
                               4.0 3.0 2.0 1.0 2.0 2.0 2.0 3.0;
                               5.0 4.0 3.0 2.0 1.0 2.0 3.0 2.0;
                               6.0 5.0 4.0 3.0 2.0 2.0 3.0 3.0;
                               7.0 6.0 5.0 4.0 3.0 3.0 2.0 3.0]
```
This functions outputs the edit distance and the alignment matrix as well.
"""
function edit_distance(s1::String, s2::String)
    D::Array = zeros(length(s1)+1,length(s2)+1)
    D[:,1]= 0:length(s1)
    D[1,:] = 0:length(s2)
    for i in 2:length(s1)+1
        for j in 2:length(s2)+1
            (s1[i-1] != s2[j-1]) ? δ = 1 : δ = 0
            D[i,j] = min(D[i-1,j-1]+δ, D[i-1,j]+1, D[i,j-1]+1)
        end
    end
    return D[length(s1)+1, length(s2)+1],D
end


#:::::::::::::::::::::::::::::::::::::::::::::::::::
# question 1
#:::::::::::::::::::::::::::::::::::::::::::::::::::
using Optim
f(x) = -x[1]^4-10x[1]^3-2x[1]^2-3x[1]-2
minusf(x) = x[1]^4+10x[1]^3+2x[1]^2+3x[1]+2
startval = rand(1)   # random starting value
result = optimize(minusf, startval, BFGS())

#:::::::::::::::::::::::::::::::::::::::::::::::::::
# question 2
#:::::::::::::::::::::::::::::::::::::::::::::::::::
using DataFrames
using CSV
using HTTP
url = "https://raw.githubusercontent.com/OU-PhD-Econometrics/fall-2020/master/ProblemSets/PS1-julia-intro/nlsw88.csv"
df = CSV.read(HTTP.get(url).body, DataFrame)
X = [ones(size(df,1),1) df.age df.race.==1 df.collgrad.==1]
y = df.married.==1

function ols(beta, X, y)
    ssr = (y.-X*beta)'*(y.-X*beta)
    return ssr
end

beta_hat_ols = optimize(b -> ols(b, X, y), rand(size(X,2)), LBFGS(), Optim.Options(g_tol=1e-6, iterations=100_000, show_trace=true))
println(beta_hat_ols.minimizer)

using GLM
bols = inv(X'*X)*X'*y
df.white = df.race.==1
bols_lm = lm(@formula(married ~ age + white + collgrad), df)

#:::::::::::::::::::::::::::::::::::::::::::::::::::
# question 3
#:::::::::::::::::::::::::::::::::::::::::::::::::::
function logit(alpha, X, d)

    # your turn

    return loglike
end

#:::::::::::::::::::::::::::::::::::::::::::::::::::
# question 4
#:::::::::::::::::::::::::::::::::::::::::::::::::::
# see Lecture 3 slides for example

#:::::::::::::::::::::::::::::::::::::::::::::::::::
# question 5
#:::::::::::::::::::::::::::::::::::::::::::::::::::
freqtable(df, :occupation) # note small number of obs in some occupations
df = dropmissing(df, :occupation)
df[df.occupation.==10,:occupation] .= 9
df[df.occupation.==11,:occupation] .= 9
df[df.occupation.==12,:occupation] .= 9
df[df.occupation.==13,:occupation] .= 9
freqtable(df, :occupation) # problem solved

X = [ones(size(df,1),1) df.age df.race.==1 df.collgrad.==1]
y = df.occupation

function mlogit(alpha, X, d)

    # your turn

    return loglike
end


function normalize(x)
    x / sqrt(x' * x)
end

function findPerpendicularPart(x, y)
    y - x' * y * x / (x' * x)
end

function gramSchmdit(xs...)
    qs = Array{Array{Float64,1}}(undef, length(xs))
    i = 1
    for x ∈ xs
        setindex!(qs, x,  i)
        i += 1
    end
    i = 1
    qsLen = length(qs)
    for x ∈ qs
        if i != 1
            for j in range(i, qsLen, step = 1)
                setindex!(qs, findPerpendicularPart(getindex(qs, i - 1), getindex(qs, j)), j)
            end
        end
        i += 1
    end
    i = 1
    for x ∈ qs
        setindex!(qs, normalize(x),  i)
        i += 1
    end
    return qs
end

a = [1;2;3;0]
b = [2;3;1;0]
c = [3;2;1;0]
d = [0;0;0;1]

qs = gramSchmdit(a, b, c, d)
println(qs)


# # Support vector machine
# ### Support Vector Machine (SVM)
# We are given two sets of points in ${\bf R}^n$, $\{x_1, \ldots, x_N\}$ and $\{y_1, \ldots, y_M\}$, and wish to find a function $f(x) = w^T x - b$ that linearly separates the points, i.e. $f(x_i) \geq 1$ for $i = 1, \ldots, N$ and $f(y_i) \leq -1$ for $i = 1, \ldots, M$. That is, the points are separated by two hyperplanes, $w^T x - b = 1$ and $w^T x - b = -1$.
#
# Perfect linear separation is not always possible, so we seek to minimize the amount that these inequalities are violated. The violation of point $x_i$ is $\text{max} \{1 + b - w^T x_i, 0\}$, and the violation of point $y_i$ is $\text{max} \{1 - b + w^T y_i, 0\}$. We tradeoff the error $\sum_{i=1}^N \text{max} \{1 + b - w^T x_i, 0\} + \sum_{i=1}^M \text{max} \{1 - b + w^T y_i, 0\}$ with the distance between the two hyperplanes, which we want to be large, via minimizing $\|w\|^2$.
#
# We can write this problem as
#
# $$
# \begin{array}{ll}
#     \text{minimize}   & \|w\|^2 + C * (\sum_{i=1}^N \text{max} \{1 + b - w^T x_i, 0\} + \sum_{i=1}^M \text{max} \{1 - b + w^T y_i, 0\})
# \end{array},
# $$
#
# where $w \in {\bf R}^n$ and $b \in {\bf R}$ are our optimization variables.
#
# We can solve the problem as follows.

using Convex, SCS

#-

## Generate data.
n = 2; # dimensionality of data
C = 10; # inverse regularization parameter in the objective
N = 10; # number of positive examples
M = 10; # number of negative examples

using Distributions: MvNormal
## positive data points
pos_data = rand(MvNormal([1.0, 2.0], 1.0), N);
## negative data points
neg_data = rand(MvNormal([-1.0, 2.0], 1.0), M);

#-

function svm(pos_data, neg_data, solver=() -> SCS.Optimizer(verbose=0))
    ## Create variables for the separating hyperplane w'*x = b.
    w = Variable(n)
    b = Variable()
    ## Form the objective.
    obj = sumsquares(w) + C*sum(max(1+b-w'*pos_data, 0)) + C*sum(max(1-b+w'*neg_data, 0))
    ## Form and solve problem.
    problem = minimize(obj)
    solve!(problem, solver)
    return evaluate(w), evaluate(b)
end;

#-

w, b = svm(pos_data, neg_data);

#-

## Plot our results.
using Plots
# Generate the separating hyperplane
line_x = -2:0.1:2;
line_y = (-w[1] * line_x .+ b)/w[2];
# Plot the positive points, negative points, and separating hyperplane.
plot(pos_data[1,:], pos_data[2,:], st=:scatter, label="Positive points")
plot!(neg_data[1,:], neg_data[2,:], st=:scatter, label="Negative points")
plot!(line_x, line_y, label="Separating hyperplane")


# conversion between cartesian and spherical coordinate systems

# spherical coordinates relative to origin

@fastmath function cart2sph(x::Float64, y::Float64, z::Float64)
    xy2 = x*x + y*y
    r = sqrt(xy2 + z*z)  # r
    theta = atan2(sqrt(xy2), z)  # theta (polar angle)
    phi = atan2(y,x)  # phi (azimuthal angle)
    r,theta,phi
end

cart2sph(p::Point3{Float64}) = cart2sph(p[1],p[2],p[3])

# spherical coordinates reference to origin p0=(x0,y0,z0)

cart2sph(x::Float64, y::Float64, z::Float64, x0::Float64, y0::Float64, z0::Float64) = cart2sph(x-x0,y-y0,z-z0)
cart2sph(x::Float64, y::Float64, z::Float64, p0::Point3{Float64}) = cart2sph(x-p0[1],y-p0[2],z-p0[3])
cart2sph(p::Point3{Float64}, p0::Point3{Float64}) = cart2sph(p[1]-p0[1],p[2]-p0[2],p[3]-p0[3])

function cart2sph(dst::AbstractArray{Point3{Float64}}, c::AbstractArray{Point3{Float64}}, p0::Point3{Float64})
    @assert size(dst) == size(c)
    for i in eachindex(c)
        dst[i] = cart2sph(c[i],p0)
    end
end

function cart2sph(c::AbstractArray{Point3{Float64}}, p0::Point3{Float64})
    dst = Array{Point3{Float64}}(size(c))
    cart2sph(dst,c,p0)
    dst
end

@vectorize cart2sph Point3{Float64} Point3{Float64}

@fastmath function sph2cart(r::Float64, θ::Float64, ϕ::Float64)
    x = r*sin(θ)*cos(ϕ)
    y = r*sin(θ)*sin(ϕ)
    z = r*cos(θ)
    x,y,z
end

sph2cart(p::Point3{Float64}) = sph2cart(p[1],p[2],p[3])

@vectorize sph2cart Point3{Float64} Point3{Float64}


# This is the main output file for PS3 Question 1
# We write down matrices T, Z, Q, H of each process
# α_{t} = T * α_{t-1} + η_{t} ~ N(0, Q)
# y_{t} = Z * α_{t} + ϵ_{t} ~ N(0, H)

## Packages
using Random
using Distributions
using Plots
using Optim
using ForwardDiff
using LinearAlgebra

## Files
include("KalmanFilter.jl")

rng = Random.seed!(1234);
n = 100; # number of periods

############################## Part 1 (AR(1)) #############################
## Question 1.1: AR(1) process (x_{t} = ρ * x_{t-1} + ϵ_{t})
# Parameters
ρ_AR1 = 0.6;
σ_AR1 = 0.5;

# Simulate AR(1) process
y_AR1 = repeat([reshape([0.0], 1, 1)], n); # [0.0] is a Vector{Float64}, 'reshape' changes it to Matrix{Float64}
[y_AR1[i] = [ρ_AR1] * y_AR1[i - 1] + σ_AR1 * randn(rng, 1) for i in 2:n];

## Getting Matrices to apply kalman filter
T_AR1 = reshape([ρ_AR1], 1, 1);
Z_AR1 = reshape([1.0], 1, 1);
Q_AR1 = reshape([σ_AR1^2], 1, 1);
H_AR1 = reshape([0.0], 1, 1);
α0_AR1 = reshape(mean(y_AR1), 1, 1);

## Applying kalman_filter
αhat_AR1, v_AR1, yhat_AR1, P_AR1, F_AR1, G_AR1, K_AR1  = KalmanFilter(T_AR1, Z_AR1, Q_AR1, H_AR1, n, α0_AR1, y_AR1)

## Plots
vec_y_AR1 = [y_AR1[i][1, 1] for i in 1:n]
vec_yhat_AR1 = [yhat_AR1[i][1, 1] for i in 2:n]

plot(vec_y_AR1, xlabel = " Time ", label =" Simulated AR(1) ", legend=:bottomright)
plot!(vec_yhat_AR1, label = "Estimate from Kalman Filter")
savefig("figs/AR1.png")

## Writing the likelhood function
# For the AR1 process we need to estimate two Parameters ρ and σ:
function log_L_AR1(θ::Vector)  # θ = [ρ, log_σ]
    ρ1 = θ[1]
    σ1 = exp(θ[2])
    T1 = reshape([ρ1], 1, 1)
    Q1 = reshape([σ1^2], 1, 1)

    v1 = [KalmanFilter(T1, Z_AR1, Q1, H_AR1, n, α0_AR1, y_AR1)[2][i][1] for i in 1:n]
    F1 = [KalmanFilter(T1, Z_AR1, Q1, H_AR1, n, α0_AR1, y_AR1)[5][i][1] for i in 1:n]
    Ft = abs.(det.(vec(F1)))

    return sum([-0.5 * log(Ft[i]) - 0.5 * v1[i]' * inv(Ft[i]) * v1[i] for i in 1:n])
end

## Optimization
res_AR1 = optimize(θ->-log_L_AR1(θ), [-0.6, -0.69])
ρ_hat_AR1, σ_hat_AR1 = Optim.minimizer(res_AR1)[1], exp(Optim.minimizer(res_AR1)[2]) # (0.61, 0.48)

############################## Part 2 (AR(2)) #############################
## Question 1.2: AR(2) process (x_{t} = ρ1 * x_{t-1} + ρ2 * x_{t-2} + ϵ_{t})
# Parameters
ρ1_AR2 = 0.3
ρ2_AR2 = 0.4
σ_AR2 = 0.5

# Simulate AR(2) process
y_AR2 = repeat([reshape([0.0], 1, 1)], n);
[y_AR2[i] = [ρ1_AR2] * y_AR2[i - 1] + [ρ2_AR2] * y_AR2[i - 2] + σ_AR2 * randn(rng, 1) for i in 3:n];

## Getting Matrices to apply kalman filter
T_AR2 = [ρ1_AR2 ρ2_AR2; 1.0 0.0];
Z_AR2 = [1.0 0.0];
Q_AR2 = [σ_AR2^2 0.0; 0.0 0.0];
H_AR2 = reshape([0.0], 1, 1);
α0_AR2 = [mean(y_AR2); mean(y_AR2)];

αhat_AR2, v_AR2, yhat_AR2, P_AR2, F_AR2, G_AR2, K_AR2 = KalmanFilter(T_AR2, Z_AR2, Q_AR2, H_AR2, n, α0_AR2, y_AR2)

vec_y_AR2 = [y_AR2[i][1, 1] for i in 3:n]
vec_y_hat_AR2 = [yhat_AR2[i][1, 1] for i in 3:n]

plot(vec_y_AR2, xlabel = " Time ", label =" Simulated AR(2) ", legend=:bottomright)
plot!(vec_y_hat_AR2, label = "Estimate from Kalman Filter")
savefig("figs/AR2.png")

## Writing the likelhood function
# For the AR2 process we need to estimate three Parameters ρ1, ρ2, and σ:
function log_L_AR2(θ::Vector)  # θ = [ρ1, ρ2, log_σ]
    ρ1 = θ[1]
    ρ2 = θ[2]
    σ1 = exp(θ[3])
    T1 = [ρ1 ρ2; 1.0 0.0]
    Q1 = [σ1^2 0.0; 0.0 0.0]

    v1 = [KalmanFilter(T1, Z_AR2, Q1, H_AR2, n, α0_AR2, y_AR2)[2][i][1] for i in 1:n]
    F1 = [KalmanFilter(T1, Z_AR2, Q1, H_AR2, n, α0_AR2, y_AR2)[5][i][1] for i in 1:n]
    Ft = abs.(det.(vec(F1)))

    return sum([-0.5 * log(Ft[i]) - 0.5 * v1[i]' * inv(Ft[i]) * v1[i] for i in 3:n])
end

## Optimization
res_AR2 = optimize(θ->-log_L_AR2(θ), [0.3, 0.7, -0.6])
ρ1_hat_AR2,ρ2_hat_AR2,σ_hat_AR2 = Optim.minimizer(res_AR2)[1],Optim.minimizer(res_AR2)[2],exp(Optim.minimizer(res_AR2)[3])
# (0.26, 0.48, 0.54)

############################## Part 3 (MA(1)) #############################
## Question 1.3: MA(1) process (x_{t} = ϵ_{t} + ρ * ϵ_{t-1})
# Parameters
ρ_MA1 = 0.7
σ_MA1 = 0.8

# Simulate MA(1) process
ϵ_MA1 = rand(Normal(0, σ_MA1), n)
y_MA1 = repeat([reshape([0.0], 1, 1)], n);
[y_MA1[i] = reshape([ϵ_MA1[i]], 1,1) + [ρ_MA1]*ϵ_MA1[i-1] for i in 2:n]

## Getting Matrices to apply kalman filter
T_MA1 = [0.0 0.0; 1.0 0.0]
Z_MA1 = [1.0 ρ_MA1]
Q_MA1 = [σ_MA1^2 0.0; 0.0 0.0]
H_MA1 = reshape([0.0], 1, 1)

α0_MA1 = reshape([0.0; 0.0],2,1);
αhat_MA1, v_MA1, yhat_MA1, P_MA1, F_MA1, G_MA1, K_MA1 = KalmanFilter(T_MA1, Z_MA1, Q_MA1, H_MA1, n, α0_MA1, y_MA1)

vec_y_MA1 = [y_MA1[i][1, 1] for i in 2:n]
vec_y_hat_MA1= [yhat_MA1[i][1, 1] for i in 2:n]

plot(vec_y_MA1, xlabel = " Time ", label =" Simulated MA(1) ", legend=:bottomright)
plot!(vec_y_hat_MA1, label = "Estimate from Kalman Filter")
savefig("figs/MA1.png")

## Writing the likelhood function
# For the MA1 process we need to estimate two Parameters ρ and σ:
function log_L_MA1(θ::Vector)  # θ = [ρ, log_σ]
    ρ1 = θ[1]
    σ1 = exp(θ[2])
    Z1 = [1.0 ρ1]
    Q1 = [σ1^2 0.0; 0.0 0.0]

    v1 = [KalmanFilter(T_MA1, Z1, Q1, H_MA1, n, α0_MA1, y_MA1)[2][i][1] for i in 1:n]
    F1 = [KalmanFilter(T_MA1, Z1, Q1, H_MA1, n, α0_MA1, y_MA1)[5][i][1] for i in 1:n]
    Ft = abs.(det.(vec(F1)))

    return sum([-0.5 * log(Ft[i]) - 0.5 * v1[i]' * inv(Ft[i]) * v1[i] for i in 2:n])
end

## Optimization
res_MA1 = optimize(θ->-log_L_MA1(θ), [0.6, -0.6])
ρ_hat_MA1,σ_hat_MA1 = Optim.minimizer(res_MA1)[1],exp(Optim.minimizer(res_MA1)[2]) # (0.61, 0.73)

############################## Part 4 (Random Walk) #############################
## Question 1.4: Random Walk process
# Parameters
σ_ϵ = 0.8
σ_η = 0.6

# Simulate MA(1) process
ϵ_RW = rand(Normal(0, σ_ϵ), n)
η_RW = rand(Normal(0, σ_η), n)

μ_RW = repeat([reshape([0.0], 1, 1)], n)
[μ_RW[i] = μ_RW[i-1] + reshape([η_RW[i]], 1,1) for i in 2:n]

y_RW = repeat([reshape([0.0], 1, 1)], n)
[y_RW[i] = μ_RW[i] + reshape([ϵ_RW[i]], 1,1) for i in 1:n]

## Getting Matrices to apply kalman filter
T_RW = reshape([1.0], 1, 1)
Z_RW = copy(T_RW)
Q_RW = reshape([σ_η^2], 1, 1)
H_RW = reshape([σ_ϵ^2], 1, 1)

α0_RW = reshape([0.0],1,1);
αhat_RW, v_RW, yhat_RW, P_RW, F_RW, G_RW, K_RW = KalmanFilter(T_RW, Z_RW, Q_RW, H_RW, n, α0_RW, y_RW)

vec_y_RW = [y_RW[i][1, 1] for i in 2:n]
vec_y_hat_RW= [yhat_RW[i][1, 1] for i in 2:n]

plot(vec_y_RW, xlabel = " Time ", label =" Simulated Random Walk ", legend=:bottomright)
plot!(vec_y_hat_RW, label = "Estimate from Kalman Filter")
savefig("figs/RW.png")

## Writing the likelhood function
# For the Random Walk process we need to estimate two Parameters σ_ϵ and σ_η:
function log_L_RW(θ::Vector)  # θ = [log_σ_ϵ, log_σ_η]
    σ_ϵ = exp(θ[1])
    σ_η = exp(θ[2])
    Q1 = reshape([σ_η^2], 1, 1)
    H1 = reshape([σ_ϵ^2], 1, 1)

    v1 = [KalmanFilter(T_RW, Z_RW, Q1, H1, n, α0_RW, y_RW)[2][i][1] for i in 1:n]
    F1 = [KalmanFilter(T_RW, Z_RW, Q1, H1, n, α0_RW, y_RW)[5][i][1] for i in 1:n]
    Ft = abs.(det.(vec(F1)))

    return sum([-0.5 * log(Ft[i]) - 0.5 * v1[i]' * inv(Ft[i]) * v1[i] for i in 2:n])
end

## Optimization
res_RW = optimize(θ->-log_L_RW(θ), [-0.3, -0.6])
σ_ϵ_hat, σ_η_hat= exp(Optim.minimizer(res_RW)[1]), exp(Optim.minimizer(res_RW)[2]) # (0.70, 0.65)