MuzzammilShah commited on
Commit
73facba
·
verified ·
1 Parent(s): 93142d5

Initial commits for files

Browse files
Files changed (5) hide show
  1. A-main-notebook.ipynb +589 -0
  2. B-main-notebook.ipynb +0 -0
  3. C-main-notebook.ipynb +649 -0
  4. README.md +61 -0
  5. names.txt +0 -0
A-main-notebook.ipynb ADDED
@@ -0,0 +1,589 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "cells": [
3
+ {
4
+ "cell_type": "code",
5
+ "execution_count": 1,
6
+ "metadata": {},
7
+ "outputs": [],
8
+ "source": [
9
+ "import torch\n",
10
+ "import torch.nn.functional as F\n",
11
+ "import matplotlib.pyplot as plt # for making figures\n",
12
+ "%matplotlib inline"
13
+ ]
14
+ },
15
+ {
16
+ "cell_type": "code",
17
+ "execution_count": 2,
18
+ "metadata": {},
19
+ "outputs": [
20
+ {
21
+ "data": {
22
+ "text/plain": [
23
+ "['emma', 'olivia', 'ava', 'isabella', 'sophia', 'charlotte', 'mia', 'amelia']"
24
+ ]
25
+ },
26
+ "execution_count": 2,
27
+ "metadata": {},
28
+ "output_type": "execute_result"
29
+ }
30
+ ],
31
+ "source": [
32
+ "# read in all the words\n",
33
+ "words = open('names.txt', 'r').read().splitlines()\n",
34
+ "words[:8]"
35
+ ]
36
+ },
37
+ {
38
+ "cell_type": "code",
39
+ "execution_count": 3,
40
+ "metadata": {},
41
+ "outputs": [
42
+ {
43
+ "data": {
44
+ "text/plain": [
45
+ "32033"
46
+ ]
47
+ },
48
+ "execution_count": 3,
49
+ "metadata": {},
50
+ "output_type": "execute_result"
51
+ }
52
+ ],
53
+ "source": [
54
+ "len(words)"
55
+ ]
56
+ },
57
+ {
58
+ "cell_type": "code",
59
+ "execution_count": 3,
60
+ "metadata": {},
61
+ "outputs": [
62
+ {
63
+ "name": "stdout",
64
+ "output_type": "stream",
65
+ "text": [
66
+ "{1: 'a', 2: 'b', 3: 'c', 4: 'd', 5: 'e', 6: 'f', 7: 'g', 8: 'h', 9: 'i', 10: 'j', 11: 'k', 12: 'l', 13: 'm', 14: 'n', 15: 'o', 16: 'p', 17: 'q', 18: 'r', 19: 's', 20: 't', 21: 'u', 22: 'v', 23: 'w', 24: 'x', 25: 'y', 26: 'z', 0: '.'}\n"
67
+ ]
68
+ }
69
+ ],
70
+ "source": [
71
+ "# build the vocabulary of characters and mappings to/from integers\n",
72
+ "chars = sorted(list(set(''.join(words))))\n",
73
+ "stoi = {s:i+1 for i,s in enumerate(chars)}\n",
74
+ "stoi['.'] = 0\n",
75
+ "itos = {i:s for s,i in stoi.items()}\n",
76
+ "print(itos)"
77
+ ]
78
+ },
79
+ {
80
+ "cell_type": "code",
81
+ "execution_count": 13,
82
+ "metadata": {},
83
+ "outputs": [
84
+ {
85
+ "name": "stdout",
86
+ "output_type": "stream",
87
+ "text": [
88
+ "... ---> e\n",
89
+ "..e ---> m\n",
90
+ ".em ---> m\n",
91
+ "emm ---> a\n",
92
+ "mma ---> .\n",
93
+ "... ---> o\n",
94
+ "..o ---> l\n",
95
+ ".ol ---> i\n",
96
+ "oli ---> v\n",
97
+ "liv ---> i\n",
98
+ "ivi ---> a\n",
99
+ "via ---> .\n",
100
+ "... ---> a\n",
101
+ "..a ---> v\n",
102
+ ".av ---> a\n",
103
+ "ava ---> .\n",
104
+ "... ---> i\n",
105
+ "..i ---> s\n",
106
+ ".is ---> a\n",
107
+ "isa ---> b\n",
108
+ "sab ---> e\n",
109
+ "abe ---> l\n",
110
+ "bel ---> l\n",
111
+ "ell ---> a\n",
112
+ "lla ---> .\n",
113
+ "... ---> s\n",
114
+ "..s ---> o\n",
115
+ ".so ---> p\n",
116
+ "sop ---> h\n",
117
+ "oph ---> i\n",
118
+ "phi ---> a\n",
119
+ "hia ---> .\n"
120
+ ]
121
+ }
122
+ ],
123
+ "source": [
124
+ "# build the dataset\n",
125
+ "\n",
126
+ "block_size = 3 # context length: how many characters do we take to predict the next one?\n",
127
+ "X, Y = [], []\n",
128
+ "for w in words[:5]:\n",
129
+ " \n",
130
+ " #print(w)\n",
131
+ " context = [0] * block_size\n",
132
+ " for ch in w + '.':\n",
133
+ " ix = stoi[ch]\n",
134
+ " X.append(context)\n",
135
+ " Y.append(ix)\n",
136
+ " print(''.join(itos[i] for i in context), '--->', itos[ix])\n",
137
+ " context = context[1:] + [ix] # crop and append\n",
138
+ " \n",
139
+ "X = torch.tensor(X)\n",
140
+ "Y = torch.tensor(Y)"
141
+ ]
142
+ },
143
+ {
144
+ "cell_type": "code",
145
+ "execution_count": 6,
146
+ "metadata": {},
147
+ "outputs": [
148
+ {
149
+ "data": {
150
+ "text/plain": [
151
+ "(torch.Size([32, 3]), torch.int64, torch.Size([32]), torch.int64)"
152
+ ]
153
+ },
154
+ "execution_count": 6,
155
+ "metadata": {},
156
+ "output_type": "execute_result"
157
+ }
158
+ ],
159
+ "source": [
160
+ "X.shape, X.dtype, Y.shape, Y.dtype"
161
+ ]
162
+ },
163
+ {
164
+ "cell_type": "markdown",
165
+ "metadata": {},
166
+ "source": [
167
+ "So our dataset looks like this^ \\\n",
168
+ "\\\n",
169
+ "So, for each of those above 5 words, \\\n",
170
+ "`torch.Size([32, 3])` we have created a dataset of 32 examples and each input of the neural net is 3 integers => X \\\n",
171
+ "`torch.Size([32])` and these are the labels (single row, 32 values) => Y"
172
+ ]
173
+ },
174
+ {
175
+ "cell_type": "code",
176
+ "execution_count": 13,
177
+ "metadata": {},
178
+ "outputs": [
179
+ {
180
+ "data": {
181
+ "text/plain": [
182
+ "tensor([[ 0, 0, 0],\n",
183
+ " [ 0, 0, 5],\n",
184
+ " [ 0, 5, 13],\n",
185
+ " [ 5, 13, 13],\n",
186
+ " [13, 13, 1],\n",
187
+ " [ 0, 0, 0],\n",
188
+ " [ 0, 0, 15],\n",
189
+ " [ 0, 15, 12],\n",
190
+ " [15, 12, 9],\n",
191
+ " [12, 9, 22],\n",
192
+ " [ 9, 22, 9],\n",
193
+ " [22, 9, 1],\n",
194
+ " [ 0, 0, 0],\n",
195
+ " [ 0, 0, 1],\n",
196
+ " [ 0, 1, 22],\n",
197
+ " [ 1, 22, 1],\n",
198
+ " [ 0, 0, 0],\n",
199
+ " [ 0, 0, 9],\n",
200
+ " [ 0, 9, 19],\n",
201
+ " [ 9, 19, 1],\n",
202
+ " [19, 1, 2],\n",
203
+ " [ 1, 2, 5],\n",
204
+ " [ 2, 5, 12],\n",
205
+ " [ 5, 12, 12],\n",
206
+ " [12, 12, 1],\n",
207
+ " [ 0, 0, 0],\n",
208
+ " [ 0, 0, 19],\n",
209
+ " [ 0, 19, 15],\n",
210
+ " [19, 15, 16],\n",
211
+ " [15, 16, 8],\n",
212
+ " [16, 8, 9],\n",
213
+ " [ 8, 9, 1]])"
214
+ ]
215
+ },
216
+ "execution_count": 13,
217
+ "metadata": {},
218
+ "output_type": "execute_result"
219
+ }
220
+ ],
221
+ "source": [
222
+ "X"
223
+ ]
224
+ },
225
+ {
226
+ "cell_type": "code",
227
+ "execution_count": 14,
228
+ "metadata": {},
229
+ "outputs": [
230
+ {
231
+ "data": {
232
+ "text/plain": [
233
+ "tensor([ 5, 13, 13, 1, 0, 15, 12, 9, 22, 9, 1, 0, 1, 22, 1, 0, 9, 19,\n",
234
+ " 1, 2, 5, 12, 12, 1, 0, 19, 15, 16, 8, 9, 1, 0])"
235
+ ]
236
+ },
237
+ "execution_count": 14,
238
+ "metadata": {},
239
+ "output_type": "execute_result"
240
+ }
241
+ ],
242
+ "source": [
243
+ "Y"
244
+ ]
245
+ },
246
+ {
247
+ "cell_type": "code",
248
+ "execution_count": 8,
249
+ "metadata": {},
250
+ "outputs": [],
251
+ "source": [
252
+ "C = torch.rand((27, 2))"
253
+ ]
254
+ },
255
+ {
256
+ "cell_type": "code",
257
+ "execution_count": 9,
258
+ "metadata": {},
259
+ "outputs": [
260
+ {
261
+ "data": {
262
+ "text/plain": [
263
+ "torch.Size([32, 3, 2])"
264
+ ]
265
+ },
266
+ "execution_count": 9,
267
+ "metadata": {},
268
+ "output_type": "execute_result"
269
+ }
270
+ ],
271
+ "source": [
272
+ "emb = C[X]\n",
273
+ "\n",
274
+ "emb.shape"
275
+ ]
276
+ },
277
+ {
278
+ "cell_type": "markdown",
279
+ "metadata": {},
280
+ "source": [
281
+ "(PyTorch indexing is awesome) \\\n",
282
+ "\\\n",
283
+ "To index simultaneously all the elements of X, We simply do C[X]"
284
+ ]
285
+ },
286
+ {
287
+ "cell_type": "code",
288
+ "execution_count": 10,
289
+ "metadata": {},
290
+ "outputs": [],
291
+ "source": [
292
+ "W1 = torch.randn((6, 100))\n",
293
+ "b1 = torch.rand(100)"
294
+ ]
295
+ },
296
+ {
297
+ "cell_type": "code",
298
+ "execution_count": 11,
299
+ "metadata": {},
300
+ "outputs": [],
301
+ "source": [
302
+ "h = torch.tanh(emb.view(-1, 6) @ W1 + b1)"
303
+ ]
304
+ },
305
+ {
306
+ "cell_type": "code",
307
+ "execution_count": 12,
308
+ "metadata": {},
309
+ "outputs": [
310
+ {
311
+ "data": {
312
+ "text/plain": [
313
+ "tensor([[ 0.9910, 0.8405, 0.4715, ..., 0.9999, 0.8814, 0.9998],\n",
314
+ " [ 0.9763, 0.9163, 0.3350, ..., 0.9991, 0.8249, 0.9992],\n",
315
+ " [ 0.9791, 0.8450, -0.0272, ..., 0.9997, 0.9230, 0.9997],\n",
316
+ " ...,\n",
317
+ " [ 0.8995, 0.6590, 0.4667, ..., 0.9995, -0.4144, 0.9988],\n",
318
+ " [ 0.9777, 0.7397, 0.2623, ..., 0.9999, 0.9593, 0.9999],\n",
319
+ " [ 0.9402, 0.7154, 0.2493, ..., 0.9980, -0.6247, 0.9979]])"
320
+ ]
321
+ },
322
+ "execution_count": 12,
323
+ "metadata": {},
324
+ "output_type": "execute_result"
325
+ }
326
+ ],
327
+ "source": [
328
+ "h"
329
+ ]
330
+ },
331
+ {
332
+ "cell_type": "code",
333
+ "execution_count": 13,
334
+ "metadata": {},
335
+ "outputs": [
336
+ {
337
+ "data": {
338
+ "text/plain": [
339
+ "torch.Size([32, 100])"
340
+ ]
341
+ },
342
+ "execution_count": 13,
343
+ "metadata": {},
344
+ "output_type": "execute_result"
345
+ }
346
+ ],
347
+ "source": [
348
+ "h.shape"
349
+ ]
350
+ },
351
+ {
352
+ "cell_type": "markdown",
353
+ "metadata": {},
354
+ "source": [
355
+ "Hidden layer is now made^"
356
+ ]
357
+ },
358
+ {
359
+ "cell_type": "code",
360
+ "execution_count": 15,
361
+ "metadata": {},
362
+ "outputs": [],
363
+ "source": [
364
+ "W2 = torch.randn((100, 27))\n",
365
+ "b2 = torch.rand(27)"
366
+ ]
367
+ },
368
+ {
369
+ "cell_type": "code",
370
+ "execution_count": 16,
371
+ "metadata": {},
372
+ "outputs": [],
373
+ "source": [
374
+ "logits = h @ W2 + b2"
375
+ ]
376
+ },
377
+ {
378
+ "cell_type": "code",
379
+ "execution_count": 17,
380
+ "metadata": {},
381
+ "outputs": [
382
+ {
383
+ "data": {
384
+ "text/plain": [
385
+ "torch.Size([32, 27])"
386
+ ]
387
+ },
388
+ "execution_count": 17,
389
+ "metadata": {},
390
+ "output_type": "execute_result"
391
+ }
392
+ ],
393
+ "source": [
394
+ "logits.shape"
395
+ ]
396
+ },
397
+ {
398
+ "cell_type": "code",
399
+ "execution_count": 18,
400
+ "metadata": {},
401
+ "outputs": [],
402
+ "source": [
403
+ "counts = logits.exp()"
404
+ ]
405
+ },
406
+ {
407
+ "cell_type": "code",
408
+ "execution_count": 19,
409
+ "metadata": {},
410
+ "outputs": [],
411
+ "source": [
412
+ "prob = counts / counts.sum(1, keepdims=True)"
413
+ ]
414
+ },
415
+ {
416
+ "cell_type": "code",
417
+ "execution_count": 21,
418
+ "metadata": {},
419
+ "outputs": [
420
+ {
421
+ "data": {
422
+ "text/plain": [
423
+ "torch.Size([32, 27])"
424
+ ]
425
+ },
426
+ "execution_count": 21,
427
+ "metadata": {},
428
+ "output_type": "execute_result"
429
+ }
430
+ ],
431
+ "source": [
432
+ "prob.shape"
433
+ ]
434
+ },
435
+ {
436
+ "cell_type": "code",
437
+ "execution_count": 22,
438
+ "metadata": {},
439
+ "outputs": [
440
+ {
441
+ "data": {
442
+ "text/plain": [
443
+ "tensor(13.4043)"
444
+ ]
445
+ },
446
+ "execution_count": 22,
447
+ "metadata": {},
448
+ "output_type": "execute_result"
449
+ }
450
+ ],
451
+ "source": [
452
+ "loss = -prob[torch.arange(32), Y].log().mean()\n",
453
+ "loss"
454
+ ]
455
+ },
456
+ {
457
+ "cell_type": "markdown",
458
+ "metadata": {},
459
+ "source": [
460
+ "We've made the final output layer^ \\\n",
461
+ "Found the loss function value, which we have to reduce"
462
+ ]
463
+ },
464
+ {
465
+ "cell_type": "markdown",
466
+ "metadata": {},
467
+ "source": [
468
+ "---------------------"
469
+ ]
470
+ },
471
+ {
472
+ "cell_type": "markdown",
473
+ "metadata": {},
474
+ "source": [
475
+ "### **Summarising what we've done so far to make this more respectable :)**"
476
+ ]
477
+ },
478
+ {
479
+ "cell_type": "code",
480
+ "execution_count": 14,
481
+ "metadata": {},
482
+ "outputs": [
483
+ {
484
+ "data": {
485
+ "text/plain": [
486
+ "(torch.Size([32, 3]), torch.Size([32]))"
487
+ ]
488
+ },
489
+ "execution_count": 14,
490
+ "metadata": {},
491
+ "output_type": "execute_result"
492
+ }
493
+ ],
494
+ "source": [
495
+ "#Run the first 5 cells and then start from here\n",
496
+ "X.shape, Y.shape #dataset"
497
+ ]
498
+ },
499
+ {
500
+ "cell_type": "code",
501
+ "execution_count": 15,
502
+ "metadata": {},
503
+ "outputs": [],
504
+ "source": [
505
+ "g = torch.Generator().manual_seed(2147483647) #For consistency ofcourse, to keep the same values as andrej\n",
506
+ "C = torch.randn((27,2), generator=g)\n",
507
+ "W1 = torch.rand((6, 100), generator=g)\n",
508
+ "b1 = torch.rand(100, generator=g)\n",
509
+ "W2 = torch.rand((100, 27), generator=g)\n",
510
+ "b2 = torch.rand(27, generator=g)\n",
511
+ "parameters = [C, W1, b1, W2, b2]"
512
+ ]
513
+ },
514
+ {
515
+ "cell_type": "code",
516
+ "execution_count": 16,
517
+ "metadata": {},
518
+ "outputs": [
519
+ {
520
+ "data": {
521
+ "text/plain": [
522
+ "3481"
523
+ ]
524
+ },
525
+ "execution_count": 16,
526
+ "metadata": {},
527
+ "output_type": "execute_result"
528
+ }
529
+ ],
530
+ "source": [
531
+ "sum(p.nelement() for p in parameters) #to check number of parameters in total"
532
+ ]
533
+ },
534
+ {
535
+ "cell_type": "code",
536
+ "execution_count": 17,
537
+ "metadata": {},
538
+ "outputs": [
539
+ {
540
+ "data": {
541
+ "text/plain": [
542
+ "tensor(6.4365)"
543
+ ]
544
+ },
545
+ "execution_count": 17,
546
+ "metadata": {},
547
+ "output_type": "execute_result"
548
+ }
549
+ ],
550
+ "source": [
551
+ "emb = C[X]\n",
552
+ "h = torch.tanh(emb.view(-1,6) @ W1 + b1)\n",
553
+ "logits = h @ W2 + b2\n",
554
+ "counts = logits.exp()\n",
555
+ "prob = counts / counts.sum(1, keepdims=True)\n",
556
+ "loss = - prob[torch.arange(32), Y].log().mean()\n",
557
+ "loss"
558
+ ]
559
+ },
560
+ {
561
+ "cell_type": "markdown",
562
+ "metadata": {},
563
+ "source": [
564
+ "--------------"
565
+ ]
566
+ }
567
+ ],
568
+ "metadata": {
569
+ "kernelspec": {
570
+ "display_name": "venv",
571
+ "language": "python",
572
+ "name": "python3"
573
+ },
574
+ "language_info": {
575
+ "codemirror_mode": {
576
+ "name": "ipython",
577
+ "version": 3
578
+ },
579
+ "file_extension": ".py",
580
+ "mimetype": "text/x-python",
581
+ "name": "python",
582
+ "nbconvert_exporter": "python",
583
+ "pygments_lexer": "ipython3",
584
+ "version": "3.10.0"
585
+ }
586
+ },
587
+ "nbformat": 4,
588
+ "nbformat_minor": 2
589
+ }
B-main-notebook.ipynb ADDED
The diff for this file is too large to render. See raw diff
 
C-main-notebook.ipynb ADDED
@@ -0,0 +1,649 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "cells": [
3
+ {
4
+ "cell_type": "code",
5
+ "execution_count": 1,
6
+ "metadata": {},
7
+ "outputs": [],
8
+ "source": [
9
+ "import torch\n",
10
+ "import torch.nn.functional as F\n",
11
+ "import matplotlib.pyplot as plt # for making figures\n",
12
+ "%matplotlib inline"
13
+ ]
14
+ },
15
+ {
16
+ "cell_type": "code",
17
+ "execution_count": 2,
18
+ "metadata": {},
19
+ "outputs": [],
20
+ "source": [
21
+ "# read in all the words\n",
22
+ "words = open('names.txt', 'r').read().splitlines()\n",
23
+ "\n",
24
+ "\n",
25
+ "# build the vocabulary of characters and mappings to/from integers\n",
26
+ "chars = sorted(list(set(''.join(words))))\n",
27
+ "stoi = {s:i+1 for i,s in enumerate(chars)}\n",
28
+ "stoi['.'] = 0\n",
29
+ "itos = {i:s for s,i in stoi.items()}"
30
+ ]
31
+ },
32
+ {
33
+ "cell_type": "code",
34
+ "execution_count": 3,
35
+ "metadata": {},
36
+ "outputs": [
37
+ {
38
+ "name": "stdout",
39
+ "output_type": "stream",
40
+ "text": [
41
+ "torch.Size([182625, 3]) torch.Size([182625])\n",
42
+ "torch.Size([22655, 3]) torch.Size([22655])\n",
43
+ "torch.Size([22866, 3]) torch.Size([22866])\n"
44
+ ]
45
+ }
46
+ ],
47
+ "source": [
48
+ "# build the dataset\n",
49
+ "block_size = 3 # context length: how many characters do we take to predict the next one?\n",
50
+ "\n",
51
+ "def build_dataset(words): \n",
52
+ " X, Y = [], []\n",
53
+ " for w in words:\n",
54
+ "\n",
55
+ " #print(w)\n",
56
+ " context = [0] * block_size\n",
57
+ " for ch in w + '.':\n",
58
+ " ix = stoi[ch]\n",
59
+ " X.append(context)\n",
60
+ " Y.append(ix)\n",
61
+ " #print(''.join(itos[i] for i in context), '--->', itos[ix])\n",
62
+ " context = context[1:] + [ix] # crop and append\n",
63
+ "\n",
64
+ " X = torch.tensor(X)\n",
65
+ " Y = torch.tensor(Y)\n",
66
+ " print(X.shape, Y.shape)\n",
67
+ " return X, Y\n",
68
+ "\n",
69
+ "import random\n",
70
+ "random.seed(42)\n",
71
+ "random.shuffle(words)\n",
72
+ "n1 = int(0.8*len(words))\n",
73
+ "n2 = int(0.9*len(words))\n",
74
+ "\n",
75
+ "Xtr, Ytr = build_dataset(words[:n1])\n",
76
+ "Xdev, Ydev = build_dataset(words[n1:n2])\n",
77
+ "Xte, Yte = build_dataset(words[n2:])"
78
+ ]
79
+ },
80
+ {
81
+ "cell_type": "code",
82
+ "execution_count": 4,
83
+ "metadata": {},
84
+ "outputs": [
85
+ {
86
+ "data": {
87
+ "text/plain": [
88
+ "(torch.Size([182625, 3]), torch.Size([182625]))"
89
+ ]
90
+ },
91
+ "execution_count": 4,
92
+ "metadata": {},
93
+ "output_type": "execute_result"
94
+ }
95
+ ],
96
+ "source": [
97
+ "Xtr.shape, Ytr.shape #dataset"
98
+ ]
99
+ },
100
+ {
101
+ "cell_type": "code",
102
+ "execution_count": 20,
103
+ "metadata": {},
104
+ "outputs": [],
105
+ "source": [
106
+ "g = torch.Generator().manual_seed(2147483647) #For consistency ofcourse, to keep the same values as andrej\n",
107
+ "C = torch.randn((27,10), generator=g)\n",
108
+ "W1 = torch.rand((30, 300), generator=g)\n",
109
+ "b1 = torch.rand(300, generator=g)\n",
110
+ "W2 = torch.rand((300, 27), generator=g)\n",
111
+ "b2 = torch.rand(27, generator=g)\n",
112
+ "parameters = [C, W1, b1, W2, b2]"
113
+ ]
114
+ },
115
+ {
116
+ "cell_type": "code",
117
+ "execution_count": 21,
118
+ "metadata": {},
119
+ "outputs": [
120
+ {
121
+ "data": {
122
+ "text/plain": [
123
+ "17697"
124
+ ]
125
+ },
126
+ "execution_count": 21,
127
+ "metadata": {},
128
+ "output_type": "execute_result"
129
+ }
130
+ ],
131
+ "source": [
132
+ "sum(p.nelement() for p in parameters) # number of parameters in total"
133
+ ]
134
+ },
135
+ {
136
+ "cell_type": "code",
137
+ "execution_count": 22,
138
+ "metadata": {},
139
+ "outputs": [],
140
+ "source": [
141
+ "for p in parameters:\n",
142
+ " p.requires_grad = True"
143
+ ]
144
+ },
145
+ {
146
+ "cell_type": "code",
147
+ "execution_count": 8,
148
+ "metadata": {},
149
+ "outputs": [],
150
+ "source": [
151
+ "\n",
152
+ "lre = torch.linspace(-3, 0, 1000)\n",
153
+ "lrs = 10**lre"
154
+ ]
155
+ },
156
+ {
157
+ "cell_type": "code",
158
+ "execution_count": 30,
159
+ "metadata": {},
160
+ "outputs": [],
161
+ "source": [
162
+ "lri = []\n",
163
+ "lossi = []\n",
164
+ "stepi = []\n",
165
+ "\n",
166
+ "for i in range(40000):\n",
167
+ "\n",
168
+ " #Minibatch\n",
169
+ " xi = torch.randint(0, Xtr.shape[0], (32,))\n",
170
+ "\n",
171
+ " #forward pass\n",
172
+ " emb = C[Xtr[xi]]\n",
173
+ " h = torch.tanh(emb.view(-1,30) @ W1 + b1)\n",
174
+ " logits = h @ W2 + b2\n",
175
+ " loss = F.cross_entropy(logits, Ytr[xi])\n",
176
+ " #print(loss.item())\n",
177
+ "\n",
178
+ " #backward pass\n",
179
+ " for p in parameters:\n",
180
+ " p.grad = None\n",
181
+ " loss.backward()\n",
182
+ "\n",
183
+ " #update\n",
184
+ " #lr = lrs[i]\n",
185
+ " lr = 0.01\n",
186
+ " for p in parameters:\n",
187
+ " p.data += -lr * p.grad\n",
188
+ "\n",
189
+ " #keeping track\n",
190
+ " #lri.append(lr)\n",
191
+ " stepi.append(i)\n",
192
+ " lossi.append(loss.item())\n",
193
+ "\n",
194
+ "#print(loss.item())"
195
+ ]
196
+ },
197
+ {
198
+ "cell_type": "markdown",
199
+ "metadata": {},
200
+ "source": [
201
+ "The above cell will take a couple of seconds to run. Training a neural net can take a while, but luckily this is a very small neural network."
202
+ ]
203
+ },
204
+ {
205
+ "cell_type": "markdown",
206
+ "metadata": {},
207
+ "source": [
208
+ "**Evaluation:**"
209
+ ]
210
+ },
211
+ {
212
+ "cell_type": "code",
213
+ "execution_count": 31,
214
+ "metadata": {},
215
+ "outputs": [
216
+ {
217
+ "data": {
218
+ "text/plain": [
219
+ "tensor(2.1091, grad_fn=<NllLossBackward0>)"
220
+ ]
221
+ },
222
+ "execution_count": 31,
223
+ "metadata": {},
224
+ "output_type": "execute_result"
225
+ }
226
+ ],
227
+ "source": [
228
+ "emb = C[Xdev]\n",
229
+ "h = torch.tanh(emb.view(-1,30) @ W1 + b1)\n",
230
+ "logits = h @ W2 + b2\n",
231
+ "devloss = F.cross_entropy(logits, Ydev)\n",
232
+ "devloss"
233
+ ]
234
+ },
235
+ {
236
+ "cell_type": "code",
237
+ "execution_count": 32,
238
+ "metadata": {},
239
+ "outputs": [
240
+ {
241
+ "data": {
242
+ "text/plain": [
243
+ "tensor(2.0482, grad_fn=<NllLossBackward0>)"
244
+ ]
245
+ },
246
+ "execution_count": 32,
247
+ "metadata": {},
248
+ "output_type": "execute_result"
249
+ }
250
+ ],
251
+ "source": [
252
+ "emb = C[Xtr]\n",
253
+ "h = torch.tanh(emb.view(-1,30) @ W1 + b1)\n",
254
+ "logits = h @ W2 + b2\n",
255
+ "trloss = F.cross_entropy(logits, Ytr)\n",
256
+ "trloss"
257
+ ]
258
+ },
259
+ {
260
+ "cell_type": "markdown",
261
+ "metadata": {},
262
+ "source": [
263
+ "Training and Dev loss are almost the same. So we know we are not overfitting. But what it typically means is that the Neural Net is very small, so essentially it is underfitting the data. \\\n",
264
+ "\\\n",
265
+ "Therefore to improve the performance we'll need to increase the size of the neural net."
266
+ ]
267
+ },
268
+ {
269
+ "cell_type": "code",
270
+ "execution_count": 15,
271
+ "metadata": {},
272
+ "outputs": [
273
+ {
274
+ "data": {
275
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAp4AAAKTCAYAAACw6AhNAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAABQSUlEQVR4nO3df3xU9Z3v8feZYRJJSEJCCEoISRDqLxpiE4JYa7UtVN1a7NK01Ltbba16veou0sUf3a5Kt7YVvNWu9VptrdLtskhkla4/WqirohYIiY0RQSGShBB+hJCQn5oMM+f+gRMJmZ/JzJk5yev5ePBo58x3zvlOvoBvPud8v1/DNE1TAAAAQIw54t0BAAAAjA0ETwAAAFiC4AkAAABLEDwBAABgCYInAAAALEHwBAAAgCUIngAAALDEuHh3IBiv16sDBw4oLS1NhmHEuzsAAAA4hWma6urq0tSpU+VwBK9pJnTwPHDggPLy8uLdDQAAAITQ1NSkadOmBW2T0MEzLS1N0okvkp6eHrK92+3Wxo0btXDhQrlcrlh3DyPEeNkL42UvjJe9MF72wngN1tnZqby8vIHcFkxCB0/f7fX09PSwg2dKSorS09P5jWADjJe9MF72wnjZC+NlL4yXf+E8FsnkIgAAAFiC4AkAAABLEDwBAABgCYInAAAALEHwBAAAgCUIngAAALAEwRMAAACWIHgCAADAEgRPAAAAWILgCQAAAEvENHg++uijKioqGtjycv78+XrppZdieUkAAAAkqJgGz2nTpulnP/uZqqurVVVVpS984QtatGiR3n333VheFgAAAAloXCxPfuWVVw56fd999+nRRx/V1q1bdd5558Xy0gAAAEgwMQ2eJ/N4PKqoqFBPT4/mz5/vt01fX5/6+voGXnd2dkqS3G633G53yGv42oTTFvHHeNkL42UvjJe9MF72wngNFsnPwTBN04xhX/TOO+9o/vz5+uijjzRhwgStWbNGV1xxhd+29957r1asWDHk+Jo1a5SSkhLLbgIAAGAYent7dfXVV6ujo0Pp6elB28Y8ePb392vfvn3q6OjQM888o9/85jd67bXXdO655w5p66/imZeXp9bW1pBfRDqRuDdt2qQFCxbI5XJF9Xsg+hgve2G87IXxshfGy14Scbx2HezUs39tVnVju/Ye6Zbba8rlMDRj8gSV5Gfqa+fn6pwzQmep4ejs7FR2dnZYwTPmt9qTkpI0c+ZMSVJJSYm2b9+uX/ziF3rssceGtE1OTlZycvKQ4y6XK6KBjbQ94ovxshfGy14YL3thvOwlEcarobVHt6+vVWV9m5wOQx6vr55oqM8j1R7o1ruHevTkliaVFWZp5eIiFWSnRrUPkfwMLF/H0+v1DqpqAgAAIHIbapq18MHNqm5sl6STQudgvuPVje1a+OBmbahptqyPp4ppxfOuu+7S5ZdfrunTp6urq0tr1qzRq6++qj/96U+xvCwAAMCotqGmWUvX1iiS5yU9XlMemVq6tkaStKg4NyZ9CyamwbOlpUXf/va3dfDgQWVkZKioqEh/+tOftGDBglheFgAAYNSqb+3R8oraiELnyUxJyytqNWfaxKjfdg8lpsHziSeeiOXpAQAAxpw71tfKM8K54R7T1O3ra7XuRv9LXMYKe7UDAADYxDv7O1RZ3xbwec5webymKuvbtKO5I0o9Cw/BEwAAwCaeqW7SOIfh97037rhU3/1swaBjL/7DRVr6pVl+2zsdhiqqmqLdxaAIngAAADZR2dCm4yOsdvp4vKa2N7RH5VzhIngCAADYRF1Ld1TPt6elK6rnC4XgCQAAYANerym3J7obTro9prxRqqCGg+AJAABgAw6HIZfT//OdkuT1SoYx+P1xzuBRz+U05AjwzGgsEDwBAABsYmbOhIDvtfX0aXLaJ1uPT0gep7zMlKDnm5WTFrW+hYPgCQAAYBNlBVlyBqhQ/uWDo/rb83M1tyBTZ01J0//9xpyg6306HYbmFmTGqqt+xXQBeQAAAERPeWmeVm9p9Pve/3v1A+VlpeiJa+eq66Pj+vnG95WXOT7guTxeU+WlebHqql8ETwAAAJuYnZuhssIsVTe2D1lEvrvvuG79z78OOrb+rWa/53E6DJXkZ2p2bkbM+uoPt9oBAABsZOXiIjmNkU0IchqGVi4uilKPwkfwBAAAsJGC7FStKi/ScKOnIWlVeZEKslOj2a2wcKsdAADAZhYV50qSllfUymOaYe3d7nQYchqGVpUXDXzealQ8AQAAbGhRca423naxSvJPzEwPNNvdd7w0P1Mbb7s4bqFTouIJAABgWwXZqVp343ztaO5QRVWTtje0a09Ll9weUy6noVk5aZpbkKny0jzLJxL5Q/AEAACwudm5GYOCpddrWrojUbi41Q4AADDKJGLolAieAAAAsAjBEwAAAJYgeAIAAMASBE8AAABYguAJAAAASxA8AQAAYAmCJwAAACxB8AQAAIAlCJ4AAACwBMETAAAAliB4AgAAwBIETwAAAFiC4AkAAABLEDwBAABgCYInAAAALEHwBAAAgCUIngAAALAEwRMAAACWIHgCAADAEgRPAAAAWILgCQAAAEsQPAEAAGAJgicAAAAsQfAEAACAJQieAAAAsATBEwAAAJYgeAIAAMASBE8AAABYguAJAAAASxA8AQAAYAmCJwAAACxB8AQAAIAlCJ4AAACwBMETAAAAliB4AgBixus1490FAAlkXLw7AAAYPXY0d6iiqkmVDW2qa+mW22PK5TQ0M2eCygqyVF6ap9m5GfHuJoA4IXgCAEasobVHt6+vVWV9m5wOQ56TKp1uj6ldB7u0+3C3Vm9pVFlhllYuLlJBdmocewwgHrjVDgAYkQ01zVr44GZVN7ZL0qDQeTLf8erGdi18cLM21DRb1kcAiYGKJwBg2DbUNGvp2hpF8iSnx2vKI1NL19ZIkhYV58akbwASDxVPAMCw1Lf2aHlFbUSh82SmpOUVtWpo7YlmtwAkMIInAGBY7lhfK485slnrHtPU7etro9QjAImO4AkAiNg7+ztUWd8W8HnOcHm8pirr27SjuSNKPQOQyHjGEwAQsWeqmzTOYei4n+CZmuTUfV/7tBaeN0XdHx3XY5v3asG5U7TzQKd+9PzOIe2dDkMVVU0sswSMAQRPAEDEKhva/IZOSfrhV85VaUGmvre6Sq3dfVq24CydNzVdOw90+m3v8Zra3tAey+4CSBDcagcARKyupdvv8dQkpxZ/Zprue2GX/vLBUe0+3K3lFW/L6TCCnm9PS1csugkgwRA8AQAR8XpNuT3+q53TJ6UoaZxDbzcdGzjW1Xdce48En7nu9phsrwmMAQRPAEBEHA5DLmfwCmakXE5DjhBVUQD2R/AEAERsZs4Ev8f3He1V/3GvivImDhxLSx6nwhDbY87KSYtm9wAkKCYXAQAiVlaQpd2Hu4csp9TT79H6t/brB5efo45et1q7+3Tbgk/Ja5oyAyw173QYmluQaUW3AcQZFU8AQMTKS/MCruH54+d36q197Xri2lL9x/fmqbqxXR+0dKvP7fXb3uM1VV6aF8vuAkgQVDwBABGbnZuhssIsVTe2+616Ln26ZuD1eJdT//jFWVpT2TTkPE6HoZL8TNbwBMYIKp4AgGFZubhITmPohKDzpqbrq3OmanpWis6bmq5fLCmWJG3aeWhIW6dhaOXiolh3FUCCoOIJABiWguxUrSov0tK1NUOe3rz+czM0Y3Kq3B6v3mnuUPmvtqi91z2ojSFpVXmRCkJMPAIwehA8AQDDtqg4V5K0vKJWHtOUx2vq3QOduvKXbwT8jNNhyGkYWlVeNPB5AGMDt9oBACOyqDhXG2+7WCX5J2amB9qlyHe8ND9TG2+7mNAJjEFUPAEAI1aQnap1N87XjuYOVVQ1aXtDu/a0dMntMeVyGpqVk6a5BZkqL81jIhEwhhE8AQBRMzs3Y1Cw9HpNdiQCMIBb7QCAmCF0AjgZwRMAAACWIHgCAADAEgRPAAAAWILgCQAAAEsQPAEAAGAJgicAAAAsQfAEAACAJQieAAAAsATBEwAAAJYgeAIAAMASBE8AAABYguAJAAAASxA8AQAAYAmCJwAAACxB8AQAAIAlCJ4AAACwBMETAAAAliB4AgAAwBIETwAAAFiC4AkAAABLEDwBAABgCYInAAAALEHwBAAAgCUIngAAALAEwRMAAACWIHgCAADAEgRPAAAAWCKmwfOnP/2p5s6dq7S0NOXk5Oiqq67S+++/H8tLAgAAIEHFNHi+9tpruvnmm7V161Zt2rRJbrdbCxcuVE9PTywvCwAAgAQ0LpYn/+Mf/zjo9VNPPaWcnBxVV1fr4osvjuWlAQAAkGBiGjxP1dHRIUnKysry+35fX5/6+voGXnd2dkqS3G633G53yPP72oTTFvHHeNkL42UvjJe9MF72wngNFsnPwTBN04xhXwZ4vV599atf1bFjx/TGG2/4bXPvvfdqxYoVQ46vWbNGKSkpse4iAAAAItTb26urr75aHR0dSk9PD9rWsuB500036aWXXtIbb7yhadOm+W3jr+KZl5en1tbWkF9EOpG4N23apAULFsjlckWt74gNxsteGC97YbzshfGyF8ZrsM7OTmVnZ4cVPC251X7LLbfo+eef1+bNmwOGTklKTk5WcnLykOMulyuigY20PeKL8bIXxsteGC97YbzshfE6IZKfQUyDp2mauvXWW/Xss8/q1VdfVWFhYSwvBwAAgAQW0+B58803a82aNdqwYYPS0tJ06NAhSVJGRobGjx8fy0sDAAAgwcR0Hc9HH31UHR0duuSSS3TGGWcM/Hr66adjeVkAAAAkoJjfagcAAAAk9moHAACARQieAAAAsATBEwAAAJYgeAIAAMASBE8AAABYguAJAAAASxA8AQAAYAmCJwAAACxB8AQAAIAlCJ4AAACwBMETAAAAliB4AgAAwBIETwAAAFiC4AkAAABLEDwBAABgCYInAAAALEHwBAAAgCUIngAAALAEwRMAAACWIHgCAADAEgRPAAAAWILgCQAAAEsQPAEAAGAJgicAAAAsQfAEAACAJQieAAAAsATBEwAAAJYgeAIAAMASBE8AAABYguAJAAAASxA8AQAAYAmCJwAAACxB8AQAAIAlCJ4AAACwBMETAAAAliB4AgAAwBIETwAAAFiC4AkAAABLEDwBAABgCYInAAAALEHwBAAAgCUIngAAALAEwRMAAACWIHgCAADAEgRPAAAAWILgCQAAAEsQPAEAAGAJgicAAAAsQfAEAACAJQieAAAAsATBEwAAAJYgeAIAAMASBE8AAABYguAJAAAASxA8AQAAYAmCJwAAACxB8AQAAIAlCJ4AAACwBMETAAAAliB4AgAAwBIEz1HK6zXj3QUAAIBBxsW7A4iOHc0dqqhqUmVDm+pauuX2mHI5Dc3MmaCygiyVl+Zpdm5GvLsJAADGMIKnzTW09uj29bWqrG+T02HIc1Kl0+0xtetgl3Yf7tbqLY0qK8zSysVFKshOjWOPAQDAWMWtdhvbUNOshQ9uVnVjuyQNCp0n8x2vbmzXwgc3a0NNs2V9BAAA8KHiaVMbapq1dG2NInmS0+M15ZGppWtrJEmLinNj0jcAAAB/qHjaUH1rj5ZX1EYUOk9mSlpeUauG1p5odgsAACAogqcN3bG+Vh5zZLPWPaap29fXRqlHAAAAoRE8bead/R2qrG8L+DxnuDxeU5X1bdrR3BGlnsEKLJMFALAznvG0mWeqmzTOYei4nwCy9oYLtOtgp/qOe7Vkbp7cHq/+Y9s+PfTnPX7P5XQYqqhqYpmlBMYyWQCA0YTgaTOVDW1+Q6fP4pJpeuL1el31yJv6TH6mHvj6HFU1tOuNutYhbT1eU9sb2mPZXQwTy2QBAEYjbrXbTF1Ld9D33zvYpV+8vEcNR3v1X281q7a5Q5+dOSlg+z0tXdHuIkaIZbIAAKMVwdNGvF5Tbk/wZ/zeO9Q56PWRro80aUJywPZuj8lzgwnEt0xWv8c7JHCuveEC3f2Vc4d8xuM11e/xaunaGsInACChETxtxOEw5HIaQdscPyWYmqbkCPIRl9OQI1gDWIZlsgAAox3B02Zm5kyI6vlm5aRF9XwYPpbJAgCMdgRPmykryJIzShVKp8PQ3ILMqJwLI8MyWQCAsYDgaTPlpXkjDic+Hq+p8tK8qJwLI+NbJisafMtkAQCQaFhOyWZm52aorDBL1Y3tQwLokse3Dml/w79X+z2P02GoJD+TNSATRKhlsiLBMlkAgERFxdOGVi4uktMYWXXMaRhaubgoSj3CSIVaJitSLJMFAEhEBE8bKshO1aryIg03ehqSVpWz4HiiCGeZrEixTBYAIBFxq92mFhXnSjqxfI7HNMN67tPpMOQ0DK0qLxr4POLPt0xWNMMny2QBABIRFU8bW1Scq423XayS/BMz0wPNdvcdL83P1MbbLiZ0JiCWyQIAjAVUPG2uIDtV626crx3NHaqoatL2hnbtaemS22PK5TQ0KydNcwsyVV6ax0SiBFZWkKXdh7ujsmIBy2QBABIVwXOUmJ2bMShYer0mt1ptpLw0T6u3NAZt42/VAn9YJgsAkKi41T5KETrtxbdM1kg3B3A6DJUVZlHdBgAkJIInkCBYJgsAMNoRPIEEwTJZAIDRjmc8gQTCMlkAgNGMiieQYFgmCwAwWlHxBBIQy2QBAEYjgieQwFgmCwAwmnCrHbARQicAwM4IngAAALAEwRMAAACWIHgCAADAEgRPAAAAWILgCQAAAEsQPAEAAGAJgicAAAAsQfAEAACAJQieAAAAsATBEwAAAJYgeAIAAMASMQ2emzdv1pVXXqmpU6fKMAw999xzsbwcAAAAElhMg2dPT4/mzJmjRx55JJaXAQAAgA2Mi+XJL7/8cl1++eWxvAQAAABsIqbBM1J9fX3q6+sbeN3Z2SlJcrvdcrvdIT/vaxNOW8Qf42UvjJe9MF72wnjZC+M1WCQ/B8M0TTOGffnkQoahZ599VldddVXANvfee69WrFgx5PiaNWuUkpISw94BAABgOHp7e3X11Vero6ND6enpQdsmVPD0V/HMy8tTa2tryC8inUjcmzZt0oIFC+RyuaLRbcQQ42UvjJe9MF72wnjZC+M1WGdnp7Kzs8MKngl1qz05OVnJyclDjrtcrogGNtL2iC/Gy14YL3thvOyF8bIXxuuESH4GrOMJAAAAS8S04tnd3a26urqB1/X19aqpqVFWVpamT58ey0sDAAAgwcQ0eFZVVenSSy8deL1s2TJJ0jXXXKOnnnoqlpcGAABAgolp8Lzkkktk0dwlAAAAJDie8QQAAIAlCJ4AAACwBMETAAAAliB4AgAAwBIETwAAAFiC4AkAAABLEDwBAABgCYInAAAALEHwBAAAgCUIngAAALAEwRMAAACWIHgCAADAEgRPAAAAWILgCQAAAEsQPAEAAGAJgicAAAAsQfAEAACAJQieAAAAsATBE4gSr9eMdxcAAEho4+LdAcCudjR3qKKqSZUNbapr6ZbbY8rlNDQzZ4LKCrJUXpqn2bkZ8e4mAAAJg+AJRKihtUe3r69VZX2bnA5DnpMqnW6PqV0Hu7T7cLdWb2lUWWGWVi4uUkF2ahx7nLi8XlMOhxHvbgAALELwBCKwoaZZyytq5TFPhE1PgNvrvuPVje1a+OBmrSov0qLiXMv6maioEgPA2EbwBMK0oaZZS9fWKJInOT1eUx6ZWrq2RpLGbPikSgwAkJhcBISlvrVHyytqIwqdJzMlLa+oVUNrTzS7ZQsbapq18MHNqm5slxR+lXhDTbNlfQQAWIPgCYThjvWf3F4fLo9p6vb1tVHqkT34qsT9Hm/AwHkqj9dUv8erpWtrCJ8AMMoQPIEQ3tnfocr6trCDk8vpf7KMx2uqsr5NO5o7otm9hEWVGABwKp7xBEJ4prpJ4xyGjgcInmtvuEDvH+qSx2vqqvNz9f6hLn3r11v9tnU6DFVUNY2JCTTRrBKvu3F+lHoFAIgngicQQmVDW8DQ6bO4ZJp+v7VRX3/0L0Hbebymtje0R7N7CclXJR6pk6vEYyGsA8Box612IIS6lu6QbRpae/Szl97T3tYe7Q1xa3hPS1e0upawfFXiU32rLE/bfvBFGae89etvl2jl14v8nstXJQYA2B/BEwjC6zXl9oS+XfxOBM9tuj3mqN9eM1CV+IV3DmpiikvzZ0waOJYx3qWLPzVZz/3V/0SisVIlBoCxgOAJBOFwGAEnC53sw35P2Od0OY1Rv1tPoCpx54fH9dr7RwatZ3rFp09Xe49bW/YeDXi+sVAlBoCxgOAJhDAzZ0JUzzcrJy2q50s0oarEz9U06/LZpyvJeeKvn6uKc/XftQcUbB7SWKgSA8BYQPAEQigryJIzShVKp8PQ3ILMqJwrUYWqEr+8q0UypEvPztEZGadpbkFWwNvsPmOhSgwAYwHBEwihvDQv7DU8Q/F4TZWX5kXlXIksWJW477hXf9pxSFedP1VfnTNVe1t79O6BzqDnG+1VYgAYK1hOCQhhdm6GygqzVN3Y7jeALnnc/5qdp3I6DJXkZ46JZYHKCrK0+3B3wMD+XE2zfnvNXH0qJ03PhtidaCxUiQFgrKDiCYRh5eIiOU9dAyhCTsPQysX+lwwabUJVif/ywVEd+9CtM3MmhNwWc6xUiQFgLCB4AmEoyE7VqvIiDTd6GpJWlRepIDs1mt1KWL4qcaBnY01TmveTl1Vw5wtqavsw4HmcDkNlhVljokoMAGMBwRMI06LiXD20pFhJTkfYk42cDkNJToceWlI8aAmhsYAqMQDgVARPIAKLinO18baLVZJ/4pnDQAHUd7w0P1Mbb7t4zIVOiSoxAGAoJhcBESrITtW6G+drR3OHKqqatL2hXXtauuT2mHI5Dc3KSdPcgkyVl+aN+VvEvsD9TxVvh7UDlI/LaeiB8jkxDexer8kSTQBgMYInMEyzczMGBUuCTBCRrkYVg7Xiff9QqGxoU11L98A/FGbmTFBZQRb/UAAACxA8R4iwAR9+Hwy1oaZZS9fWRJwj3V5TS9fWSNKIq54NrT26fX2tKuvb5HQYg2bbuz2mdh3s0u7D3Vq9pVFlhVlauZjb+wAQKwTPCFE1AcJT39qj5RW1QUPnA+VFSj/NpRv+vXrIe6ak5RW1mjNt4rCD4IaaZi2vqJXn4/04Ay3x5Dte3diuhQ9u1qryojH5XC4AxBrBM0xUTYDI3LH+k8AXyIo/7FSwie8e09Tt62u17sb5EV9/ONVWj9eUR9GrtgIABmNWexg21DRr4YObVd3YLin8qkmohbGB0eqd/R2qrG8LudVoV99xdX50POD7Hq+pyvo27WjuiOj64VRbg/FVWxtae4Z5BgCAPwTPEHxVk36PN+z9uj1eU/0er5aurSF8Ykx6prpJ48J45vWB8iI9/vclQds4HYYqqpoiun441dZQfNVWAED0EDyDoGoCDE9lQ5uOh/kPtVA8XlPbG9rDbh9utTWc6w6n2goACIxnPIOIZtVkOM+oAXZV19Id1fPtaekKu62v2uoLvmtvuEDvHeqS12tqcck09R/36v9ufF8bag7oR4vO0+WfPkOtXX269w/v6tXdRwady1dtZcIgAEQHFc8AqJoAw+P1mhEtFh8Ot8eUN8w/i/6qrYs/k6u23n4t+uUbWr2lQT++arb+3//6jKob2/WVf3tdr+9p1c+/WazTXIP/Soy02goACI7gGUCgZ9T+9jO5+uu/LFCSc/CP7vG/L9HPvzHH77mG84waYFcOhyGXM7prmrqcRtjrpPqrtu462KVf/k+dGo726v+9Uqe+41619fZr7fYmNRzt1b+9vEdZqUk65/T0IZ+NpNoKAAiO4BlAoGfUXqg9KKfD0JfOzRk4Nik1SZeenaOKqv1+z0XVBGPNzJwJUT3frJy0sNoFqra+d6jzkzam1N7br/cPfRIoj3T3SZImTUga8tlIqq0AgOAIngEEekat77hXG2oOqLwkb+DYVefn6sCxD7Vl79GA56NqgrGkrCBLzijt5OR0GJpbkBlW20DV1uN+wuhxj3fo5/0sKhpJtRUAEBzB049Qz6it3b5Pn5uVrSnpyZKkr5dM0zPV/qudPlRNMJaUl+aN+PloH4/XVHlpXuiGH4tXtRUAEBrB049Qz6i9e6BTuw52afFnpml2bro+NSUtZPCkaoKxZHZuhsoKQ1c9k5wO9fR7Ar7vdBgqK8yKaFZ5vKqtAIDQCJ4BhKqaPL19nxaXTFN5SZ7erGvVwY6PgranaoKxZuXiIjkD7IfpdBiamTNBn8nP1J7DgR9DcRqGVi4uiui68ay2AgCCI3gGEKpqsqHmgM7IOE1LyvK0LsSMdaomGIsKslO1qrxI/v4UnTUlTf99y0Xafbhbv9/W6PfzhqRV5UUqyE6N6LqnVluXPL5VP3p+56A2F93/in77ZsPg/t75gjbuPDzwejjVVgBAcATPAEJVTbr6juulHYfU2+fRxncPB2wnUTUZLXhGN3KLinP10JJiJTkdg/4ht/Ngp865+4/67lPb1fnh4L3anQ5DSU6HHlpSrEXFucO6brBqa7iGU20FAATHzkUB+Kom1Y3tAQPo6emn6bmaZvX7mR3r43QYKsnPpGpiQzuaO1RR1aTKhjbVtXTL7THlcp64RVxWkKXy0jzGNQyLinM1Z9pE3b6+VpX1bXI6DL9/pnzHS/Mzdf/iyCudJ/NVW5eurRnWlrfDrbYCAIIjeAaxcnGRFj64WZ5T/tOVPn6c5s+YpAtmTNK/PLcj6DmomthPQ2tPwJDk9pjadbBLuw93a/WWRpUVZmnlCEPSWFCQnap1N84fCPPbG9q1p6XrkzA/eYLKCqMb5n3V0uUVJ7a+Dee5T6fDkNMwtKq8aNjVVgBAYATPIAJVTV78h88pfbxLP3vpPe1t7Qn4eaom9rOhpnkgqEgKGFZ8x6sb27Xwwc0ElTDNzs3Q7NyMT6rJ9W2qO9KtXYe6VHekW5UNbVGtJsej2goACIzgGYK/qslF978S9DNUTexpQ01zxLdmPV5THplaurZGkhjvEOJRTQ5VbZ2Vk6a5BZk8OgEAFiB4hoGqyehX39qj5RW1w3oeUJJMnfjHyZxpExn3AOJdTfZVW328XpO1dQHAYgTPMFE1Gd3uWP9JIBouj2nq9vW1Wnfj/Cj1KnFFGtoSsZpM6AQA6xE8I0TVZPR5Z3+HKuvbRnwej9dUZX2bdjR3jLp/fOxo7tD6qkYVSzr/RxvV7VbYM/ypJgMAfFjHc4QInfb3THWTxvkZx6zUJG3/5y/q/1xy5sCxz0zP1O4fX64Lz5zk91xOh6GKEBsK2ElDa4++8dgWfeXhN/T0x9/L/fGtcN8zmb/ftk9fefgNfeOxLWrwM9kumtVkAIC9UfHEmFfZ0Kbjfp43bOvp1/JnavX435fq9T2t2nukWw9+c45+t6VBf/ngqN9zebymtje0x7rLlojGM5nBqsnTMsfrjTu+MOT41r1HteTxrUOuMVqryQAwlhA8MebVtXQHfO/V949o7fZ9emhJsd7Z36Hefo9W/vH9oOfb0xJ473G78PdM5u+um69s85hUucvvZ/w9k+mrJvsL9geOfai5P/7zwOvJacn6/ffmaVuAoOqrJhM8AcC+uNWOMc3rNeX2BL8NfN8LuzTOYeiKT5+hpWtrgu5UJZ24BW3n7TWj9UxmQ2tPwGqyJHlN6Uh3n45096nzI7fu+9psvbWvXQ/9ebff9qOpmgwAYxXBE2Oaw2HI5Qz+nG7+pBRNST9NDkOaljU+5DldTsPWz/5G85nMYNXkk638epFSk8fpH//zrwp26dFQTQaAsYzgiTFvZs6EgO+5nIYe+maxnq89oJ9v2q2f/W2RJqUmBT3frJy0aHfRMr5nMsPZXvLSs3JUe+9CLSqeOuQ93zOZoarJknTLF2bq4lmT9b3VVerp9wRta/dqMgCMdQRPjHllBVlyBqhQ/tPCs5R2mkv3/mGnHn3tA9W39mjl14sCnsvpMDS3IDNWXY25QDP8T/XVOVP1b98q1tK1NdpQc8BvG6fDUKhTXTb7dP3DF2bp5jVvaV9bb8jr2r2aDABjHcETY155aZ7fCt8FM7L03YsKddvTNeruOy7TlJatq9Hcwiz93bzpfs/l8ZoqL82LdZdjJtgzmT5Xz8vXj6+are+trtL/vNcSsJ3Ha8rlDPxXzKemTNDPvzFHv3rtA+053K3JE5I1eUKyMsa7An7GztVkAACz2gHNzs1QWWGWqhvbBwXQrXvbNOufXxrUdn/7hyq6d6Pf8zgdhkryM2096zrUM5lTp07VXfnJ+vqv/qLa/R0hz+f2eANuMVs0baJSksbpH744S//wxVkDx/0tpyTZv5oMACB4ApKklYuLtPDBzfIMey635DQMrVwc+DZ8ogtnhn9HR4eMlIn6RmleWMHTa0qBZgs9U71fz1TvD7t/dq8mAwC41Q5IkgqyU7WqvEjDfXrQkLSqvChuWzpGY8JNODP8e3p69O3fbNGCc6doxVfPG/E1w+V0GCorzLJ1NRkAQMUTGODbbce3W084M7udDkNOwxi0W48VdjR3qKKqSZUNbapr6ZbbYwbdO93rNcOalDMzZ4J2HQy+ZFHD0R596/GtWnvDBfJ4Tf3o+Z0j/j6hhKomh/v9AADxRfAETrKoOFdzpk3U7etrVVnfFvD5RN/x0vxM3b/YukpnQ2tPwL759k7ffbhbq7c0Kic9WWnJ47SvrTdkMPUpK8jS7sPdIUP33tYefevX206ET9PUfS/4380oGvxVkyMN3gCAxEDwBE5RkJ2qdTfOHwg32xvataelayDczMpJ09yCTMvDTaR7p7d09qlFfQPHTw2mZYVZWnlKaC4vzdPqLY1+z/vtJ7ZoZZlHklOS9MGRbs29789+20ZLktMxqJocSfD29/0AAPFF8AQCmJ2bMazb1bHgb+/04fKFterGdi18cPOgYBdohv9IfOHsHD30zWIV/2ijvKZ07hnpevEfP6dHX63T/R/ve/+zxZ9W8jinbnu6ZtBnf3n1+Vp43umSIg/e/r4fACC+mFwEhCleobO+tUf/tO7tqITOk3m8pvo93o8XgW8eOL5ycZGcRvS+6/b6NqUmj9N5U0+E+HkzsnS0u08XzJg00GZe4SRt3Xt00OecDkNv1rVK+iR493u8YQfiQN8PABA/BE8gwV23ervcI6g+rr3hAt39lXMDvm/qxISqhtYeSSOf4X+qrr7j2nmgcyBoXjBjkp54o17nTk1XSpJTU9KTVZidqm2nBE+P19T2hnbVt/ZoeUXtkOAd6nv5nPr9AADxQ/AEEtjDL+/R3iOxD0we09Tt62sHXi8qztVDS4qV5HQE3E70VMHabas/qgtmZEmS5hZk6U/vHtIHLd2aW5CleYWTdKjjIzUcHbpl5p6WLt2x/pPb68N16vcDAMQHz3gCCaq+tUcP/nl30DaGId3wuRn6Vtl0nTHxNLV292vNtn165JU6SdID5UW6YMYkXTBjkr57UaEk6aL7/0f72z8cdB6P11RlfZt2NHcMPNfqb4a/P75JPiXTJ6qyod1vm617j+obpXk694x0Hfd49cGRHm3d26YLZmQpY7xL2+qP+v2c23OiXyPl7/sBAKxH8AQS1B3raxXqDvsdXz5bS8ry9K/P79T2hnblpCXrzJwJA++v+MNOFWZP0PuHuvTgphMh9mhPn99zOR2GKqqaBgWzk2f4r69qlFQvl8NQn0d+Z/ifedeLfquTlQ0nnvO87qJCbfs4SG7de1Q3XXKm0se79JvX9/rtk8OQHIYRcP94w5DuvPxsLZmbJ7fHq//Ytk8P/XlP2N8PAGAtgieQgN7Z3xGy0pea5NR3Plugu//wrta/dWLyzL62XlU1flJ17Oo7LrfHq4/cHh3p9h84fXzPVPozOzdDZ+WcoxdfrNdf714op3PckMlW9a098ga4Jd754XG9d6hTi4qn6p4/vKu1N1ygupZunTc1Q0njHNq21/93dTkd6jvuDdjnxSXT9MTr9brqkTf1mfxMPfD1OapqaNcbH09KCvf7AQCswTOeQAJ6prpJ44I8M3n57NP1wj9+Tskup35wxTn6/XXzNN7lHPF197QE37XIx98M/ztCPEO5bW+bxjkdA7PX+497VdfSpZbOj7Q3wMSfYKFTkt472KVfvLxHDUd79V9vNau2uUOfnTkpYPtwvx8AIDaoeAIJqLKhLeDt5clpyfq3b52vJ16v1/++5EzduuYtTZ+UqmisgOT2mMNarzScCu2Pnt85ZHvNK/7tjYj7eLL3DnUOen2k6yNNmpAcsP1wvx8AIDoInkACqmvpDvheTlqyXE6Hnq7ap2suLNAZE8fr91v97zYknagshhu0XE5jWKHMV6H1F5bHu5z68ddm67LzTldP33E9HuB5zp987dO64tOna2JKkq74xevaebDTb7uTHfcMvp5pnnguNJDhfj8AQHQQPIEE4/WacnsCzyradbBTb+xp1R9uuUhN7b265yvnapzD0Ot7WjUpNUmzpqRpXVXTQPv97R+qOG+ipmWOV0/fcR370K1AqxPNykkbVp+DVWh/cMU5mleYpet/V6Wj3f1aftlZOm9qunYe+CRYXvKpyfp6yTQteXyrmtp61dbbP6x+hDLc7wcAiA6e8QQSjMNhyOUMXJXzmtLfPbFN1z65XS/tOKiefo9+tGi2/rzs83r46vOVPSFpUPtfv75XXq+pTbd9Xn+9e6FyJ473e16nw9Dcgsxh9TlQhTYlyalvzJ2mn7y4S3/54KjeP9yl7697W+Mcg//qmT4pRS1dH+mtfe060t0Xte06TzaS7wcAiA4qnkACmpkzQbsOBp8IU93YrurGdv3iz3v05p1f0G9er9cTb9QPaVff2qO/ffQvIa/p8ZoqL82TFNm+9MEqtPmTUpQ8zqmafccGjnV86Nbe1k+C6gPlRfp6yYnrNvzsb7S/vVcX3f9KWNeOxMnfDwAQHwRPIAGVFWRp9+Fuv5W/4ryJuvDMSXp9T6uOdvepePpEZaUm6YMgz4WG4jCkyROStfyZt1XX0i23x5TLaWhmzgSVFWSpvDRPZ+Wk+P/sxxXaYI8HBLPiDzvVeLRX3yqbrkW/fDPsXYqWPL51yLEb/r3ab1unw1BJfiZreAJAnBE8gQRUXpqn1Vv8Txjq+ui45hVm6bsXFSoteZz2H/tQ972wS6/uPjLs63lN6Uh3nw53fbLWp9tjatfBLu0+3K3VWxp14YyJ+uYU/58PVKFtPNqr/uNeFU+fqAPvHJIkpY8f9/He7CdmwXf1HVdP33F5TTPkWqPD5TQMrVxcFJNzAwDCR/AEEtDs3AyVFWapurF9SNXzgyPduubJ7VG/ZqDHKn3Xr2k6pm9OkV5856AWfWb6oDaBKrS9/R6tq2rSD644R+29bh3t7tPyL58VckemaDIkrSovUkF2qnUXBQD4xeQiIEGtXFwkZzQW54wSX6i8Y32tNtQ0D3qvvDQv4ISgn7y4S5X1bXrimlL9x/fmaXtDu3Y0d8S8v06HoSSnQw8tKdai4tyYXw8AEBoVTyBBFWSnalV5kZaurVE0C4ROwwj7OUp/TEnLK2o1Z9rEgSpisAptb79Hy9a9rWXr3h449vhm/2t5hsP4uA9Oh+E37PqOl+Zn6v7FVDoBIJFQ8QQS2KLiXD20pFhJToecYc4ydzoMjXMY+tysbJ17RvrA0kwup6Fzz0jXpLSkoIush8Njmrr9lC0yrarQnnNGup6/9SL93bzpfr/f382brudvvUhP3zif0AkACYaKJ5DgFhXnas60ibp9fa0q69uGVenzLY/0zv4OXfnLkW1TKZ247V5Z36YdzR0DM8VjVaE9mW8tztm5GYNmqLMNJgDYA8ETsIGC7FStu3G+djR3qKKqSdsb2rWnpWtg2aNZOWmaW5Cp8tI8v0sG+UJZsK0t195wgd4/dGJm+tc+k6vjHlO/39qon2/a7bdPToehiqqmQdfzPUu5vKJWHtMMeyH4377ZoN++2RCyXaC1OAmdAGAPBE/ARkZa6Qu2taUkLS6ZpnXbm3TVL9/Up6dl6Kd/+2kdOPah1m5vGtLW4zW1vaF9yPFwK7SRYi1OALA/gidgY5FW+gJtbelz8NiH+tHzOyVJe1t7dPbpabruokK/wVOS9rT4313p1Artm3VHVXdk+AvcS6zFCQCjAcETGCOCbW3p89emY4Nev7XvmL73uRkBJyO5PWbQquvJFdoNNc3Dfv6TtTgBYHRgVjswRvi2towml9OQw2HIG8at9OHO0GctTgAYPSwJno888ogKCgp02mmnad68eaqsrLTisoizcMIIrDUzZ0LQ94vzJg56fX7eRDW09gTcachhGJr1zy9qxg9e1Kx/flGX/2Kz7tmwI+AC8YuKc7XxtotVkp8pSQEDqO94aX6mNt52MaETAEaJmN9qf/rpp7Vs2TL96le/0rx58/TQQw/py1/+st5//33l5OTE+vKwkO95vsqGNtW1dA/MuJ6ZM0FlBVkBZ1zDOoG2tvSZOnG8fvg352jNtn2anZuhay4s0H0v7Ap4vr7j3oH/f+re7mWFWVrpZwH3kc7QBwDYV8yD589//nNdf/31+s53viNJ+tWvfqUXXnhBv/3tb3XnnXcOatvX16e+vr6B152dnZIkt9stt9sd8lq+NuG0RfTsO9qrf/nDDlU3tg/MYHZISnZKkqm9LZ1qbO3S2soGleRn6l+/OlvTJ6UwXnGw+PwztLayQeOcQ98zJG34636lJjm04ZbPyuM19bst9Vpf3ahkp5TsOBFWff8bmKlxTmnH/jZd+YvX9OOvzdYVnz5jSKuzclL0wyvOGnjt71lRfm8MH3++7IXxshfGa7BIfg6GaY5g77wQ+vv7lZKSomeeeUZXXXXVwPFrrrlGx44d04YNGwa1v/fee7VixYoh51mzZo1SUlJi1U0Akj772c+qo6NDO3bsiHdXAAA20tvbq6uvvlodHR1KT08P2jamFc/W1lZ5PB5NmTJl0PEpU6bovffeG9L+rrvu0rJlywZed3Z2Ki8vTwsXLgz5RaQTiXvTpk1asGCBXC7XyL8AgnrxnYO6Y33tsGcp3/+186TmWsbLYvuO9uqqR95Uv9c76PjvPm3ovUOGflLppxyqE5XOfy316l+qHOrzGvrddfP13sEO/eTFnSGvmeRw6LmbP6vpk/gHpFX4+9BeGC97YbwG892hDkdCLaeUnJys5OTkIcddLldEAxtpe0SuvrVHy9e/q35P4NnJa2+4QDsPdA6sC3mqH27YpR+XMl5WO/P0DP148ZwhSxuZkjymob4gYypJfd4TbcJtL0nHTemuDTu17sb5I+o7IsefL3thvOyF8Tohkp9BTINndna2nE6nDh8+POj44cOHdfrpp8fy0oixO9af2BIxmBv/vVrHPd6A73titqM3QvG3teWSx7fG7Hr+9nYHAIw9MV1OKSkpSSUlJXr55ZcHjnm9Xr388suaP5/Kh129s79DlfVtIbdB7PjQrZ5+T8D3fZ/fdTD8Ej2iJ9yljYJxOgyt+Op5qr13od76lwVatuBTQdtWVPnfAQkAMDbEfB3PZcuW6de//rVWr16tXbt26aabblJPT8/ALHfYzzPVTRoXRkhZe8MFuvsr54Zs9+xfm6PRLQyDb2mj52+9SH83b7rOPSN9YJF5l9NQ8rjgf0UsLpkmj9fUVb98Uyv++11973OFWjI3z2/bQHu7AwDGjpg/4/nNb35TR44c0d13361Dhw6puLhYf/zjH4dMOIJ9VDa06XgUF4d/q/FY1M6F4Tl5a0vpk6WNZv3zi0E/F6293QEAY4MlOxfdcsstamxsVF9fn7Zt26Z58+ZZcVnESF1Ld1TP98ERwkii8W2DOZy93QuyU0Pu7Q4AGJvYqx0RCSeMRMrtJYwkolju7Q4AGJsInohITMKIgzCSqKK9t/usnLQo9QwAYEcET0QsVBiJ1JmTCSOJqqwgK+hsd9/e7jOyU/XVOVN1zYUFevLNBr9tnQ5DcwsyY9RTAIAdEDwRsVBhJFKfyZ8YtXMhuspL84Ium/Vfb+3XaS6nnrvls/rRovP05JsNWlO5z29bj9dUean/Ge8AgLEhoXYugj2Ul+Zp9ZbGqJ3va+fnRu1ciK7ZuRkqK8xSdWO7dMqC/ycvOP/D54Lv7+50GCrJz2TxeAAY46h4ImK+MBKq6rnk8a0Bt8uUPlmw/Jwz0qPaP0TXysVFchojq3A7DUMrFxdFqUcAALsieGJYohJGxIQiOyjITtWq8qJhj5YhaVV5kQqyU6PZLQCADRE8MSzRCCM//trsaHYJMbSoOFf3f1yxDPf5XqfDUJLToYeWFA/sDQ8AGNsInhi2RcW5emhJsZKcjmGFkSs+fUaMe4ho8o2XbwmlQGPuO16an6mNt11M6AQADGByEUZkUXGu5kybqNvX16qyvk1Oh+F3FrTveGl+pu5ffOK2q9vtjkOPMVKrv1Om91t6VVHVpO0N7drT0iW3x5TLaWhWTprmFmSqvDSPiUQAgCEInhixguxUrbtxvnY0dxBGxohAe7sDABAMwRNRQxgZuxhnAEA4eMYTMUMYAQAAJyN4AgAAwBIETwAAAFiC4AkAAABLEDwBAABgCYInAAAALEHwBAAAgCUIngAAALAEwRMAAACWIHgCAADAEgRPAAAAWILgCQAAAEsQPAEAAGAJgidGLa/XjHcXAADAScbFuwNAtOxo7lBFVZMqG9pU19Itt8eUy2loZs4ElRVkqbw0T7NzM+LdTQAAxiyCJ2yvobVHt6+vVWV9m5wOQ56TKp1uj6ldB7u0+3C3Vm9pVFlhllYuLlJBdmocewwAwNjErXbY2oaaZi18cLOqG9slaVDoPJnveHVjuxY+uFkbapot6yMAADiBiidsa0NNs5aurVEkT3J6vKY8MrV0bY0kaVFxbkz6BgAAhqLiCVuqb+3R8oraiELnyUxJyytq1dDaE81uAQCAIAiesKU71tfKY45s1rrHNHX7+too9QgAAITCrXbYzjv7O1RZ3xbwfcOQbvr8mfpW2XRNTktWfWuP/u3lPXppx6FB7TxeU5X1bdrR3MFsdwAALEDwhO08U92kcQ5DxwNMJPo/l8zU187P1T8/+47qj/ZoXuEkPfTNYrX1VGrbKYHV6TBUUdVE8AQAwAIET9hOZUNbwNCZ5HTo5kvP1N/9Zpve2ndMktTUtl+lBZm6et70IcHT4zW1vaE91l0GAAAieMKG6lq6A76XPylFKUnj9O/XzRt03OV0aOeBDr+f2dPSFdX+AQAA/wiesBWv15TbE3hSUWryid/S331quw51fjTovf7jXr+fcXtMeb2mHA4jeh0FAABDEDxhKw6HIZfTCBg+9xzuUp/bo6kTxw+5rR6Iy2kQOgEAsADBE7YzM2eCdh30f3u8p9+jx1/fq3/5yrlyGNL2hnalnTZOpQVZ6v7IrfVvDd2xaFZOWqy7DAAARPCEDZUVZGn34e6A22P+34271dbTr/9zyUzlZaWo8yO33m3u0COvfjCkrdNhaG5BZqy7DAAARPCEDZWX5mn1lsagbZ58s0FPvtkQ8lwer6ny0rwo9QwAAATDzkWwndm5GSorzJJzhM9lOh2GygqzWMMTAACLEDxhSysXF8lpjDB4GoZWLi6KUo8AAEAoBE/YUkF2qlaVF2m40dOQtKq8SAXZqdHsFgAACIJnPGFbi4pzJUnLK2rlMc2Ak41O5nQYchqGVpUXDXweAABYg4onbG1Rca423naxSvJPzEwP9Nyn73hpfqY23nYxoRMAgDig4gnbK8hO1bob52tHc4cqqpq0vaFde1q65PaYcjkNzcpJ09yCTJWX5jGRCACAOCJ4YtSYnZsxKFiyDSYAAImFW+0YtQidAAAkFoInAAAALEHwBAAAgCUIngAAALAEwRMAAACWIHgCAADAEgRPAAAAWILgCQAAAEsQPAEAAGAJgicAAAAsQfAEAACAJQieAAAAsATBEwAAAJYgeAKjjNdrxrsLAAD4NS7eHQAwMjuaO1RR1aTKhjbVtXTL7THlchqamTNBZQVZKi/N0+zcjHh3EwAAgidgVw2tPbp9fa0q69vkdBjynFTpdHtM7TrYpd2Hu7V6S6PKCrO0cnGRCrJT49hjAMBYx612wIY21DRr4YObVd3YLkmDQufJfMerG9u18MHN2lDTbFkfAQA4FRVPwGY21DRr6doaRfIkp8dryiNTS9fWSJIWFefGpG8AAARDxROwkfrWHi2vqI0odJ7MlLS8olYNrT3R7BYAAGEheAI2csf6WnnMkc1a95imbl9fG6UeAQAQPoInYBPv7O9QZX1bwOc5w+Xxmqqsb9OO5o4o9QwAgPDwjCdgE89UN2mcw9BxP8Hz85+arFu+MFNnTUmTx2vqrX3tWvHfO7WvrdfvuZwOQxVVTSyzBACwFBVPwCYqG9r8hk5JGp/k1G9er9eVv3xD/+s32+Q1pcf+vkSG4f9cHq+p7Q3tMeytPbDYPgBYi4onYBN1Ld0B3/vjjkODXt/+zNv6690LNStngnYf9v+5PS1dUe2fHbDYPgDEF8ETsAGv15TbE7g6VzApRcsWfErFeZnKTHXJ8XGpc+rE8QGDp9tjyus15XAEKIuOIiy2DwCJgeAJ2IDDYcjlNAKGzyeumavmYx/qzv+q1eHOPjkMadOyzyvJGfhpGpfTGBOhc0NNs5ZXfLIaQLiL7a8qL2K9UwCIMp7xBGxiZs4Ev8cnprh0Zs4EPfw/e/SXD47qgyPdyhjvCnm+WTlp0e5iwvEttt/v8Ya9GoDHa6rf49XStTXs9AQAUUbwBGyirCBLTj8Vyo4P3Wrr6de3yqYrf1KK5p85ST/8yrlBz+V0GJpbkBmrriYEFtsHgMRD8ARsorw0z2/VzjSlW//zLX06N0Mbl16su79yrn764q6g5/J4TZWX5sWqqwmBxfYBIPHwjCdgE7NzM1RWmKXqxvYhAfTNuqNa8ODmQccK7nzB73mcDkMl+Zmjeva2b7H9kTp5sf3R/PMCAKtQ8QRsZOXiIjkDLc4ZJqdhaOXioij1KDH5FtuPBt9i+wCAkSN4Ykyz2wLiBdmpWlVepOFGKkPSqvLRv1RQsMX2I8Vi+wAQPdxqx5gyGhYQ9y3x41siKJzZ2k6HIadhjJklgoIttj8cY3GxfQCIBYInxoTRtoD4ouJczZk2MeB38vEdL83P1P0J/p2iJdRi+8MxlhbbB4BYInhi1ButC4gXZKdq3Y3zB6q42xvataela6CKOysnTXMLMm1RxY2mUIvtD8dYWWwfAGKN4IlRzbeAeCQRxOM15ZGppWtrJCmhw6d0Yrb7ycGSytyJxfZ3HYze7fGxsNg+AFiByUUYtcbqAuJjPXRKgRfbP9m35+frP743L+S5xsJi+wBgFYInRi0WEB+7Ai22f7Ks1CTlT0oJea6xsNg+AFiF4IlRybeAeLj7cwdy8gLisA/fYvvBqp4P/XmPLrr/laDncToMlRVmjalnZAEglgieGJVCLSCe5HTonivPVdUPv6T3//UyVfzv+Sqa5j9csIC4PbHYPgAkHoInRqVQC4jfdcXZunz2GfqndW/rbx5+Q41He/S775YpY7xrSFsWELcnFtsHgMRD8MSoFGwB8fEup/7XvHz95MVdenX3EdW1dOvO9e/oI7dX35zr/1k+FhC3p0XFuXpoSbGSnI6Qk418nA5DSU6HHlpSnPArGgCA3RA8MeqEWkA8f1KKksY5VN34SRXzuNfU2/uPaWbOBL+f8S0gDvtZVJyrjbddrJL8EzPTAwVQ3/HS/ExtvO1iQicAxADreGLUYQFxnIrF9gEgMRA8MSoFW0C88Wiv+o57VJKfqeZjH0qSxjkMFU3L0G/faPD7GRYQHx1YbB8A4ovgiVGprCBLuw93+11O6UO3R/+xdZ9+cMU56vjQreZjH+p/f36Gxrucerpq35D2LCA+ehE6AcBaBE+MSuWleVq9pTHg+/f/8T0ZhvTzb8zRhORxqm3u0Ld/W6nOD48PacsC4gAARAfBE6OSbwHx6sZ2v1XPvuNerfjvnVrx3zuDnsfpMFSSn8lzfwAARAGz2jFqsYA4AACJheCJUYsFxAEASCzcaseo5luLcXlFrTymGdbe7U6HIadhaFV5EWs5AgAQRVQ8MeqxgDgAAImBiifGBBYQBwAg/gieGFNYQBwAgPiJ2a32++67TxdeeKFSUlI0ceLEWF0GGBFCJwAA1olZ8Ozv71d5ebluuummWF0CAAAANhKzW+0rVqyQJD311FNhf6avr099fX0Drzs7OyVJbrdbbrc75Od9bcJpi/hjvOyF8bIXxsteGC97YbwGi+TnYJimGXp9mRF46qmntHTpUh07dixk23vvvXcgsJ5szZo1SklJiUHvAAAAMBK9vb26+uqr1dHRofT09KBtE2py0V133aVly5YNvO7s7FReXp4WLlwY8otIJxL3pk2btGDBArlcrlh2FVHAeNkL42UvjJe9MF72wngN5rtDHY6Iguedd96p+++/P2ibXbt26eyzz47ktAOSk5OVnJw85LjL5YpoYCNtj/hivOyF8bIXxsteGC97YbxOiORnEFHw/P73v69rr702aJsZM2ZEckoAAACMEREFz8mTJ2vy5Mmx6gsAAABGsZg947lv3z61tbVp37598ng8qqmpkSTNnDlTEyZMiNVlAQAAkKBiFjzvvvturV69euD1+eefL0l65ZVXdMkll8TqsgAAAEhQMVtA/qmnnpJpmkN+EToBAADGppgFTwAAAOBkBE8AAABYguAJAAAASxA8AQAAYAmCJwAAACxB8AQAAIAlCJ4AAACwBMETAAAAliB4AgAAwBIETwAAAFiC4AkAAABLEDwBAABgCYInAAAALEHwBAAAgCUIngAAALAEwRMAAACWIHgCAADAEgRPAAAAWILgCQAAAEsQPAEAAGAJgicAAAAsQfAEAACAJQieAAAAsATBEwAAAJYgeAIAAMASBE8AAABYguAJAAAASxA8AQAAYAmCJwAAACxB8AQAAIAlCJ4AAACwBMETAAAAliB4AgAAwBIETwAAAFiC4AkAAABLEDwBAABgCYInAAAALEHwBAAAgCUIngAAALAEwRMAAACWIHgCAADAEgRPAAAAWILgCQAAAEsQPAEAAGAJgicAAAAsQfAEAACAJQieAAAAsATBEwAAAJYgeAIAAMASBE8AAABYguAJAAAASxA8AQAAYAmCJwAAACxB8AQAAIAlCJ4AAACwBMETAAAAliB4AgAAwBIETwAAAFiC4AkAAABLEDwBAABgCYInAAAALEHwBAAAgCUIngAAALAEwRMAAACWIHgCAADAEgRPAAAAWILgCQAAAEsQPAEAAGAJgicAAAAsQfAEAACAJQieAAAAsATBEwAAAJYgeAIAAMASBE8AAABYguAJAAAASxA8AQAAYAmCJwAAACxB8AQAAIAlCJ4AAACwBMETAAAAliB4AgAAwBIETwAAAFiC4AkAAABLEDwBAABgCYInAAAALEHwBAAAgCUIngAAALAEwRMAAACWIHgCAADAEgRPAAAAWILgCQAAAEsQPAEAAGAJgicAAAAsQfAEAACAJQieAAAAsATBEwAAAJYgeAIAAMASBE8AAABYguAJAAAASxA8AQAAYAmC5ym8XjPeXQAAABiVxsW7A/G2o7lDFVVNqmxoU11Lt9weUy6noZk5E1RWkKXy0jzNzs2IdzcBAABsL2YVz4aGBl133XUqLCzU+PHjdeaZZ+qee+5Rf39/rC4ZkYbWHn3jsS36ysNv6Pfb9mnXwS65PSeqnW6PqV0Hu/T7bfv0lYff0Dce26KG1p449xgAAMDeYlbxfO+99+T1evXYY49p5syZ2rFjh66//nr19PTogQceiNVlw7KhplnLK2rlMU8ETU+A2+u+49WN7Vr44GatKi/SouJcy/oJAAAwmsQseF522WW67LLLBl7PmDFD77//vh599NG4Bs8NNc1aurZGkTzJ6fGa8sjU0rU1kkT4BAAAGAZLn/Hs6OhQVlZWwPf7+vrU19c38Lqzs1OS5Ha75Xa7Q57f1yZQ28ajvfrh+reV5Bz+BKIfrn9bs0+foOmTUoZ9DpwQaryQWBgve2G87IXxshfGa7BIfg6GaZqWTOOuq6tTSUmJHnjgAV1//fV+29x7771asWLFkONr1qxRSgpBDwAAINH09vbq6quvVkdHh9LT04O2jTh43nnnnbr//vuDttm1a5fOPvvsgdfNzc36/Oc/r0suuUS/+c1vAn7OX8UzLy9Pra2tIb+IdCJxb9q0SQsWLJDL5Rr03s4DnfrG41tCniNcFTfO1zlnhO4TAgs2Xkg8jJe9MF72wnjZC+M1WGdnp7Kzs8MKnhHfav/+97+va6+9NmibGTNmDPz/AwcO6NJLL9WFF16oxx9/POjnkpOTlZycPOS4y+WKaGD9tf+vmoPymA4dDzCRaLzLqR9/bbYuO+909fQd1+Ov79WXzpminQc69aPndw5q63QYWv/Xg1oxfVLYfUJgkY4v4ovxshfGy14YL3thvE6I5GcQcfCcPHmyJk+eHFbb5uZmXXrppSopKdGTTz4phyN+69VXNrQFDJ2S9IMrztG8wixd/7sqHe3u1/LLztJ5U9O180DnkLYer6ntDe2x7C4AAMCoE7Mk2NzcrEsuuUTTp0/XAw88oCNHjujQoUM6dOhQrC4ZVF1Ld8D3UpKc+sbcafrJi7v0lw+O6v3DXfr+urc1LkhQ3tPSFYtuAgAAjFoxm9W+adMm1dXVqa6uTtOmTRv0nkXzmQZ4vebA4vD+5E9KUfI4p2r2HRs41vGhW3tbA4dVt8eU12vK4TCi2VUAAIBRK2YVz2uvvVamafr9ZTWHw5DLGd2A6HIahE4AAIAIxO+hS4vNzJkQ8L3Go73qP+5V8fSJA8fSx49TYXZqwM/MykmLZvcAAABGPUsXkI+nsoIs7T7c7Xd7zN5+j9ZVNekHV5yj9l63jnb3afmXz1KguUhOh6G5BZkx7jEAAMDoMmaCZ3lpnlZvaQz4/k9e3KWUJKeeuKZUPX3H9evX65V2mv/lATxeU+WlebHqKgAAwKg0ZoLn7NwMlRVmqbqxPWDVc9m6t7Vs3dsDx75wds6Qdk6HoZL8TM3OzYhpfwEAAEabMfOMpyStXFwkpzGyCUFOw9DKxUVR6hEAAMDYMaaCZ0F2qlaVF2m40dOQtKq8SAVBJh0BAADAvzFzq91nUXGuJGl5Ra08pun3trvPkse3Sjpxe91pGFpVXjTweQAAAERmTFU8fRYV52rjbRerJP/EzHRngPU4fcdL8zO18baLCZ0AAAAjMOYqnj4F2alad+N87WjuUEVVk7Y3tGtPS5fcHlMup6FZOWmaW5Cp8tI8JhIBAABEwZgNnj6zczMGBUu2wQQAAIiNMXmrPRhCJwAAQGwQPAEAAGAJgicAAAAsQfAEAACAJQieAAAAsATBEwAAAJYgeAIAAMASBE8AAABYguAJAAAASxA8AQAAYAmCJwAAACxB8AQAAIAlCJ4AAACwBMETAAAAliB4AgAAwBLj4t2BYEzTlCR1dnaG1d7tdqu3t1ednZ1yuVyx7BqigPGyF8bLXhgve2G87IXxGsyX03y5LZiEDp5dXV2SpLy8vDj3BAAAAMF0dXUpIyMjaBvDDCeexonX69WBAweUlpYmwzBCtu/s7FReXp6ampqUnp5uQQ8xEoyXvTBe9sJ42QvjZS+M12Cmaaqrq0tTp06VwxH8Kc6Erng6HA5NmzYt4s+lp6fzG8FGGC97YbzshfGyF8bLXhivT4SqdPowuQgAAACWIHgCAADAEqMqeCYnJ+uee+5RcnJyvLuCMDBe9sJ42QvjZS+Ml70wXsOX0JOLAAAAMHqMqoonAAAAEhfBEwAAAJYgeAIAAMASBE8AAABYguAJAAAAS4za4NnQ0KDrrrtOhYWFGj9+vM4880zdc8896u/vj3fX4Md9992nCy+8UCkpKZo4cWK8uwM/HnnkERUUFOi0007TvHnzVFlZGe8uwY/Nmzfryiuv1NSpU2UYhp577rl4dwlB/PSnP9XcuXOVlpamnJwcXXXVVXr//ffj3S0E8Oijj6qoqGhgx6L58+frpZdeine3bGXUBs/33ntPXq9Xjz32mN599109+OCD+tWvfqUf/OAH8e4a/Ojv71d5ebluuummeHcFfjz99NNatmyZ7rnnHr311luaM2eOvvzlL6ulpSXeXcMpenp6NGfOHD3yyCPx7grC8Nprr+nmm2/W1q1btWnTJrndbi1cuFA9PT3x7hr8mDZtmn72s5+purpaVVVV+sIXvqBFixbp3XffjXfXbGNMreO5atUqPfroo9q7d2+8u4IAnnrqKS1dulTHjh2Ld1dwknnz5mnu3Ln65S9/KUnyer3Ky8vTrbfeqjvvvDPOvUMghmHo2Wef1VVXXRXvriBMR44cUU5Ojl577TVdfPHF8e4OwpCVlaVVq1bpuuuui3dXbGHUVjz96ejoUFZWVry7AdhKf3+/qqur9aUvfWngmMPh0Je+9CVt2bIljj0DRp+Ojg5J4r9VNuDxeLR27Vr19PRo/vz58e6ObYyLdwesUldXp4cfflgPPPBAvLsC2Epra6s8Ho+mTJky6PiUKVP03nvvxalXwOjj9Xq1dOlSffazn9Xs2bPj3R0E8M4772j+/Pn66KOPNGHCBD377LM699xz490t27BdxfPOO++UYRhBf536H8Pm5mZddtllKi8v1/XXXx+nno89wxkrABirbr75Zu3YsUNr166Nd1cQxFlnnaWamhpt27ZNN910k6655hrt3Lkz3t2yDdtVPL///e/r2muvDdpmxowZA///wIEDuvTSS3XhhRfq8ccfj3HvcLJIxwqJKTs7W06nU4cPHx50/PDhwzr99NPj1CtgdLnlllv0/PPPa/PmzZo2bVq8u4MgkpKSNHPmTElSSUmJtm/frl/84hd67LHH4twze7Bd8Jw8ebImT54cVtvm5mZdeumlKikp0ZNPPimHw3YFXluLZKyQuJKSklRSUqKXX355YJKK1+vVyy+/rFtuuSW+nQNszjRN3XrrrXr22Wf16quvqrCwMN5dQoS8Xq/6+vri3Q3bsF3wDFdzc7MuueQS5efn64EHHtCRI0cG3qNKk3j27duntrY27du3Tx6PRzU1NZKkmTNnasKECfHtHLRs2TJdc801Ki0tVVlZmR566CH19PToO9/5Try7hlN0d3errq5u4HV9fb1qamqUlZWl6dOnx7Fn8Ofmm2/WmjVrtGHDBqWlpenQoUOSpIyMDI0fPz7OvcOp7rrrLl1++eWaPn26urq6tGbNGr366qv605/+FO+u2Yc5Sj355JOmJL+/kHiuueYav2P1yiuvxLtr+NjDDz9sTp8+3UxKSjLLysrMrVu3xrtL8OOVV17x+2fpmmuuiXfX4Eeg/049+eST8e4a/Pjud79r5ufnm0lJSebkyZPNL37xi+bGjRvj3S1bGVPreAIAACB+eOgRAAAAliB4AgAAwBIETwAAAFiC4AkAAABLEDwBAABgCYInAAAALEHwBAAAgCUIngAAALAEwRMAAACWIHgCAADAEgRPAAAAWOL/A/VSmETEDFqiAAAAAElFTkSuQmCC",
276
+ "text/plain": [
277
+ "<Figure size 800x800 with 1 Axes>"
278
+ ]
279
+ },
280
+ "metadata": {},
281
+ "output_type": "display_data"
282
+ }
283
+ ],
284
+ "source": [
285
+ "plt.figure(figsize=(8,8))\n",
286
+ "plt.scatter(C[:,0].data, C[:,1].data, s=200)\n",
287
+ "for i in range(C.shape[0]):\n",
288
+ " plt.text(C[i,0].item(), C[i, 1].item(), itos[i], ha=\"center\", va=\"center\", color=\"white\")\n",
289
+ "plt.grid('minor')"
290
+ ]
291
+ },
292
+ {
293
+ "cell_type": "markdown",
294
+ "metadata": {},
295
+ "source": [
296
+ "------------"
297
+ ]
298
+ },
299
+ {
300
+ "cell_type": "markdown",
301
+ "metadata": {},
302
+ "source": [
303
+ "-------------"
304
+ ]
305
+ },
306
+ {
307
+ "cell_type": "markdown",
308
+ "metadata": {},
309
+ "source": [
310
+ "Not much changes to what we have done so far, but just some code improvement for the lr value to change based on the iterations. "
311
+ ]
312
+ },
313
+ {
314
+ "cell_type": "markdown",
315
+ "metadata": {},
316
+ "source": [
317
+ "Here basically we are open to experimenting with different values, whether it is the inputs, size of the layers or the loss rate values to see how we can decrease the final loss value."
318
+ ]
319
+ },
320
+ {
321
+ "cell_type": "code",
322
+ "execution_count": null,
323
+ "metadata": {},
324
+ "outputs": [],
325
+ "source": [
326
+ "# ------------ now made respectable :) ---------------"
327
+ ]
328
+ },
329
+ {
330
+ "cell_type": "code",
331
+ "execution_count": 33,
332
+ "metadata": {},
333
+ "outputs": [],
334
+ "source": [
335
+ "g = torch.Generator().manual_seed(2147483647) # for reproducibility\n",
336
+ "C = torch.randn((27, 10), generator=g)\n",
337
+ "W1 = torch.randn((30, 200), generator=g)\n",
338
+ "b1 = torch.randn(200, generator=g)\n",
339
+ "W2 = torch.randn((200, 27), generator=g)\n",
340
+ "b2 = torch.randn(27, generator=g)\n",
341
+ "parameters = [C, W1, b1, W2, b2]"
342
+ ]
343
+ },
344
+ {
345
+ "cell_type": "code",
346
+ "execution_count": 34,
347
+ "metadata": {},
348
+ "outputs": [
349
+ {
350
+ "data": {
351
+ "text/plain": [
352
+ "11897"
353
+ ]
354
+ },
355
+ "execution_count": 34,
356
+ "metadata": {},
357
+ "output_type": "execute_result"
358
+ }
359
+ ],
360
+ "source": [
361
+ "sum(p.nelement() for p in parameters) # number of parameters in total"
362
+ ]
363
+ },
364
+ {
365
+ "cell_type": "code",
366
+ "execution_count": 35,
367
+ "metadata": {},
368
+ "outputs": [],
369
+ "source": [
370
+ "for p in parameters:\n",
371
+ " p.requires_grad = True"
372
+ ]
373
+ },
374
+ {
375
+ "cell_type": "code",
376
+ "execution_count": null,
377
+ "metadata": {},
378
+ "outputs": [],
379
+ "source": [
380
+ "lre = torch.linspace(-3, 0, 1000)\n",
381
+ "lrs = 10**lre"
382
+ ]
383
+ },
384
+ {
385
+ "cell_type": "code",
386
+ "execution_count": 36,
387
+ "metadata": {},
388
+ "outputs": [],
389
+ "source": [
390
+ "lri = []\n",
391
+ "lossi = []\n",
392
+ "stepi = []"
393
+ ]
394
+ },
395
+ {
396
+ "cell_type": "code",
397
+ "execution_count": 37,
398
+ "metadata": {},
399
+ "outputs": [],
400
+ "source": [
401
+ "for i in range(200000):\n",
402
+ " \n",
403
+ " # minibatch construct\n",
404
+ " ix = torch.randint(0, Xtr.shape[0], (32,))\n",
405
+ " \n",
406
+ " # forward pass\n",
407
+ " emb = C[Xtr[ix]] # (32, 3, 10)\n",
408
+ " h = torch.tanh(emb.view(-1, 30) @ W1 + b1) # (32, 200)\n",
409
+ " logits = h @ W2 + b2 # (32, 27)\n",
410
+ " loss = F.cross_entropy(logits, Ytr[ix])\n",
411
+ " #print(loss.item())\n",
412
+ " \n",
413
+ " # backward pass\n",
414
+ " for p in parameters:\n",
415
+ " p.grad = None\n",
416
+ " loss.backward()\n",
417
+ " \n",
418
+ " # update\n",
419
+ " #lr = lrs[i]\n",
420
+ " lr = 0.1 if i < 100000 else 0.01\n",
421
+ " for p in parameters:\n",
422
+ " p.data += -lr * p.grad\n",
423
+ "\n",
424
+ " # track stats\n",
425
+ " #lri.append(lre[i])\n",
426
+ " stepi.append(i)\n",
427
+ " lossi.append(loss.log10().item())\n",
428
+ "\n",
429
+ "#print(loss.item())"
430
+ ]
431
+ },
432
+ {
433
+ "cell_type": "code",
434
+ "execution_count": 38,
435
+ "metadata": {},
436
+ "outputs": [
437
+ {
438
+ "data": {
439
+ "text/plain": [
440
+ "[<matplotlib.lines.Line2D at 0x17d66872770>]"
441
+ ]
442
+ },
443
+ "execution_count": 38,
444
+ "metadata": {},
445
+ "output_type": "execute_result"
446
+ },
447
+ {
448
+ "data": {
449
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAicAAAGdCAYAAADJ6dNTAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAABQdUlEQVR4nO3deVhU1eMG8HfYBlAWEVlFwX1HxCRyTxTNbC8zS7OyNC2LMqNcfrZpZra6lOXSt1LLzBbNyoXccAHBJZVERVxYRGVXtjm/P2CGucwMMwMDc4H38zw8yp27nMuFue+cexaFEEKAiIiISCZsrF0AIiIiIm0MJ0RERCQrDCdEREQkKwwnREREJCsMJ0RERCQrDCdEREQkKwwnREREJCsMJ0RERCQrdtYugClUKhWuXLkCFxcXKBQKaxeHiIiITCCEQF5eHvz8/GBjY3p9SIMIJ1euXEFAQIC1i0FEREQ1cPHiRbRu3drk9RtEOHFxcQFQfnKurq5WLg0RERGZIjc3FwEBAZr7uKkaRDhRP8pxdXVlOCEiImpgzG2SwQaxREREJCsMJ0RERCQrDCdEREQkKwwnREREJCsMJ0RERCQrDCdEREQkKwwnREREJCsMJ0RERCQrDCdEREQkKwwnREREJCsMJ0RERCQrDCdEREQkKw1i4r+68vXe87h4vRCP9gtAFx9OKEhERCQHTbrmZMuxK1izPwWp1wqtXRQiIiKq0KTDCREREckPwwkRERHJCsMJAGHtAhAREZFGkw4nCoXC2kUgIiKiKpp0OCEiIiL5YTghIiIiWWE4ISIiIlkxO5zs3r0bY8aMgZ+fHxQKBTZv3mzytvv27YOdnR169+5t7mHrlGCLWCIiItkwO5wUFBQgODgYS5cuNWu77OxsTJgwAcOGDTP3kHWGzWGJiIjkx+zh60eNGoVRo0aZfaApU6bgscceg62trVm1LURERNS01Eubk9WrV+PcuXOYN2+eSesXFRUhNzdX8kVERERNQ52HkzNnzuD111/Ht99+Czs70ypqFixYADc3N81XQEBAHZeSjU6IiIjkok7DSVlZGR577DHMnz8fnTp1Mnm76Oho5OTkaL4uXrxYJ+XjGGxERETyY3abE3Pk5eUhLi4OCQkJmD59OgBApVJBCAE7Ozv89ddfuPPOO3W2UyqVUCqVdVk0IiIikqk6DSeurq44fvy4ZNmyZcuwc+dObNy4EUFBQXV5eCIiImqAzA4n+fn5SE5O1nx//vx5JCYmwsPDA23atEF0dDQuX76Mb775BjY2NujRo4dkey8vLzg6OuosJyIiIgJqEE7i4uIwdOhQzfdRUVEAgIkTJ2LNmjVIS0tDamqq5UpYDzgIGxERkXwohJD/rTk3Nxdubm7IycmBq6urxfb7yIpYHEq5juXj+2BUT1+L7ZeIiIhqfv/m3DpEREQkKwwnREREJCsMJ+AQbERERHLStMMJB2EjIiKSnaYdToiIiEh2GE6IiIhIVhhOiIiISFYYTsBB2IiIiOSkSYcTtoclIiKSnyYdToiIiEh+GE6IiIhIVhhOiIiISFYYTgAIjhFLREQkG006nCjYIpaIiEh2mnQ4ISIiIvlhOCEiIiJZYTgBB2EjIiKSkyYdThQcho2IiEh2mnQ4ISIiIvlhOCEiIiJZYTghIiIiWWE4ATgEGxERkYw06XDCQdiIiIjkp0mHEyIiIpIfhhMiIiKSFYYTAIKjsBEREclGkw4nbHNCREQkP006nBAREZH8MJwQERGRrDCcEBERkawwnBAREZGsNOlwwlmJiYiI5KdJhxMiIiKSH4YTIiIikhWGEwAcg42IiEg+mnQ44SBsRERE8tOkwwkRERHJD8MJERERyQrDCREREcmK2eFk9+7dGDNmDPz8/KBQKLB58+Zq19+0aROGDx+OVq1awdXVFeHh4fjzzz9rWt46IcAWsURERHJhdjgpKChAcHAwli5datL6u3fvxvDhw7F161bEx8dj6NChGDNmDBISEswuLBERETV+duZuMGrUKIwaNcrk9T/++GPJ9++99x5++eUX/PbbbwgJCTH38ERERNTImR1OakulUiEvLw8eHh4G1ykqKkJRUZHm+9zc3PooGhEREclAvTeIXbx4MfLz8/HII48YXGfBggVwc3PTfAUEBNRpmTgIGxERkXzUazj5/vvvMX/+fPzwww/w8vIyuF50dDRycnI0XxcvXqyT8ig4ChsREZHs1NtjnfXr1+OZZ57Bjz/+iIiIiGrXVSqVUCqV9VQyIiIikpN6qTlZt24dJk2ahHXr1mH06NH1cUgiIiJqoMyuOcnPz0dycrLm+/PnzyMxMREeHh5o06YNoqOjcfnyZXzzzTcAyh/lTJw4EZ988gnCwsKQnp4OAHBycoKbm5uFToOIiIgaC7NrTuLi4hASEqLpBhwVFYWQkBDMnTsXAJCWlobU1FTN+l9++SVKS0sxbdo0+Pr6ar5mzJhhoVOoOVHREpYNYomIiORDIYT8b825ublwc3NDTk4OXF1dLbbfwNe3AABC2rjj5+f7W2y/REREVPP7N+fWAZCQmm3tIhAREVEFhhMiIiKSFYYTIiIikhWGEyIiIpIVhhMiIiKSFYYTIiIikhWGEyIiIpIVhhMiIiKSFYYTIiIikhWGEyIiIpIVhhMiIiKSFYYTIiIikhWGEyIiIpIVhhMiIiKSFYYTIiIikhWGEyIiIpIVhhMiIiKSFYYTIiIikhWGEyIiIpIVhhMiIiKSFYYTIiIikhWGEyIiIpIVhhMiIiKSFYYTIiIikhWGEyIiIpIVhhMiIiKSFYYTIiIikhWGEyIiIpIVhhMiIiKSFYYTIiIikhWGEyIiIpIVhhMiIiKSFYYTIiIikhWGEyIiIpIVhhMiIiKSFYYTIiIikhWGEyIiIpIVhhMiIiKSFYYTIiIikhWGEyIiIpIVs8PJ7t27MWbMGPj5+UGhUGDz5s1Gt4mJiUGfPn2gVCrRoUMHrFmzpgZFJSIioqbA7HBSUFCA4OBgLF261KT1z58/j9GjR2Po0KFITEzESy+9hGeeeQZ//vmn2YUlIiKixs/O3A1GjRqFUaNGmbz+ihUrEBQUhA8//BAA0LVrV+zduxcfffQRIiMjzT08ERERNXJ13uYkNjYWERERkmWRkZGIjY01uE1RURFyc3MlX0RERNQ01Hk4SU9Ph7e3t2SZt7c3cnNzcfPmTb3bLFiwAG5ubpqvgICAui4mERERyYQse+tER0cjJydH83Xx4kVrF4mIiIjqidltTszl4+ODjIwMybKMjAy4urrCyclJ7zZKpRJKpbKui0ZEREQyVOc1J+Hh4dixY4dk2d9//43w8PC6PjQRERE1QGaHk/z8fCQmJiIxMRFAeVfhxMREpKamAih/JDNhwgTN+lOmTMG5c+fw2muv4fTp01i2bBl++OEHvPzyy5Y5AyIiImpUzA4ncXFxCAkJQUhICAAgKioKISEhmDt3LgAgLS1NE1QAICgoCFu2bMHff/+N4OBgfPjhh/jqq6/YjZiIiIj0UgghhLULYUxubi7c3NyQk5MDV1dXi+038PUtmv/Hz45Ay+Zs50JERGQpNb1/y7K3jjUkpedZuwhEREQEhhMiIiKSGYaTCrJ/tkVERNREMJwQERGRrDCcEBERkawwnFSQf58lIiKipoHhhIiIiGSF4aSCYJNYIiIiWWA4ISIiIllhOKnANidERETywHBSgdmEiIhIHhhOKtwsLrV2EYiIiAgMJxpFpSprF4GIiIjAcKLBNidERETywHBS4Z//rlq7CERERASGE42fEy5buwhEREQEhhMiIiKSGYYTIiIikhWGEyIiIpIVhhMiIiKSFYYTIiIikhWGEyIiIpIVhhMiIiKSFYYTIiIikhWGEyIiIpIVhhMiIiKSFYYTIiIikhWGEyIiIpIVhhMiIiKSFYYTIiIikhWGEyIiIpIVhhMiIiKSFYYTIiIikhWGEyIiIpIVhhMiIiKSFYYTIiIikhWGEyIiIpIVhhMiIiKSFYYTIiIikpUahZOlS5ciMDAQjo6OCAsLw6FDh6pd/+OPP0bnzp3h5OSEgIAAvPzyy7h161aNCkxERESNm9nhZMOGDYiKisK8efNw5MgRBAcHIzIyEpmZmXrX//777/H6669j3rx5OHXqFL7++mts2LABb7zxRq0LT0RERI2P2eFkyZIlmDx5MiZNmoRu3bphxYoVcHZ2xqpVq/Suv3//fvTv3x+PPfYYAgMDMWLECIwbN85obQsRERE1TWaFk+LiYsTHxyMiIqJyBzY2iIiIQGxsrN5t7rjjDsTHx2vCyLlz57B161bcddddBo9TVFSE3NxcyRcRERE1DXbmrJyVlYWysjJ4e3tLlnt7e+P06dN6t3nssceQlZWFAQMGQAiB0tJSTJkypdrHOgsWLMD8+fPNKRoRERE1EnXeWycmJgbvvfceli1bhiNHjmDTpk3YsmUL3n77bYPbREdHIycnR/N18eLFui4mERERyYRZNSeenp6wtbVFRkaGZHlGRgZ8fHz0bjNnzhw88cQTeOaZZwAAPXv2REFBAZ599lm8+eabsLHRzUdKpRJKpdKcohEREVEjYVbNiYODA0JDQ7Fjxw7NMpVKhR07diA8PFzvNoWFhToBxNbWFgAghDC3vERERNTImVVzAgBRUVGYOHEi+vbti379+uHjjz9GQUEBJk2aBACYMGEC/P39sWDBAgDAmDFjsGTJEoSEhCAsLAzJycmYM2cOxowZowkpRERERGpmh5OxY8fi6tWrmDt3LtLT09G7d29s27ZN00g2NTVVUlMye/ZsKBQKzJ49G5cvX0arVq0wZswYvPvuu5Y7CyIiImo0FKIBPFvJzc2Fm5sbcnJy4OrqarH9Br6+RfJ9ysLRFts3ERFRU1fT+zfn1tGSmcch9YmIiKyN4UTL/uRr1i4CERFRk8dwoqWguNTaRSAiImryGE6IiIhIVhhOiIiISFYYToiIiEhWGE60bD2eZu0iEBERNXkMJ1r2sbcOERGR1TGcEBERkawwnBAREZGsMJwQERGRrDCcEBERkawwnBAREZGsMJwQERGRrDCcEBERkawwnBAREZGsMJxUcb2g2NpFICIiatIYTqp4beNRaxeBiIioSWM4qeLQ+evWLgIREVGTxnBCREREssJwQkRERLLCcFLFrRKVtYtARETUpDGcVFFcxnBCRERkTQwnREREJCsMJ3qoVMLaRSAiImqyGE6IiIhIVhhOiIiISFYYTvRQKKxdAiIioqaL4YSIiIhkheGEiIiIZIXhhIiIiGSF4USP+b+dhBDsTkxERGQNDCd6rNmfgqDorYi/wBmKiYiI6hvDSTUeXB5r7SIQERE1OQwnREREJCsMJ0RERCQrDCdGRH60GzeLy6xdDCIioiaD4cSIpIw8/Hr0srWLQURE1GQwnJiAvYqJiIjqD8MJERERyUqNwsnSpUsRGBgIR0dHhIWF4dChQ9Wun52djWnTpsHX1xdKpRKdOnXC1q1ba1RgOThxOQcvb0jEpRuF1i4KERFRo2N2ONmwYQOioqIwb948HDlyBMHBwYiMjERmZqbe9YuLizF8+HCkpKRg48aNSEpKwsqVK+Hv71/rwteX7acyJN/f/dle/JxwGdO+T7BSiYiIiBovO3M3WLJkCSZPnoxJkyYBAFasWIEtW7Zg1apVeP3113XWX7VqFa5fv479+/fD3t4eABAYGFi7Utez7acycexSNnq1dpcsP3c13zoFIiIiasTMqjkpLi5GfHw8IiIiKndgY4OIiAjExuofTfXXX39FeHg4pk2bBm9vb/To0QPvvfceysoMd88tKipCbm6u5MvazmQwiBAREdUHs8JJVlYWysrK4O3tLVnu7e2N9PR0vducO3cOGzduRFlZGbZu3Yo5c+bgww8/xDvvvGPwOAsWLICbm5vmKyAgwJxiEhERUQNW5711VCoVvLy88OWXXyI0NBRjx47Fm2++iRUrVhjcJjo6Gjk5OZqvixcv1knZmjnY1sl+DUm8mI0bBcX1ekwiIqKGxqxw4unpCVtbW2RkSBuIZmRkwMfHR+82vr6+6NSpE2xtK4NA165dkZ6ejuJi/TdqpVIJV1dXyVdduKunb622z7tVavK6+5KzcN/SfRi0aFetjrkvOQtPrTmMy9k3a7UfIiIiuTIrnDg4OCA0NBQ7duzQLFOpVNixYwfCw8P1btO/f38kJydDpVJplv3333/w9fWFg4NDDYttGQqF6eu+9ftJ/BR/CZNWV99tGgB+O3oF4786gKz8IgBAaZkKa/enAADyikwPNPqM/+ogdp7OxGsbj9ZqP2SeA+euYV9ylrWLQUTUJJj9WCcqKgorV67E2rVrcerUKUydOhUFBQWa3jsTJkxAdHS0Zv2pU6fi+vXrmDFjBv777z9s2bIF7733HqZNm2a5s6gHOTdL8MqPR7Er6arRdV9Yl4B9ydewaNtpAMBTa+Pw18kMI1uZJz3nlkX3R4YVl6rw6JcHMP6rg8i9VWLt4hARNXpmdyUeO3Ysrl69irlz5yI9PR29e/fGtm3bNI1kU1NTYWNTmXkCAgLw559/4uWXX0avXr3g7++PGTNmYNasWZY7C5nKuVl+I9v9n/FAQ/JVUlZZ65d3qxSujvZ1fsz8olIkZ+YjuLUbFOZU8RERNQJmhxMAmD59OqZPn673tZiYGJ1l4eHhOHDgQE0ORdQk3fP5Xpy7WoDPxoVgTLCftYtDRFSvOLeOhQg9swMqYPgT745TGYhY8g+OX8pBSZkKX+89j/8y8kw+Hj9NW4e+61wXzl0tAAD8evRKvRyPiEhOGE4soLC4FEMWxyB603HJ8sSL2Qa3eXptHJIz8zFpzWF8tec83v79JEZ8tLtOyrf3TBaOpN6ok303BcyBRET1q0mHkyfvCLLIfn5NvIIL1wqx7lCqZHl6rvFGq4XFpXi/ouEsAGxOuIwL1wosUi4AyMy7hce/PogHlu232D6rc6ukDL8dvcLxXIiIqMaadDjp5lf78VMuXCvAT0cu1Xj7wmLpMP4vbUjE4A9ijG5X9cO8EAL7k7NwvUooyMwtqnHZamLhH6fxwroEPPbVwXo9bn2pp6c6Gqy0IaKmqEYNYqmSKUGiPvx69ApmrE9Ey2YOiJ8z3Grl+P1YeRuJU2nWnw9JLlQqgTIhYG9r/mcBPlIioqaoSdecWMu0745YfJ/qcVSuFRRj939X8UviZYvs91p+Ed7begrJmdaZ+DArvwiTVh/Cn//qn7uppq7mFWHdoVQUFhsfFK+6hs2mePiLWPR5+2/cLDY82WV1jl3Kxo5Tlh0nh4hIzhhOLGzCKukIshevF+qss+V4mln7NLeHyIRVhzBjfSJSr+ke21wzNx7Dl7vPYdQnu1Fapqq33ipqC7aexq6kq3juf/EW3e/YL2IRvek45v960uxtS8pUKC5VGV+xQvyFG8i7VYrDKdfNPpYCCtzz+T48vTYOZ69yZmwiahoYTiys6oBrA2s4l05Senm34pIyFUZ/uhcz1idIXq9a3a/vs/3V/Nq3Nzla0eOopEyg1/y/8OL6xFrvEwC2nUjDIyticaVijqAjqTcQ9t52zWMhtSwLnIM+57LKGx3/bWaNhBDAgPd3IvSdvyWDs9UHfUG3vv35bzr+MDNcExGZi+FEptQ36UPnr+NkWi5+SbT+eBeFxeU9cSxhyrdHcCjlOuZsPgEAePabOGTkFmH69wlGtqx/2kGwuKwMGblFyLtVirRs86YQqEn7ETm1OblVUobn/hePqd8d4TD+hMLiUrz641HsPM1HjmR5DCcyZ+pTFEODshnb/sC5a1YdA0U9xH9JWT13g7EAAd0y3ygoxrKYZL1zH5nSduVafpGkMbH2ZVVf4zKVwDNrD2vmbqovRVqPsmrafoYaj2W7zmJj/CU8tSbO2kWhRoi9dWTK2Jt/qdbN3FA7kA//SpKMOltapkKZEHhmbRz6d/CEl4sSUT+Uz268d9ZQtG7hrFl324l0ONgpjH5yLylT4ecjlxHeviUCPJyrX9lEa/adx5P9LTMGTX2bsSERu/+7io3xl7DzlSGS10ypBQl9Z7vB19Sbx569hu2nMrH9VCaOX87B8sdD0Vxp/E/554RLyCksqfnPtuHlR6pDV3JuWrsI1IgxnMjUV3vPo1QlsGZ/imbZC+sqH3mo20xcyb6JB5bt1zvg2/6z1yTfd3jzD0R09cKeM1nYcyZL8trJK7macJJdWIwp35Y3QHVz0p3k7v1tpzFrZBcAwMo957BoWxJsbRQ4+95dNThTXf/320k82T8IJy7n4B8ZT5qoLxOq2xyph5/Xpp1NcgpLsCf5KiK6esPR3tbgMbRrW9Thpqi0MrjuOZOFL3efQ9TwTvjwryTEpdzA2qf6wcFOt1L05Q3lQXRYV+9aB0kZPW0iokaIj3VkTDuYANDb3uPDv/4zaSRate2nMvUuP6t1M827Vdm9tkylewdeHnNW8//YigCkbz1tGw6n4uPt/xl8XV+twt2f7ZV8/19GHlKyCnCrpAy3SspwTaux7PmsAgS+vgXPflNZxXyrpAwHz11DaUXDVX09bMy/yVZuUZuKhCfXHML07xPwzhYjvYW0CnjsUg4mf6Pbayevov3HZzuTEXvumtFu15du3DR6vfTRfoxVVKrCP/9dxZZj+hvHFhSVYvvJDNwq4eMfIjIfa04asKgNiRaraX9/22lM6h+IhNRseLsqNcuN3byzCysbRpaUqZCVr3/Y+lk/lc87NLKHD7r4VI7Ma06DT/XcQ118XHAl+yZyb5Xin5lD0Exph6GLYwBUjvcCAFO/jceupKt48c4O6NnaHZO/icP8e7pj4h2BmnXq9UmF1rkmpGYDAH5JuIJ37utp0uYf/JkEAPj7pLQB4sFz0i7Kxro5j1t5ACFt3PHz8/2NHrNMJbBg6yn0C/JAvyAPzfKfjlzCx9vPAAD6Bg6Dt6ujZLvp3x/BrqSrePS2ACx8sJfR4xARaWPNSQO2KeEyfk6wzGBrAPDg8v0Yt/IA3tlyyuRt8rR6bVT9NF5apiofHVVr+fFLOThxOUfz/eGUG1h/KFUScow5nZ6H3Iranbs+2YMNhy/qXW9XUvkjlrWxFzD9+/KB7+b9+q/JxwGA/KJSHLuUo/e1tftTMO+XEyaP/WKsQewnFTd7c52swWi86nBkzOaEy/hq73k8+794yXW8qVUjcqOwPJCeycjT1JSof/brDVwbIqLqsOaENP69Un6T23m68tFPXpH+EVRvlZThWkGxpJdQ1Rve0l1nsSwmWdLLY+bGY5g2tL1kvderzOZsjoLiMly6oTv+h/ZkilXDQ5zWYGi5N0tw/7J9uK+3Px6/vS1sbaQB4t7P90oeee0/W9lWR/3YbXQvP0mtgtrF64WSth2x566hZ2s3HNcKO9ol+0jPY6+atO1Q7zMh9QYKi8vQv4On2fu4VVKGP06k4ZUfj2qWaTfWrRq0dp7OwFNr4tDFxwXbXhpUg1ITEVViOKEa6TX/L53HB+NWHpB8r+9ma6qqg7FVZ90h6afzzLxbknYxgDQEPLQiVvP/UpVAQmo2ElKz8fXe89j5ymDYac2Bc7ZKw9a5v+jWvBw6fw3pubdwT7CfZPnoT/fg1cjOmu8/3XEGn+7QXztSdcJGNUNdxE1xf8VM1N8+HYYBHQ0HlL9PZqCotAx396os/8I/Tuu0eZKWq/L/QgA/HSmvwTudnmdgC13/XsmBl4sjWrkoja9MjZ4QAkWlqmobiFPTwcc6VCPmDN9elSnjfdRmMLbcm9LaHlPblaReL8TZqwW4cK0AZSphcmPOxX/9hxfXJeDgOWnvqNxbpXrDjKRsFbU6UT8kmlhK/dTjxWjvU+3xrw3PEF1SpsLkb+Iw/fsESUAyNtieOZHp+4OpOhNBJqXnYfSne3Hbu4a7Tusra2NQXKrChWu6vbkaHAs32HpxfSK6zNmGlKxG8LOhWmM4oXr3+a7kOt1/xJJ/JN/n3So1OUxFfrwbgz+IQfs3tqLLnG1mHXdrDYZ1FyhvmxOTpL/LdJnKtHIHz/9Lss9HtGqHUHEM/fuvvMPkmTHqq3bNSdXQsCxGen3f+Pk4Rn2yR7LskJnzDP2SeBkd3/zDYiMUW9O4lQcw+IMY7ErS33OuIbpmgWkm1Nd2bWxKrfd1Lb8I3x28YPR3urEEXnMt+fs/3P3ZHhQYeGwvBwwnRBayNvaC2dsUFpehw5t/GHx963HzZ2MuLlXp3Pz1HSMz75bB3lLGniYt3VX52Oyez/dJuhQv2pZktIyXb1QO4GXKXD0zKuZ00h7rpyE4nHIdCVVGYI6/UP79hkMNvLGw1u/I+3U8WrG+ru/VfeCYuPoQ3vz5BF7beMzgOou2nUbHN/+QNNC3pJvFZfjnv6uScYmqc+FaAV5an4BtJ9LwzNo4nd8bS/p0xxmcuJwr6wbrDCdEjYzKxN5D/d7dgYvXK0NCXU84Pe27I5j/27/Yl5wlecwz9bsjdXtgK8m5WYKHV8Ti/mX7DdZcGVNQVIqRH+/Gwj9OI7uwGK/+eBQHqjw+lANzetsZU/X38ND56+g6Zxu+0apRib9wHZ1m/4Elf+kPwicul/9+/XHCcLhfVtEubdGfxsN0TcxYn4CJqw7hnd9N6/341JrD2Jx4BVO+PYLtpzI0bcbqUk1/L+sDwwlRI2OsnYs27Udgc3/9F/d8vrfik57lx4DdcjwNq/elYPxXB3VG/i0tU2FZTHKdflqsa8WlKpxKy9W0+ckurGzDU6Yn+embm6mqH+Mu4nR6Hlb8cxbvbT2FjfGX8OiX5Q3PhRBQ1WAwvbpQl6V4eUMiistUkt/rt34rH7zw0511+4i4NtRjLv3vgGk1qlUb3wPA6XTzhwloLBhOiAhA+dD7xy7l4PejaciyQBsCc6w/fBGLtiXVyafFMpXAthNpyDAwknLOzRKDz96FEDidnltt4+jkzDxsP5mByd/EYdQnezS9x2pbE7V2fwpitELc4RRpcJv8TRxGfLy70bWbyL1ZgsDXtyDw9S0G1zHnR5tU0YOsTCWw4I9TuPfzvXhy9SHN6zXqri+EJHzWxIVrBUbDpSXHsQKAxIvZkr/t/KJSzc9HbtiVmIgktMc2qS9nMvS/QWblF+HHuEsGtzt+KQcr/jmLLcfTsP7Z23F7u5Y666w7lIrZm0/A2cEWJ98aKXntVkmZpjHx3y8PQrtWzSVj3Ww5nobp3yegd4A7+ndoic4+rmjdwgk5hSUY2sULABCxZLdkn2/8fBwujnbo6e+mWWZKDzVth1Ou6wwYeL5KLxb1VBSJF7NxW6DuODv6nE7PxcI/TuOV4Z3Rs3V5+bLyi1BYVIY2LSvH5PnfgQvIzL2FV0ZUdoW/kn0Tp9NzMbSzl04Xd0s+EtykdUMuLDYUGk3fX+THu5GycDR+TriML/45p/O6Ob31U7IK8NORSzh7NR9bj6dj9aTbMLSzl0nbxqVcx9bj6XhlRCf8evQKojcdxwMh/lgytrfpBTDial4R/jqZjnt7++tMBhqXch0PrYiVnO9nO5Px2c5kLHigJ8b1a2OxclhCkw8nXX1ddbo5ElH9umGgzcKz38ThSJXB/XJuliD+wnUM6tgKYz6vnH/p0S8PIGXhaJ19qHtCFeqZ6Tv1euUAfsM/2o37Q/zxkdbNYn1FLUjixWwkXpSWo+pM3tpeWJeAmFeHSJZdzSuqtkYqOTMfMzcexYt3dpR0Dbek8SsP4lpBMfacydJM1Nm3YnC9+NkRKFMJvLv1FH5JLO85c3cvP3T2ccH2kxl4pmLeqi+eCEVkdx+zj33pRiFOXM5BZHcfk8fvseRTqyvZ+mdRjkm6ipSsAgR6NjO6jzGf7ZUMTLnkr/9wNa8IP8VfworHQ9GimYPBbdXjK63ad16zbFPCZYuGkye+PojT6Xk4eO46Ph0XInlNPdmrvmAXvem47MJJk3+s00vr0w0RWcevBroIVw0mQHm36afWxGkaNFZVphIGB7V7cV0CTlzOgUol8MzaOMyr0j7HnGr02ZtP4Eg1bWSq3gNue3e7pEt11bA0/fsjSEjNxqQ1h40eO/VaZag6VzER5JXsm7iaV4R8rZtnZt4tTPv+iGZk42sVPxd9vV/OZRXglR+PaoIJUF7rcPxSjiaYAMCBc9dq9EhjwPu7MOXbI1iqZygBcyeINKW9jjnu+nSP8ZWgO2K2gMBrG4/h4PnrWPDHKZOnsjCZ1u6S0vPwyfYzemuT0nJuorRMpRkEcZueyT/l0TrJdE2+5sTLlaNTEjVES/7WHYH4VkmZZvTiLS8OQHc/6YePX49ewa9Hr2DNpNuw/VSGzvZVVfcBPybpqsHxaUzZz54zWUi8mI0D565h8sB2kkCVllP9TONTv4vX/H/WT8dxV09f3LFwp2SdDx7qhR2nMrHt33RsOZaGU1Ueaemjr/3BC+ukvalW70vB6n0pRvdlyOK//sP0Oztqvv/+YCre+Fn/FBaf7ZSOqFymErC1UaC0TP+t9sC5a9imp4eOsdCgDoo3Coqx9UQa7u7lBzcne822b24+gbOZ+TrbqXsFAcAPcZeQX1SKZeNDddazRLugyI/LHx/m3CzB7NFdcenGTbRu4YSxX8bicMoNyRQaxaUqpOXchK+bE5buSsZf/6ajr4mP/uSiyYeTKYPb4zMZt/gmamqEEDUetl974Ly3fz+J9c+G612voMjwJ/W4lOsWeSN/0YQxWe5bug8ANDdCtWOXsqvdTrvmBAAu63lkMbPKGB/LqwyOp1JJ6x8U0P/pWl9Po6rUjVgHdWqFb57qZ3R9oPwxl7ERgqu2EWn/xtZq11f3ZKoqOTMf/xlo16SWlV+kecS1/WQGVk8qP4/fj6Xh+4Op1W6rtvV4OvKLSnXm+wp562+Ttq/qx/hL+GL3OTx+e+Ujl2OXsvHm5uM603YcOi8d22j4kt04MT9SM5v5KZk2fDWkyYeTZsom/yMgkpVNRy7Do5mDWQ0V9Tlw7jp+SbxsUg2JtodWxCJl4ehaV9Ef1xrcy9ip7DlzFblao5lWNz4HoPt4wVhNCyBtaAoA9y3bh2v5lbU1D1UZVdgc6kH/dv93tbyWJi0XwQHuGN7N2+A25kxdUFvDP9ptdB3tmqddWjVi838zbybzyI9264TF/BqOxKquTfv2QGU4irtwA3EXjHe5zy8qlfR4qs2UI9bAOzMRyYolewupR5atylibBSEE7lu6D0cvWWb00IPnqx+uvyYjAWubtNp4O5VLN6Q3zGMmnpv2QH2mmPZ95WOgRQ/1wncHLmDlhL6SdX6KN9wDy1zfxKZgf/I1nQag5jJ0887KN699jb5aLGPMmei0qWA4IaIm50yGbvsBbQMX7dK5mdfGYTPnEmos1MPH93tvh2S5JQOoenC2zRYeE6Q+1Wai08aqyffWIaKm55MdZ6p93ZLBhOqHuZNJktT6Q6a1q6kvDCdERHXsBxlPsNZYbLTgoyLAtAkpLSElS3fYemt4fdNxZOYZb7tUX/hYh4iojl0xocEqycvU747gtsAWdX6cIYtj6vwYpiosKgNcrF2Kcqw5ISIi0qPqfEZUfxhOiIiISFYYToiIiKjWYwtZEsMJERERyQrDCREREekMgW9NNQonS5cuRWBgIBwdHREWFoZDhw6ZtN369euhUChw33331eSwREREVEdyb9VsmP26YHY42bBhA6KiojBv3jwcOXIEwcHBiIyMRGZmZrXbpaSk4NVXX8XAgQNrXFgiIiJq/MwOJ0uWLMHkyZMxadIkdOvWDStWrICzszNWrVplcJuysjKMHz8e8+fPR7t27WpVYCIiImrczAonxcXFiI+PR0REROUObGwQERGB2FjDM1q+9dZb8PLywtNPP23ScYqKipCbmyv5qktyaqFMRETU1JkVTrKyslBWVgZvb+k02N7e3khP1z+r5t69e/H1119j5cqVJh9nwYIFcHNz03wFBASYU0yz1XJmdCIiogZPyOhmWKe9dfLy8vDEE09g5cqV8PT0NHm76Oho5OTkaL4uXuS8FERERE2FWXPreHp6wtbWFhkZGZLlGRkZ8PHx0Vn/7NmzSElJwZgxYzTLVCpV+YHt7JCUlIT27dvrbKdUKqFUKs0pGhERETUSZtWcODg4IDQ0FDt27NAsU6lU2LFjB8LDw3XW79KlC44fP47ExETN1z333IOhQ4ciMTGxzh/XEBERUcNj9qzEUVFRmDhxIvr27Yt+/frh448/RkFBASZNmgQAmDBhAvz9/bFgwQI4OjqiR48eku3d3d0BQGc5EREREVCDcDJ27FhcvXoVc+fORXp6Onr37o1t27ZpGsmmpqbCxqZhDTz79IAgfL33vLWLQURERAAUQk7Ncw3Izc2Fm5sbcnJy4OrqavH9H0m9gQeW7bf4fomIiBqKN+/qismDLDsWWU3v3w2riqOOyD+eERERNR0MJ0RERAQB+XxSZzgBABldECIioqaO4YSIiIiggHzmcmE4AeDl4mjtIhAREVkVH+vITICHs7WLQERERBUYToiIiEhWPVcZToiIiEhWGE6IiIhIVhhOiIiISFYYToiIiAgtmjlYuwgaDCdERESEwJbNrF0EDYYTIiIiQkFxqbWLoMFwQkRERDh5JdfaRdBgOKkQ5Cmf6iwiIqL6ppDP6PUMJ2ruzvbWLgIREZHVcG4dIiIikhXWnBAREZGs2DCcyI+MrgkREVGTxnBSYVCnVtYuAhERkdXIqc2JnbULIBfPD+kAP3cnKADM3HhMs/zJOwKxZn+K1cpFRETU1LDmpIKDnQ0e6RsAP3cnyfJufq5WKhEREVH9YYNYIiIiIgMYTkzgIaPJkIiIiBo7hhMTxEbfiT2vDYWLI5voEBFR42RvK59IIJ+SyIRNlYduSjsbKO1sEeDhjGPzRuC5we2sVDIiIqK6w3FOZKxfkIfm/7Y2CtzV01fzvUKhQPSortYoFhERUZPB5xRV2NoocH7BXcguLEELA21N3r2/B978+UQ9l4yIiKgOyai7DmtO9FAoFAaDCQCMD2uLb58Oq8cSERERNR0MJ0RERCSj8WEZTizm9xcGYNbILtYuBhERUY3I6KkOw0lN2dtWXsUdrwxGD383TB3SHmOC/eq1HCfmR9br8YiIqHGS09w6DCc1pNCKmO1bNdf8/9NHe+PfegwMzZVs00xERI0Lw0kNBXk207tcoVCgWZXA8OQdgUb352Bn+FK41sHgb68M72TxfRIRUcPFxzqNQCsXJbZHDcL+1+/U+7q3qxIAsHFKOB7s09ro/l68s4PB1zp6u5hcrupCjrYXhnXEmkm36X3NhbUxRERkRQwntdDBy0VnFmO1na8MwfaoQegb6KH39aoUBiJrzKtDdJ4COlQZYjikjbtJx6hqSGevGm1HUr9NH2DtIhARNSr8iFxHmint0MHL9BoPbcnvjkJRqQr2tjYm1YR4Nldq/u9oZ4PiUlWNjqsho6q9hkBAWLsIRES1Jqe3ftacWFkrFyU+GxciWWZna4NmSjtNMKlaqeLkYCv5XvvlLr6u1R6vmYMt3ru/p87y54e017s/c/wzc4hJ643r1waP3hags3xQp1Y1PLLUxPC28DdQo1VXvprQ1+xtooZ3wuZp/Wt97E8e7V3rfRAR9WnbwtpF0GA4qQfVNTI69MYwjAn2Qycz2pX08HfFE7e3xWsjO+u89umjIXq2qPTvWyPxWFibatcx9IjJGFsTZ42ae3c3LHywl85yf3dHzf93vDK4RmUAgGcGtsO+Km2Bfpp6B4Jbu9V4n9URAojo5m32di8O6wg/rXPWNrCjp8n70Z7/iYiopqo2GbAm+ZSkiVIHgYiuXlj4QE/8Ot34J2kFFHj7vh54fohuI1ofN/03u9ryc3PEpP6BNeo51MGrueT7qjU/am5ODnh+SHtMG9pe0j27KkM9pdQCPJx1lgW2dMb0OzuaUFrTONnrP4fqmFNL8r6e8EZEVJf0vXdaS43CydKlSxEYGAhHR0eEhYXh0KFDBtdduXIlBg4ciBYtWqBFixaIiIiodv2mYlQPHzzVP0jzvUKhwKP92qBXa3edddu2rP5mbEpFR78gD/xi4s1R3/7ef6gX5o3pjoS5I0zah7bfXzCtwahCAbw2sgtmRlpmpN2J4W0l39dmTJjmSjtJzcuuV4eYvY/eAe6a/3er5vFbeLuWBhtaExHVFVNrv+uD2eFkw4YNiIqKwrx583DkyBEEBwcjMjISmZmZetePiYnBuHHjsGvXLsTGxiIgIAAjRozA5cuXa134hmrd5Nux/PFQzB3TzaT1Z4/uikf6Gu+OXJ3Px4UgWOvmWJ3qfj1r8svraGItg6l7NnW9l7XGcrGzsalxryYAOPxmBELaVD6P1a6hatvS/E8bo3sZfhQzvZpu5eaoWmNlLZuev8PaRSCiBsbscLJkyRJMnjwZkyZNQrdu3bBixQo4Oztj1apVetf/7rvv8Pzzz6N3797o0qULvvrqK6hUKuzYsaPWhW8otGsiZkZ2Rnj7lmZt7+7sgEUPBdeqDOb0JzGnzUl1N1ljqjbMNfWwj+hpTKuPu7MDnhkQhMkDg+DmbA9He1s0M/BIad6YbjgyZ7jm+5+m3oGn+gehTxt37HhlsN5HUYfeGIZ/Zg6Bu7PhGazVLNHw1VyGztUUKQtHW6wcfdoYbmSXOHe4wdeIqP68fW93axdBwqxwUlxcjPj4eERERFTuwMYGERERiI2NNWkfhYWFKCkpgYeH4fE/ioqKkJubK/lqLLr5Vd+bxhL7UA/69v6Dur1yDGmp1R3ZnLqRCbe3rfZ134oahl56GqM+FtYGXXwMNwSeGanb4NfBzkazT23DKxqkDu0s7fEz++5ueHN0ZQ2Vdi3OybciMa5fADp5N8e4fm3g0awyZCgUwNwx3bDp+f4G2794uTrqfeTm0cwBB98Ypvne3dle8kjHGGEgSd7bu37nbaoLVRtxmzpoIBHVLWM9PeubWe8MWVlZKCsrg7e3tGeCt7c30tPTTdrHrFmz4OfnJwk4VS1YsABubm6ar4AA0z4pNwRDatFdduuLA/HisI54KULasLPqZE2LH+6FQ28MwygTenEsG98HD4e2xuO3V9+DR5t69FulCTeWdZNvBwD8Mq0//np5EOJmS6+7oRsxAEwb2gHz75GmeWWV1uShbVvAyd4WH43tjb9fHoQvzejS6+xghwUP9MKfLw0y6dGTsZoddXfsryf2hbdrZYCq7hzN8eHDwXgotDUWPxyMpY/1AQDc3s4D9gZa2FedJXvvrKHo6V83PZYA02pq2no0w+ujyss1oIMn7GwYTpqyGcMs10idaieghXwawwL13Ftn4cKFWL9+PX7++Wc4OhruVRIdHY2cnBzN18WLF+uxlJanHR5q2k0XKK8xiRreCc4O1TfsVCgU8HJ1hKujPSK7e2No51bwclHqXfeunr744OFgKO1stbavvhzfPh2GEd28sen5OxDUqrLmQN+NJrCiZ41CoUAnbxfJgHGAdACzVs11yzjxjkCcfnukwbL8+Fw4js4bgeZKO3T0djF4o1b75NEQKBTSKkx916QmV+m1kV2Q9M5ISdsUS7KztcHiioAyupcvtkcNxjdPhQGQjnVyT7Af/p0fiTs6VHZH9myuROsWzvhobOV6P00Nx/+N6aYJkMboa2+0fHyfal+vqouvC54d2A4/Tb0DX03sq7fmxE5GjfKobr1ch3N8DbbQuEnW9MUTobXa/tunw7DgAdNq0Fs2N/54uj6ZFU48PT1ha2uLjIwMyfKMjAz4+PhUu+3ixYuxcOFC/PXXX+jVq/pukkqlEq6urpKvhqyuJ1Nq0czwL9UXT/TF6kn9zAxF1a/b0dsFX07oi+5+bvByccRfLw/CvtfvhI+bI8b1q3ktV3h7/WN7aNdqCAC3tytvs+PmZA8bG4VZjwYGdPTEmXdG4YnwQL2vP9DHH/0CPfT2mjKFdshTP157QU8DV/XlqK5Lcg//6n/vO3g1N3juVSefVB9P+9egV2t3PNk/COHtW0ra26h1rVLNe1ugbugyVjunruHZPXMofpoajvatmsPGRoHQti0019VHq5bpxPxInKjHWb0tTftc6lPfOh48qy7PS98jX0sw9vdjSaY8cvWrwTAPkd2rv68a4+pkh3H92uBjrQ8lhhj7YFffzCqNg4MDQkNDJY1Z1Y1bw8PDDW63aNEivP3229i2bRv69jV/JE2q3szIzhjcqRU+f6z6AdhMZW6Y6uTtohmRdcED5o3Pof3Iw5TjCiHg7eqIw29GSNp1mMOumj/CJY/0xg9Twi3SpW7RQ72wPWownh4QZHAdF0d7fPJob3w6Tvfafft0WI2Oq+/nqKjyLyD92XvoCbjfPROGDx8Oxl8vD8K0oe3x2bg+OutIj6vAXT0r30z/e2eUpsF0m5bOCG1rfJ6p5ko7SRgNldGIlaYwt02ZvpGSa2Lj1LrrEXVbYAt8NbHu3rcNdfFvrrTDXT19cOhN6d/5nLu7YXxYG8S8OgRTBrfXuy0A2JrwyHDDs5W1hmOCa96ma/HDwUaHTBjSpX7mMns4tLJnp/pv/L4Q/1oNbGkNZkelqKgorFy5EmvXrsWpU6cwdepUFBQUYNKkSQCACRMmIDo6WrP++++/jzlz5mDVqlUIDAxEeno60tPTkZ+fb7mzaOI8mjlg7VP9cHcvyzSYrM9K9Zo2x2jlojS5i7K12Noo0MGrudFaq3t7++OeYD8M6OAJb1cl+lbUULg7O2CYiW9o7TxN6zYc5NkMAzp4YlQPH6M1Th7NHPBgaGt08nbBzMguaGXg0aAhlmjsWnXAvYiulnmD186e2g2vH+nbWqdNlzmeHdSu2tdPvTUSUVqPMl4baZkxfaqae7d0mIInjDRcr05dj7nTxsDAX3e0b4ll40Ph5SKtcXh6QBDevb8nAj0r2y+Z67WRnbHi8T4Ia1fZc3Js3wB8/4zhDwQzhnXU+yECKK916FGH7bmA8oEkten7QHHozWH44OHKnp3a76/VDWwpR2a/e4wdOxaLFy/G3Llz0bt3byQmJmLbtm2aRrKpqalIS0vTrL98+XIUFxfjoYcegq+vr+Zr8eLFljsLmTP0xydXtX0MpX5zN6WRrbBUa9F6UFfVz2r/e7of9r8+TOcxlil6mlg2hUKBb58Jw/LHa/cs25DqBperqXCtG8jAjqa1IxjXz/QpGp4b1A49/F0xuqcvFj0UjBnDOuLPlwYZ3La6UZKr3gCOzhsBT61n+U4OtpLQZmdr2h+buQ1HJ94RqOkJ9/sLA6pt22HsBq+vLVhVNRnrR22S1mCU2m2njI0ErbY9yrwaga8m9MXzQzpgZA/pI8l2rZrhjg6eOPmW/seKHbyam9we6vPHQnTmvNL36O2d+3qYVmg93rirq86yqkGuIavRkJnTp0/H9OnT9b4WExMj+T4lJaUmh2hUmintEDc7QnbP9AypbY3EjGEdcVdPX3QwM6mb8mdvzShzb7A/ikpUddbgVaFQwMR7VfX70fr/E7e3xf8OXKiTT+g/6XmUMLmi5mBIZ9NrOLzdHJGee8vg699PDkNQ9FYA5d2y1RSK8pqKL/45p7NN9F1dYGsDlKkE1h26CGcHWxQWl+ndv52tDX6bPkATWBQKBTr7uGBmZGf8L/YCNk4Nx4D3d2nWd3KwRe6tUpPOzc3JHiO6++D7g6lo3UK3BsLV0R4uSjvkFVW/v2cHtcOZzDxsPW64V+Sy8X3wwroEfPJob9jaKLCtSsBq5mCLgoqfwT3Bfvj16BUAQGeteb20y/Jgn9bYm3wVzw1uj2bKyveEQZ1aobufK5bHnAUA/PBcOPoFlT+yC3x9CwAgOMAdBUWleKCPPxZtS6r23NQ/61bNlbi3tz983Zyw9XgaXjQxkJk72GDVebD+mTkEOTdLNDVEzg52aOPhjNTrhZp1Jg8Mwt29fHGtoLj6fXf1RnruTYzq4QtbGwVmrE/UvHZfb39sP5UhuYb3hfhj9uYTJpW76nufKTmpIX34q6rm43mTWar2UpGjd+7rge8OpmLxw70w+tO9AMonoEvOzMdtgcbbC6ipe+aYS+6PaWxsyqcYsISq3b/rylv3dsf0OztIujZbgoujnU57EIWivEGwuXMYffZoCGb/cgJTBut/JKJQKPDx2N44lHIdd/fy07zh+7o64tURnTGkkxfWH07FL4lXNNu4OtrjnfvKeyk8P6QD/N2dsOjPJKz45yymD+2A5f+c1TlGVdOGdsDzQ9qb3cOuu58r/r2Sq6nxmT26K3r4uWkeSVXd2+HZEbjrkz04l1VgcJ/NlHZYNj5Uc/PX566evhjRzdtgmyobrbuZoVP6bnIY7vl8H4DyRx+LXXppzv/ovBFwsLWBk4MtYpIyNeFEHUy0eTjba6bLMBZOgPKftVq/IA+9+zRXc6Vp7yf6xir6/cUB+C89D80d7ZBdWKJphG9r5Hfhq4l9IYTQ/MyUdjYoKlUBKP/53xPsLwknVcPD7y8MwN2f7dW7b49mDrhwrTIw3dXTF8tjzuJMZs2bSHi7KpGRW6QZHkJOGE5I4/Hb2+LximfTf8wYiOZKO7Ru4QSVqLs5F7Tf+OU06ZRcmPPJp3eAOxIvZktG0FUoFBYPJpbWpqUzvnmqX7Xr3Bfij/tC/CXLFAoF7G1tEN6+JYID3ODqaI//Hbigs63692rWyM54pG9rBHk2Q+r1Qvx69IrRKR30dzOvXPbhw8F45cejktdXT7oNmxMu46HQ8uvg7GBX7Uzgjva22PHKYE3tEFDea8nZ3hbnrxXAw4QRiNWqa+xtyl+w9iNoBaTn7+ZUWWs1uFMrLH44uNpBFNV+f2EAVu09j00J9TtlSXOlPdZMug1Prj5s9raujvboa8YHMm3mhllHexvcKikPMD383TC2bwA2xF3U+UBbPp5TMF7ecLRiO1v8HTUY9y3dh8SL2Xr3bezdY93k2/H5rmRJOJSLhvGcgepdV19XBHg4lz9qkMG4E/YVzzuqdm+lSj9OCcee14biDgNdsi1B3fZB+1m5uobg0dssU6ukTV+j2sUPB6NlMwcs1RpjxdnBDmON9HxRKBRo16q8gfKCB3pi0UO9sPrJ28wqj2dzB7T3qvyk/WCodM4rpb0NvFwc8eyg9nobLFZXNnUwuL2dB5or7WBjo0D7Vs2rHSqgpgz9RZuahRUKBR4KbW1SI9Ae/m5YYkJX1rpQ9fGiumt7TVn6IYlA+cCE2v7vnu74eGxvbI/SbffUoZVuGDQnC/0zc4jmvRQA2rVqjiWP9JZlY1mGE2oQtrw4EOPD2lisu3RjZG9rU+e1Ty8P74Sj80bg3t6VtRhfTeyLb57qh1dGWG5Arbl3d0Nnbxe9PWceCm2NuNkRZk0JUFUzpR0e6RtgVoAAgG+eCjP4SK6HvytcHe31vmaKdc/ejhnDOuLzam6g2jeWAR3MC6GGPtH3qXg851+1V04dfSYJ8myGPm3cMbav5Ub+VtdMPdDHXxNob29XXvPx45Rw3BbYAlteHFCrucBq4v8qRrh+TvPIUjfeVL0sTg62uC/EX2fOrpo0Hwms8siqbctmtfq7qU98rENW9VJER0z/PkHSN1+fTt4uePd+0+cKaiyqq6a3Fu3qfaD8Rj/IwqNxPjUgCE9VMz5MbUZarg2ForwB5t7kLM2yd+/vgZ2nMiU1Ocb2oY+/u5PREVNdHe01jTL7BraQlMMcVR/XnJgfCaWdDfJNbOhbG7Y2Cmx63ryJMGeP7op3tpwy+Pr/jemOe4L9ENLGHf93T3fcKCjWtCW5LdADP06xzszY4/q1QURXb0mPrfpw8I1hKCwu0xu+G0obWYYTsqq7e/mhX6CH2WNoNHTBAaZ1/Z0zuhtOXsnFMwMN36ipfr0yohNsFArcHVz+KXx8WFuMD6v5OCL15dXIzpiz+QQevS1AZ5Zt9UBo7s72CGnjDpUwrQtxTdQkVhrrkeNgZ6NptKq0s61VDVZ1atL7RfreVnn2oW1bwEVph5eHd8Jz/4uvcU3SjGEd8eTqw5oRqQFU285M7h0P1BhOyOq8ZN5g05JiXh2Cs1fzTW4X0qalM/a9fmcdl6pxqDpkf11QKMpH9Z07ppvxleuYuffJJ25viyGdWsHf3Ql5t0pxPqsA9+tpZLypoou4tWqn9BncqRUmDwyyyKzutWHJSoeNU8KhUCgQ2d0HcbMj0NLII0YBgW5+rujm6yrpXTOksxeOzBmOFs6mBbJ37uuBp9cexnPVjK4rBwwnRPUo0LOZZjJEsqwgz2aIGt6pThqQysUDffyxcs95hLRxr9H26jZJbs72WDNJfw8pOYUSNYVCgTdHWz8QWpL2z9nUoSZsbRTY8qLuMPnmtJ0K9GyGHa8MMXl9a2E4IaJGw9SBu2rKEuPT1GYfr0Z2xm2BHghr1xKr9p6vdVmsQYbZR9bu7e2HXxKv4Pkh5d195Rge64L8WtsREcmU0gLzBdXu+LYY0d0Hbk72cHaQb9sB7dm5G6v3H+yJls0czJqgU98owcZ8PLY34mdHWLzRudyx5oSIyIiXIzoh+2axrB7JPRHeFjFJVzG8ynDs1vTu/T2wel8KZt+tO+9LY6DdmPTuXn54pG+AWTUZPfzdsPjhYN1u29VQKBRo2QBGGLc0hhMiIiNm1GKm4qosVSvv7GCHdc/ebpmdWYgpPZfqa+qGutBcaacZa6mmDbAfMjJsApVjOCEiIjLR3b38rF2EJoFtToiIqN60rOcByahhYjghIgDA0M7lDe6MTYZHtWPODN+Nyeonb0P/Di2x6KFe1i4KNQAKUZMh7+pZbm4u3NzckJOTA1dXTvxGVBdyCkvwy9HLGN3Tt0k2wKtP8RduwN/dCT5uTWcAQmqaanr/ZpsTIgJQPjDXhPBAaxejSQitmGyPiPTjYx0iIiKSFYYTIiIikhWGEyIiIpIVhhMiIiKSFYYTIiIikhWGEyIiIpIVhhMiIiKSFYYTIiIikhWGEyIiIpIVhhMiIiKSFYYTIiIikhWGEyIiIpIVhhMiIiKSlQYxK7EQAkD51MtERETUMKjv2+r7uKkaRDjJy8sDAAQEBFi5JERERGSuvLw8uLm5mby+QpgbZ6xApVLhypUrcHFxgUKhsNh+c3NzERAQgIsXL8LV1dVi+5WTxn6OPL+Gr7GfI8+v4Wvs51iX5yeEQF5eHvz8/GBjY3pLkgZRc2JjY4PWrVvX2f5dXV0b5S+ctsZ+jjy/hq+xnyPPr+Fr7OdYV+dnTo2JGhvEEhERkawwnBAREZGsNOlwolQqMW/ePCiVSmsXpc409nPk+TV8jf0ceX4NX2M/RzmeX4NoEEtERERNR5OuOSEiIiL5YTghIiIiWWE4ISIiIllhOCEiIiJZadLhZOnSpQgMDISjoyPCwsJw6NAhaxcJCxYswG233QYXFxd4eXnhvvvuQ1JSkmSdIUOGQKFQSL6mTJkiWSc1NRWjR4+Gs7MzvLy8MHPmTJSWlkrWiYmJQZ8+faBUKtGhQwesWbNGpzyW/hn93//9n07Zu3Tponn91q1bmDZtGlq2bInmzZvjwQcfREZGRoM4N7XAwECdc1QoFJg2bRqAhnf9du/ejTFjxsDPzw8KhQKbN2+WvC6EwNy5c+Hr6wsnJydERETgzJkzknWuX7+O8ePHw9XVFe7u7nj66aeRn58vWefYsWMYOHAgHB0dERAQgEWLFumU5ccff0SXLl3g6OiInj17YuvWrWaXxZzzKykpwaxZs9CzZ080a9YMfn5+mDBhAq5cuSLZh75rvnDhQlmcn7FzBIAnn3xSp/wjR46UrNNQryEAvX+PCoUCH3zwgWYdOV9DU+4LcnrvNKUsRokmav369cLBwUGsWrVK/Pvvv2Ly5MnC3d1dZGRkWLVckZGRYvXq1eLEiRMiMTFR3HXXXaJNmzYiPz9fs87gwYPF5MmTRVpamuYrJydH83ppaano0aOHiIiIEAkJCWLr1q3C09NTREdHa9Y5d+6ccHZ2FlFRUeLkyZPis88+E7a2tmLbtm2aderiZzRv3jzRvXt3SdmvXr2qeX3KlCkiICBA7NixQ8TFxYnbb79d3HHHHQ3i3NQyMzMl5/f3338LAGLXrl1CiIZ3/bZu3SrefPNNsWnTJgFA/Pzzz5LXFy5cKNzc3MTmzZvF0aNHxT333COCgoLEzZs3NeuMHDlSBAcHiwMHDog9e/aIDh06iHHjxmlez8nJEd7e3mL8+PHixIkTYt26dcLJyUl88cUXmnX27dsnbG1txaJFi8TJkyfF7Nmzhb29vTh+/LhZZTHn/LKzs0VERITYsGGDOH36tIiNjRX9+vUToaGhkn20bdtWvPXWW5Jrqv03a83zM3aOQggxceJEMXLkSEn5r1+/LlmnoV5DIYTkvNLS0sSqVauEQqEQZ8+e1awj52toyn1BTu+dxspiiiYbTvr16yemTZum+b6srEz4+fmJBQsWWLFUujIzMwUA8c8//2iWDR48WMyYMcPgNlu3bhU2NjYiPT1ds2z58uXC1dVVFBUVCSGEeO2110T37t0l240dO1ZERkZqvq+Ln9G8efNEcHCw3teys7OFvb29+PHHHzXLTp06JQCI2NhY2Z+bITNmzBDt27cXKpVKCNGwr1/VN36VSiV8fHzEBx98oFmWnZ0tlEqlWLdunRBCiJMnTwoA4vDhw5p1/vjjD6FQKMTly5eFEEIsW7ZMtGjRQnN+Qggxa9Ys0blzZ833jzzyiBg9erSkPGFhYeK5554zuSzmnp8+hw4dEgDEhQsXNMvatm0rPvroI4PbyOX8hNB/jhMnThT33nuvwW0a2zW89957xZ133ilZ1pCuYdX7gpzeO00piyma5GOd4uJixMfHIyIiQrPMxsYGERERiI2NtWLJdOXk5AAAPDw8JMu/++47eHp6okePHoiOjkZhYaHmtdjYWPTs2RPe3t6aZZGRkcjNzcW///6rWUf7/NXrqM+/Ln9GZ86cgZ+fH9q1a4fx48cjNTUVABAfH4+SkhLJMbt06YI2bdpojin3c6uquLgY3377LZ566inJpJUN+fppO3/+PNLT0yXHcXNzQ1hYmOSaubu7o2/fvpp1IiIiYGNjg4MHD2rWGTRoEBwcHCTnk5SUhBs3bph0zqaUxRJycnKgUCjg7u4uWb5w4UK0bNkSISEh+OCDDyTV5Q3h/GJiYuDl5YXOnTtj6tSpuHbtmqT8jeUaZmRkYMuWLXj66ad1Xmso17DqfUFO752mlMUUDWLiP0vLyspCWVmZ5CIBgLe3N06fPm2lUulSqVR46aWX0L9/f/To0UOz/LHHHkPbtm3h5+eHY8eOYdasWUhKSsKmTZsAAOnp6XrPTf1adevk5ubi5s2buHHjRp38jMLCwrBmzRp07twZaWlpmD9/PgYOHIgTJ04gPT0dDg4OOm/63t7eRssth3PTZ/PmzcjOzsaTTz6pWdaQr19V6vLoO452Wb28vCSv29nZwcPDQ7JOUFCQzj7Ur7Vo0cLgOWvvw1hZauvWrVuYNWsWxo0bJ5kg7cUXX0SfPn3g4eGB/fv3Izo6GmlpaViyZEmDOL+RI0figQceQFBQEM6ePYs33ngDo0aNQmxsLGxtbRvVNVy7di1cXFzwwAMPSJY3lGuo774gp/dOU8piiiYZThqKadOm4cSJE9i7d69k+bPPPqv5f8+ePeHr64thw4bh7NmzaN++fX0X0yyjRo3S/L9Xr14ICwtD27Zt8cMPP8DJycmKJasbX3/9NUaNGgU/Pz/NsoZ8/ZqykpISPPLIIxBCYPny5ZLXoqKiNP/v1asXHBwc8Nxzz2HBggWyGhLckEcffVTz/549e6JXr15o3749YmJiMGzYMCuWzPJWrVqF8ePHw9HRUbK8oVxDQ/eFxqZJPtbx9PSEra2tTuvhjIwM+Pj4WKlUUtOnT8fvv/+OXbt2oXXr1tWuGxYWBgBITk4GAPj4+Og9N/Vr1a3j6uoKJyenevsZubu7o1OnTkhOToaPjw+Ki4uRnZ1t8JgN6dwuXLiA7du345lnnql2vYZ8/dT7qu44Pj4+yMzMlLxeWlqK69evW+S6ar9urCw1pQ4mFy5cwN9//210WvmwsDCUlpYiJSWl2rJrl9ua51dVu3bt4OnpKfmdbOjXEAD27NmDpKQko3+TgDyvoaH7gpzeO00piymaZDhxcHBAaGgoduzYoVmmUqmwY8cOhIeHW7Fk5d3Mpk+fjp9//hk7d+7UqUbUJzExEQDg6+sLAAgPD8fx48clbybqN9Ru3bpp1tE+f/U66vOvr59Rfn4+zp49C19fX4SGhsLe3l5yzKSkJKSmpmqO2ZDObfXq1fDy8sLo0aOrXa8hX7+goCD4+PhIjpObm4uDBw9Krll2djbi4+M16+zcuRMqlUoTzMLDw7F7926UlJRIzqdz585o0aKFSedsSllqQh1Mzpw5g+3bt6Nly5ZGt0lMTISNjY3mUYicz0+fS5cu4dq1a5LfyYZ8DdW+/vprhIaGIjg42Oi6crqGxu4LcnrvNKUsJjG56Wwjs379eqFUKsWaNWvEyZMnxbPPPivc3d0lLZmtYerUqcLNzU3ExMRIurQVFhYKIYRITk4Wb731loiLixPnz58Xv/zyi2jXrp0YNGiQZh/qLmMjRowQiYmJYtu2baJVq1Z6u4zNnDlTnDp1SixdulRvlzFL/4xeeeUVERMTI86fPy/27dsnIiIihKenp8jMzBRClHdBa9Omjdi5c6eIi4sT4eHhIjw8vEGcm7aysjLRpk0bMWvWLMnyhnj98vLyREJCgkhISBAAxJIlS0RCQoKmt8rChQuFu7u7+OWXX8SxY8fEvffeq7crcUhIiDh48KDYu3ev6Nixo6QbanZ2tvD29hZPPPGEOHHihFi/fr1wdnbW6aZpZ2cnFi9eLE6dOiXmzZunt5umsbKYc37FxcXinnvuEa1btxaJiYmSv0l1D4f9+/eLjz76SCQmJoqzZ8+Kb7/9VrRq1UpMmDBBFudn7Bzz8vLEq6++KmJjY8X58+fF9u3bRZ8+fUTHjh3FrVu3Gvw1VMvJyRHOzs5i+fLlOtvL/Roauy8IIa/3TmNlMUWTDSdCCPHZZ5+JNm3aCAcHB9GvXz9x4MABaxdJAND7tXr1aiGEEKmpqWLQoEHCw8NDKJVK0aFDBzFz5kzJOBlCCJGSkiJGjRolnJychKenp3jllVdESUmJZJ1du3aJ3r17CwcHB9GuXTvNMbRZ+mc0duxY4evrKxwcHIS/v78YO3asSE5O1rx+8+ZN8fzzz4sWLVoIZ2dncf/994u0tLQGcW7a/vzzTwFAJCUlSZY3xOu3a9cuvb+TEydOFEKUd4+cM2eO8Pb2FkqlUgwbNkznvK9duybGjRsnmjdvLlxdXcWkSZNEXl6eZJ2jR4+KAQMGCKVSKfz9/cXChQt1yvLDDz+ITp06CQcHB9G9e3exZcsWyeumlMWc8zt//rzBv0n1uDXx8fEiLCxMuLm5CUdHR9G1a1fx3nvvSW7s1jw/Y+dYWFgoRowYIVq1aiXs7e1F27ZtxeTJk3VCbEO9hmpffPGFcHJyEtnZ2Trby/0aGrsvCCGv905TymKMouLEiYiIiGShSbY5ISIiIvliOCEiIiJZYTghIiIiWWE4ISIiIllhOCEiIiJZYTghIiIiWWE4ISIiIllhOCEiIiJZYTghIiIiWWE4ISIiIllhOCEiIiJZYTghIiIiWfl/PVzqzeHASCYAAAAASUVORK5CYII=",
450
+ "text/plain": [
451
+ "<Figure size 640x480 with 1 Axes>"
452
+ ]
453
+ },
454
+ "metadata": {},
455
+ "output_type": "display_data"
456
+ }
457
+ ],
458
+ "source": [
459
+ "plt.plot(stepi, lossi)"
460
+ ]
461
+ },
462
+ {
463
+ "cell_type": "code",
464
+ "execution_count": 39,
465
+ "metadata": {},
466
+ "outputs": [
467
+ {
468
+ "data": {
469
+ "text/plain": [
470
+ "tensor(2.1294, grad_fn=<NllLossBackward0>)"
471
+ ]
472
+ },
473
+ "execution_count": 39,
474
+ "metadata": {},
475
+ "output_type": "execute_result"
476
+ }
477
+ ],
478
+ "source": [
479
+ "emb = C[Xtr] # (32, 3, 2)\n",
480
+ "h = torch.tanh(emb.view(-1, 30) @ W1 + b1) # (32, 100)\n",
481
+ "logits = h @ W2 + b2 # (32, 27)\n",
482
+ "loss = F.cross_entropy(logits, Ytr)\n",
483
+ "loss"
484
+ ]
485
+ },
486
+ {
487
+ "cell_type": "code",
488
+ "execution_count": 40,
489
+ "metadata": {},
490
+ "outputs": [
491
+ {
492
+ "data": {
493
+ "text/plain": [
494
+ "tensor(2.1677, grad_fn=<NllLossBackward0>)"
495
+ ]
496
+ },
497
+ "execution_count": 40,
498
+ "metadata": {},
499
+ "output_type": "execute_result"
500
+ }
501
+ ],
502
+ "source": [
503
+ "emb = C[Xdev] # (32, 3, 2)\n",
504
+ "h = torch.tanh(emb.view(-1, 30) @ W1 + b1) # (32, 100)\n",
505
+ "logits = h @ W2 + b2 # (32, 27)\n",
506
+ "loss = F.cross_entropy(logits, Ydev)\n",
507
+ "loss"
508
+ ]
509
+ },
510
+ {
511
+ "cell_type": "markdown",
512
+ "metadata": {},
513
+ "source": [
514
+ "----"
515
+ ]
516
+ },
517
+ {
518
+ "cell_type": "markdown",
519
+ "metadata": {},
520
+ "source": [
521
+ "### Sampling from the model"
522
+ ]
523
+ },
524
+ {
525
+ "cell_type": "code",
526
+ "execution_count": 41,
527
+ "metadata": {},
528
+ "outputs": [
529
+ {
530
+ "data": {
531
+ "text/plain": [
532
+ "torch.Size([1, 3, 10])"
533
+ ]
534
+ },
535
+ "execution_count": 41,
536
+ "metadata": {},
537
+ "output_type": "execute_result"
538
+ }
539
+ ],
540
+ "source": [
541
+ "context = [0] * block_size\n",
542
+ "C[torch.tensor([context])].shape"
543
+ ]
544
+ },
545
+ {
546
+ "cell_type": "markdown",
547
+ "metadata": {},
548
+ "source": [
549
+ "Considering only one set of training set for simplicity rather than the entire training set^"
550
+ ]
551
+ },
552
+ {
553
+ "cell_type": "code",
554
+ "execution_count": 42,
555
+ "metadata": {},
556
+ "outputs": [
557
+ {
558
+ "name": "stdout",
559
+ "output_type": "stream",
560
+ "text": [
561
+ "mora.\n",
562
+ "kayah.\n",
563
+ "seel.\n",
564
+ "ndheyah.\n",
565
+ "reimanield.\n",
566
+ "leg.\n",
567
+ "adeerdoeliah.\n",
568
+ "milopaleigh.\n",
569
+ "eson.\n",
570
+ "arleitzion.\n",
571
+ "kalin.\n",
572
+ "shuhporxhimiel.\n",
573
+ "kin.\n",
574
+ "reelle.\n",
575
+ "joberlyn.\n",
576
+ "bren.\n",
577
+ "der.\n",
578
+ "yarue.\n",
579
+ "els.\n",
580
+ "kaysh.\n"
581
+ ]
582
+ }
583
+ ],
584
+ "source": [
585
+ "# sample from the model\n",
586
+ "g = torch.Generator().manual_seed(2147483647 + 10)\n",
587
+ "\n",
588
+ "for _ in range(20):\n",
589
+ " \n",
590
+ " out = []\n",
591
+ " context = [0] * block_size # initialize with all ...\n",
592
+ " while True:\n",
593
+ " emb = C[torch.tensor([context])] # (1,block_size,d)\n",
594
+ " h = torch.tanh(emb.view(1, -1) @ W1 + b1)\n",
595
+ " logits = h @ W2 + b2\n",
596
+ " probs = F.softmax(logits, dim=1)\n",
597
+ " ix = torch.multinomial(probs, num_samples=1, generator=g).item()\n",
598
+ " context = context[1:] + [ix]\n",
599
+ " out.append(ix)\n",
600
+ " if ix == 0:\n",
601
+ " break\n",
602
+ " \n",
603
+ " print(''.join(itos[i] for i in out))"
604
+ ]
605
+ },
606
+ {
607
+ "cell_type": "markdown",
608
+ "metadata": {},
609
+ "source": [
610
+ "To be fair, most of them could make sense lol. But atleast this time they definetely sound more name like, so we are defo making progress. So lessgoo xD"
611
+ ]
612
+ },
613
+ {
614
+ "cell_type": "markdown",
615
+ "metadata": {},
616
+ "source": [
617
+ "-----------"
618
+ ]
619
+ },
620
+ {
621
+ "cell_type": "markdown",
622
+ "metadata": {},
623
+ "source": [
624
+ "-------------"
625
+ ]
626
+ }
627
+ ],
628
+ "metadata": {
629
+ "kernelspec": {
630
+ "display_name": "venv",
631
+ "language": "python",
632
+ "name": "python3"
633
+ },
634
+ "language_info": {
635
+ "codemirror_mode": {
636
+ "name": "ipython",
637
+ "version": 3
638
+ },
639
+ "file_extension": ".py",
640
+ "mimetype": "text/x-python",
641
+ "name": "python",
642
+ "nbconvert_exporter": "python",
643
+ "pygments_lexer": "ipython3",
644
+ "version": "3.10.0"
645
+ }
646
+ },
647
+ "nbformat": 4,
648
+ "nbformat_minor": 2
649
+ }
README.md ADDED
@@ -0,0 +1,61 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ## SET 1 - MAKEMORE (PART 2) 🔗
2
+
3
+ [![Documentation](https://img.shields.io/badge/Documentation-Available-blue)](https://muzzammilshah.github.io/Road-to-GPT/Makemore-part2/)
4
+ ![Number of Commits](https://img.shields.io/github/commit-activity/m/MuzzammilShah/NeuralNetworks-LanguageModels-2?label=Commits)
5
+ [![Last Commit](https://img.shields.io/github/last-commit/MuzzammilShah/NeuralNetworks-LanguageModels-2.svg?style=flat)](https://github.com/MuzzammilShah/NeuralNetworks-LanguageModels-2/commits/main)
6
+ ![Project Status](https://img.shields.io/badge/Status-Done-success)
7
+
8
+ &nbsp;
9
+
10
+ ### **Overview**
11
+ In this repository, a **Multi-Layer Perceptron (MLP)** language model inspired by the *Bengio et al. (2003)* research paper has been implemented for **character-level predictions**, following Andrej Karpathy's approach in the **Makemore - Part 2** video.
12
+
13
+ The implementation demonstrates building and training the MLP model for sequence prediction while further enhancing the understanding of neural network architectures for language modeling.
14
+
15
+ &nbsp;
16
+
17
+ ### **🗂️Repository Structure**
18
+
19
+ ```plaintext
20
+ ├── .gitignore
21
+ ├── A-Main-Notebook.ipynb
22
+ ├── B-Main-Notebook.ipynb
23
+ ├── C-Main-Notebook.ipynb
24
+ ├── README.md
25
+ ├── notes/
26
+ │ ├── A-main-makemore-part2.md
27
+ │ ├── B-main-makemore-part2.md
28
+ │ ├── C-main-makemore-part2.md
29
+ │ └── README.md
30
+ └── names.txt
31
+ ```
32
+
33
+ - **Notes Directory**: Contains detailed notes corresponding to each notebook section.
34
+ - **Jupyter Notebooks**: Step-by-step implementation and exploration of the MLP model.
35
+ - **README.md**: Overview and guide for this repository.
36
+ - **names.txt**: Supplementary data file used in training the model.
37
+
38
+ &nbsp;
39
+
40
+ ### **📄Instructions**
41
+
42
+ To get the best understanding:
43
+
44
+ 1. Start by reading the notes in the `notes/` directory. Each section corresponds to a notebook for step-by-step explanations.
45
+ 2. Open the corresponding Jupyter Notebook (e.g., `A-Main-Notebook.ipynb` for `A-main-makemore-part2.md`).
46
+ 3. Follow the code and comments for a deeper dive into the implementation details.
47
+
48
+ &nbsp;
49
+
50
+ ### **⭐Documentation**
51
+
52
+ For a better reading experience and detailed notes, visit my **[Road to GPT Documentation Site](https://muzzammilshah.github.io/Road-to-GPT/)**.
53
+
54
+ > **💡Pro Tip**: This site provides an interactive and visually rich explanation of the notes and code. It is highly recommended you view this project from there.
55
+
56
+ &nbsp;
57
+
58
+ ### **✍🏻Acknowledgments**
59
+ Notes and implementations inspired by the **Makemore - Part 2** video by [Andrej Karpathy](https://karpathy.ai/).
60
+
61
+ For more of my projects, visit my [Portfolio Site](https://muhammedshah.com).
names.txt ADDED
The diff for this file is too large to render. See raw diff