본 포스팅에선 SVD를 이용한 추천 시스템 실습 시 사용했던 파이썬 코드를 공유합니다.
import os
import pandas as pd
from surprise import SVD
from surprise import Dataset
from surprise import dump
data = Dataset.load_builtin('ml-100k')
Dataset ml-100k could not be found. Do you want to download it? [Y/n] Y Trying to download dataset from http://files.grouplens.org/datasets/movielens/ml-100k.zip... Done! Dataset ml-100k has been saved to /Users/apple/.surprise_data/ml-100k
In [9]:
pip install scikit-surprise
The following command must be run outside of the IPython shell:
$ pip install scikit-surprise
The Python package manager (pip) can only be used from outside of IPython.
Please reissue the `pip` command in a separate terminal or command prompt.
See the Python documentation for more informations on how to install packages:
https://docs.python.org/3/installing/
In [13]:
df = pd.DataFrame(data.raw_ratings, columns=["user","item","rate","id"])
In [14]:
df.head() # 943 명의 사용자가 1682개의 영화에 대해 평점을 매긴 100,000개의 데이터 # user : 사용자 id # item : 영화 id # rate : 사용자가 영화를 평가한 점수 # id : 타임스탬프
Out[14]:
| user | item | rate | id | |
|---|---|---|---|---|
| 0 | 196 | 242 | 3.0 | 881250949 |
| 1 | 186 | 302 | 3.0 | 891717742 |
| 2 | 22 | 377 | 1.0 | 878887116 |
| 3 | 244 | 51 | 2.0 | 880606923 |
| 4 | 166 | 346 | 1.0 | 886397596 |
In [15]:
#트레이닝 * 모델 저장
import os
from surprise import SVD
from surprise import Dataset
from surprise import dump
data = Dataset.load_builtin('ml-100k')
trainset = data.build_full_trainset()
algo = SVD()
algo.train(trainset)
# 'original' 알고리즘이 컴퓨팅 예상을 한다.
predictions = algo.test(trainset.build_testset())
#Dump algorithm and reload it.
file_name = os.path.expanduser('dump_file')
dump.dump(file_name, algo=algo)
_, loaded_algo = dump.load(file_name)
#we now ensure the algo is still the same by checking the predictions.
predictions_loaded_algo = loaded_algo.test(trainset.build_testset())
assert predictions == predictions_loaded_algo
print('Predictions are the same')
/Users/apple/anaconda3/lib/python3.6/site-packages/surprise/prediction_algorithms/algo_base.py:51: UserWarning: train() is deprecated. Use fit() instead
warnings.warn('train() is deprecated. Use fit() instead', UserWarning)
Predictions are the same
In [16]:
from surprise import Dataset
from surprise import SVD
from surprise import accuracy
data = Dataset.load_builtin('ml-100k')
algo = SVD()
trainset = data.build_full_trainset()
algo.train(trainset)
testset = trainset.build_testset()
predictions = algo.test(testset)
accuracy.rmse(predictions)
/Users/apple/anaconda3/lib/python3.6/site-packages/surprise/prediction_algorithms/algo_base.py:51: UserWarning: train() is deprecated. Use fit() instead
warnings.warn('train() is deprecated. Use fit() instead', UserWarning)
RMSE: 0.6747
Out[16]:
0.6746845355925309
In [18]:
#모델 최적화
import random
from surprise import SVD
from surprise import Dataset
from surprise import accuracy
from surprise import GridSearch
#full dataset 로드
data = Dataset.load_builtin('ml-100k')
data.split(n_folds=3)
#최적의 알고리즘을 찾는과정
print('Grid Search. . .')
param_grid = {'n_epochs': [5,10], 'lr_all':[0.002, 0.005]}
grid_search = GridSearch(SVD, param_grid, measures=['RMSE'],verbose=1)
print(grid_search.best_params)
grid_search.evaluate(data)
Grid Search. . .
defaultdict(<class 'list'>, {})
Running grid search for the following parameter combinations:
{'n_epochs': 5, 'lr_all': 0.002}
{'n_epochs': 5, 'lr_all': 0.005}
{'n_epochs': 10, 'lr_all': 0.002}
{'n_epochs': 10, 'lr_all': 0.005}
/Users/apple/anaconda3/lib/python3.6/site-packages/surprise/evaluate.py:232: UserWarning: The GridSearch() class is deprecated. Please use model_selection.GridSearchCV instead. 'model_selection.GridSearchCV instead.', UserWarning) /Users/apple/anaconda3/lib/python3.6/site-packages/surprise/evaluate.py:66: UserWarning: The evaluate() method is deprecated. Please use model_selection.cross_validate() instead. 'model_selection.cross_validate() instead.', UserWarning) /Users/apple/anaconda3/lib/python3.6/site-packages/surprise/dataset.py:193: UserWarning: Using data.split() or using load_from_folds() without using a CV iterator is now deprecated. UserWarning)
Resulsts:
{'n_epochs': 5, 'lr_all': 0.002}
{'RMSE': 0.9897614364808488}
----------
{'n_epochs': 5, 'lr_all': 0.005}
{'RMSE': 0.9645422178546349}
----------
{'n_epochs': 10, 'lr_all': 0.002}
{'RMSE': 0.9698703469740928}
----------
{'n_epochs': 10, 'lr_all': 0.005}
{'RMSE': 0.9539501234622237}
----------
In [24]:
from collections import defaultdict
from surprise import SVD
from surprise import Dataset
def get_top_n(predictions, n=10):
''' Return the top-N recommendations for each user from a set of predictions.
Args:
predictions(list of Prediction objects): The list of predictions,
as
returned by the test method of an algorithm.
n(int): The number of recommendation to output for each user.
Default is 10.
Returns:
Adict where keys are users (raw) ids and values are lists of tuples:
[(raw item id, rating estimation), . . .] of size n.
'''
#first map the predictions to each other user.
top_n = defaultdict(list)
for uid, iid, true_r, est, _ in predictions:
top_n[uid].append((iid,est))
for uid, user_ratings in top_n.items():
user_ratings.sort(key=lambda x: x[1], reverse=True)
top_n[uid] = user_ratings[:n]
return top_n
#first train an SVD algorithm on the movielens dataset.
data= Dataset.load_builtin('ml-100k')
trainset = data.build_full_trainset()
algo=SVD()
algo.train(trainset)
#Than predict ratings for all pairs (u,i) that are NOT in the training set.
testset = trainset.build_anti_testset()
predictions = algo.test(testset)
top_n = get_top_n(predictions, n =10)
#print the recommended items for each user
for uid, user_ratings in top_n.items():
print(uid, [iid for (iid, _) in user_ratings])
/Users/apple/anaconda3/lib/python3.6/site-packages/surprise/prediction_algorithms/algo_base.py:51: UserWarning: train() is deprecated. Use fit() instead
warnings.warn('train() is deprecated. Use fit() instead', UserWarning)
196 ['483', '408', '169', '1449', '480', '427', '191', '174', '484', '657'] 186 ['316', '50', '318', '15', '483', '172', '315', '515', '513', '132'] 22 ['22', '520', '64', '483', '114', '318', '12', '169', '98', '496'] 244 ['178', '285', '408', '98', '14', '1142', '187', '190', '59', '127'] 166 ['169', '174', '114', '173', '50', '357', '64', '181', '408', '223'] 298 ['64', '12', '173', '313', '191', '169', '480', '96', '657', '408'] 115 ['179', '134', '180', '483', '168', '199', '175', '709', '114', '652'] 253 ['169', '302', '174', '178', '251', '498', '272', '313', '603', '480'] 305 ['603', '515', '114', '657', '132', '513', '525', '137', '1142', '493'] 6 ['654', '114', '179', '172', '603', '428', '657', '1142', '709', '659'] 62 ['408', '169', '223', '187', '496', '79', '902', '1142', '427', '647'] 286 ['178', '474', '12', '302', '64', '603', '657', '427', '513', '606'] 200 ['178', '64', '12', '480', '427', '181', '272', '603', '114', '963'] 210 ['64', '408', '169', '479', '427', '963', '603', '515', '285', '191'] 224 ['64', '79', '174', '169', '202', '95', '143', '651', '272', '114'] 303 ['180', '528', '190', '48', '169', '521', '887', '114', '481', '656'] 122 ['302', '89', '408', '479', '199', '114', '134', '173', '483', '654'] 194 ['603', '963', '480', '169', '484', '114', '430', '429', '14', '316'] 291 ['515', '127', '313', '318', '190', '357', '205', '203', '483', '520'] 234 ['302', '514', '251', '408', '169', '652', '315', '272', '529', '19'] 119 ['169', '483', '408', '923', '191', '652', '318', '57', '936', '114'] 167 ['408', '483', '515', '251', '194', '474', '705', '127', '611', '178'] 299 ['923', '178', '661', '657', '357', '963', '427', '654', '64', '272'] 308 ['173', '114', '963', '474', '272', '251', '923', '190', '1143', '529'] 95 ['313', '318', '408', '258', '169', '12', '435', '427', '251', '272'] 38 ['143', '269', '251', '64', '1449', '656', '612', '663', '427', '971'] 102 ['169', '114', '483', '318', '408', '488', '496', '603', '963', '1194'] 63 ['127', '12', '357', '56', '427', '474', '64', '318', '192', '195'] 160 ['12', '64', '134', '318', '654', '98', '709', '657', '963', '177'] 50 ['302', '474', '483', '408', '98', '318', '205', '127', '482', '603'] 301 ['169', '313', '302', '189', '603', '135', '316', '513', '165', '272'] 225 ['100', '483', '408', '56', '484', '191', '169', '513', '474', '661'] 290 ['173', '313', '169', '96', '79', '963', '178', '480', '479', '657'] 97 ['483', '315', '313', '1449', '272', '694', '1007', '963', '1039', '198'] 157 ['483', '408', '169', '480', '654', '114', '187', '657', '357', '488'] 181 ['313', '316', '79', '64', '735', '82', '204', '272', '169', '310'] 278 ['12', '169', '408', '50', '64', '134', '483', '199', '285', '45'] 276 ['134', '191', '177', '190', '178', '114', '520', '480', '651', '205'] 7 ['169', '518', '189', '276', '320', '735', '408', '705', '923', '114'] 10 ['318', '408', '114', '169', '515', '136', '520', '272', '83', '316'] 284 ['50', '318', '172', '64', '173', '12', '357', '483', '83', '603'] 201 ['484', '135', '246', '136', '205', '474', '14', '169', '177', '661'] 287 ['603', '483', '408', '134', '515', '96', '114', '318', '48', '657'] 246 ['7', '474', '530', '179', '302', '603', '479', '52', '127', '64'] 242 ['98', '603', '515', '427', '272', '79', '48', '483', '408', '178'] 249 ['474', '127', '134', '654', '285', '515', '488', '178', '482', '498'] 99 ['357', '169', '127', '408', '603', '114', '427', '195', '178', '318'] 178 ['169', '496', '272', '251', '408', '114', '963', '732', '515', '923'] 251 ['483', '408', '174', '357', '114', '318', '205', '127', '659', '134'] 81 ['313', '50', '173', '187', '174', '408', '64', '182', '114', '496'] 260 ['174', '172', '483', '318', '89', '169', '50', '114', '511', '478'] 25 ['64', '318', '272', '22', '313', '178', '515', '651', '963', '57'] 59 ['408', '246', '475', '248', '144', '154', '223', '1142', '693', '320'] 72 ['185', '178', '169', '216', '482', '14', '513', '483', '285', '659'] 87 ['173', '313', '114', '483', '143', '28', '1', '95', '478', '479'] 42 ['313', '22', '483', '480', '651', '114', '133', '520', '169', '511'] 292 ['134', '191', '318', '114', '89', '480', '513', '178', '170', '185'] 20 ['169', '59', '483', '923', '408', '178', '1449', '315', '657', '316'] 13 ['633', '923', '487', '1449', '641', '479', '169', '644', '496', '1142'] 138 ['408', '169', '657', '134', '923', '114', '64', '57', '528', '59'] 60 ['408', '169', '315', '923', '318', '114', '657', '1194', '189', '512'] 57 ['169', '657', '272', '172', '408', '483', '479', '489', '313', '174'] 223 ['174', '484', '64', '215', '50', '199', '316', '181', '114', '1007'] 189 ['427', '12', '515', '408', '1142', '169', '114', '488', '357', '633'] 243 ['408', '963', '134', '513', '169', '657', '56', '603', '921', '132'] 92 ['483', '357', '285', '603', '114', '127', '427', '480', '170', '520'] 241 ['318', '178', '483', '64', '169', '408', '98', '657', '357', '963'] 254 ['169', '318', '272', '114', '408', '651', '215', '173', '963', '705'] 293 ['276', '318', '408', '269', '191', '511', '428', '178', '656', '1142'] 127 ['302', '98', '408', '357', '169', '213', '114', '223', '12', '963'] 222 ['496', '203', '429', '963', '190', '272', '23', '114', '169', '1143'] 267 ['134', '513', '178', '357', '272', '285', '173', '603', '318', '511'] 11 ['483', '272', '408', '174', '50', '485', '172', '114', '482', '251'] 8 ['357', '192', '408', '114', '474', '318', '483', '191', '234', '100'] 162 ['408', '178', '12', '169', '513', '64', '318', '83', '480', '114'] 279 ['745', '513', '589', '475', '86', '896', '134', '45', '516', '921'] 145 ['127', '165', '357', '114', '210', '484', '520', '136', '531', '272'] 28 ['318', '64', '169', '275', '83', '496', '515', '357', '408', '190'] 135 ['169', '318', '408', '114', '313', '657', '285', '641', '199', '134'] 32 ['318', '64', '178', '98', '174', '496', '127', '603', '493', '143'] 90 ['114', '408', '169', '484', '513', '615', '487', '963', '165', '434'] 216 ['127', '178', '180', '89', '114', '603', '963', '483', '521', '484'] 250 ['285', '513', '408', '318', '493', '178', '156', '83', '61', '479'] 271 ['483', '1142', '408', '655', '966', '484', '251', '488', '114', '1007'] 265 ['169', '114', '657', '64', '318', '251', '272', '216', '603', '709'] 198 ['12', '408', '134', '114', '199', '169', '641', '479', '513', '483'] 168 ['64', '50', '318', '427', '174', '22', '483', '515', '172', '169'] 110 ['98', '318', '515', '483', '275', '478', '316', '480', '114', '136'] 58 ['23', '187', '86', '479', '707', '513', '285', '1039', '179', '170'] 237 ['515', '114', '272', '318', '60', '12', '641', '56', '316', '427'] 94 ['285', '276', '408', '661', '513', '48', '641', '251', '856', '169'] 128 ['127', '515', '513', '318', '357', '59', '474', '511', '169', '251'] 44 ['12', '408', '187', '199', '178', '519', '127', '169', '192', '302'] 264 ['178', '318', '64', '357', '199', '169', '50', '251', '223', '313'] 41 ['513', '408', '134', '114', '483', '12', '641', '251', '302', '479'] 82 ['313', '427', '114', '408', '48', '187', '173', '1194', '12', '923'] 262 ['694', '408', '474', '498', '134', '483', '513', '197', '430', '641'] 174 ['318', '64', '129', '48', '603', '258', '170', '180', '89', '275'] 43 ['515', '96', '87', '357', '22', '657', '603', '170', '136', '1019'] 84 ['483', '133', '657', '513', '169', '285', '493', '50', '474', '127'] 269 ['408', '709', '1021', '1142', '641', '199', '45', '324', '92', '14'] 259 ['127', '408', '182', '427', '480', '23', '513', '197', '114', '318'] 85 ['408', '302', '169', '114', '251', '178', '603', '12', '19', '285'] 213 ['169', '114', '427', '178', '272', '408', '137', '191', '315', '657'] 121 ['251', '178', '1142', '603', '483', '173', '187', '199', '64', '408'] 49 ['285', '474', '156', '169', '134', '603', '654', '408', '137', '276'] 155 ['483', '98', '318', '187', '178', '64', '169', '408', '173', '603'] 68 ['12', '483', '484', '408', '135', '603', '318', '168', '480', '474'] 172 ['169', '318', '98', '272', '114', '408', '357', '302', '12', '285'] 19 ['408', '169', '357', '603', '89', '657', '64', '528', '285', '963'] 268 ['23', '177', '187', '318', '430', '513', '45', '603', '48', '611'] 5 ['134', '175', '251', '199', '114', '177', '127', '7', '484', '654'] 80 ['318', '56', '963', '169', '127', '923', '12', '408', '173', '528'] 66 ['12', '64', '408', '963', '172', '174', '173', '657', '134', '513'] 18 ['657', '652', '641', '315', '124', '656', '316', '511', '606', '481'] 26 ['64', '318', '511', '272', '169', '22', '483', '357', '963', '251'] 130 ['483', '408', '318', '507', '15', '189', '114', '169', '657', '178'] 256 ['515', '483', '69', '318', '300', '496', '28', '530', '479', '313'] 1 ['519', '483', '408', '302', '511', '661', '427', '475', '313', '646'] 56 ['318', '496', '12', '187', '302', '251', '272', '357', '603', '313'] 15 ['318', '98', '427', '64', '511', '515', '196', '169', '191', '478'] 207 ['114', '496', '408', '136', '50', '651', '480', '603', '272', '169'] 232 ['923', '480', '124', '285', '12', '89', '134', '190', '513', '654'] 52 ['515', '483', '408', '178', '64', '357', '169', '50', '137', '320'] 161 ['603', '357', '169', '189', '114', '657', '963', '408', '285', '12'] 148 ['483', '480', '654', '963', '318', '657', '179', '835', '12', '513'] 125 ['265', '121', '228', '378', '736', '316', '96', '313', '515', '684'] 83 ['143', '318', '402', '12', '313', '87', '1019', '513', '257', '272'] 272 ['603', '64', '272', '169', '251', '408', '302', '479', '657', '223'] 151 ['285', '648', '527', '515', '1142', '493', '923', '656', '262', '116'] 54 ['408', '64', '169', '528', '191', '173', '198', '651', '134', '209'] 16 ['272', '82', '169', '173', '50', '651', '313', '187', '515', '210'] 91 ['12', '169', '272', '144', '190', '302', '114', '166', '316', '89'] 294 ['408', '12', '64', '318', '205', '98', '474', '357', '480', '192'] 229 ['483', '408', '127', '511', '192', '474', '134', '513', '963', '480'] 36 ['427', '480', '408', '169', '50', '172', '114', '483', '83', '474'] 70 ['318', '178', '64', '114', '427', '603', '98', '127', '480', '513'] 14 ['169', '512', '513', '134', '652', '641', '483', '1449', '199', '963'] 295 ['603', '272', '408', '178', '64', '199', '169', '57', '488', '607'] 233 ['170', '408', '479', '178', '651', '427', '657', '59', '1449', '190'] 214 ['923', '474', '657', '613', '480', '515', '1194', '709', '525', '963'] 192 ['12', '603', '318', '64', '173', '480', '483', '136', '357', '168'] 100 ['483', '474', '59', '169', '511', '484', '127', '641', '482', '408'] 307 ['318', '513', '134', '199', '357', '480', '493', '12', '45', '165'] 297 ['318', '408', '511', '483', '134', '513', '493', '169', '285', '661'] 193 ['64', '483', '498', '133', '408', '318', '191', '169', '132', '480'] 113 ['272', '114', '169', '64', '313', '318', '172', '408', '12', '174'] 275 ['603', '64', '483', '272', '316', '318', '178', '12', '114', '480'] 219 ['134', '408', '169', '514', '483', '89', '318', '474', '199', '19'] 218 ['169', '185', '285', '192', '408', '189', '318', '493', '302', '114'] 123 ['178', '603', '199', '357', '318', '641', '136', '59', '408', '170'] 158 ['318', '64', '357', '479', '641', '603', '178', '496', '1142', '480'] 302 ['64', '12', '357', '318', '169', '657', '127', '603', '178', '479'] 23 ['513', '169', '923', '168', '12', '114', '474', '285', '199', '657'] 296 ['169', '173', '408', '479', '318', '8', '512', '156', '496', '174'] 33 ['169', '173', '114', '172', '408', '64', '519', '963', '174', '272'] 154 ['127', '169', '318', '408', '134', '64', '603', '114', '513', '209'] 77 ['12', '169', '285', '178', '530', '408', '516', '603', '652', '657'] 270 ['1137', '515', '480', '496', '479', '272', '14', '483', '408', '178'] 187 ['408', '98', '480', '603', '483', '657', '169', '511', '519', '178'] 170 ['483', '408', '318', '169', '316', '498', '515', '603', '190', '114'] 101 ['64', '169', '318', '98', '603', '357', '313', '187', '427', '114'] 184 ['603', '242', '408', '136', '251', '316', '178', '513', '12', '320'] 112 ['483', '320', '318', '98', '169', '408', '493', '50', '1449', '198'] 133 ['408', '178', '12', '64', '318', '172', '474', '603', '173', '190'] 215 ['12', '114', '408', '178', '169', '603', '479', '427', '318', '493'] 69 ['64', '318', '178', '483', '251', '136', '127', '513', '316', '23'] 104 ['318', '64', '174', '169', '483', '172', '251', '178', '603', '515'] 240 ['169', '657', '318', '64', '12', '479', '515', '357', '408', '483'] 144 ['483', '427', '515', '408', '520', '498', '114', '736', '132', '496'] 191 ['169', '114', '963', '98', '483', '12', '50', '408', '64', '513'] 61 ['98', '64', '357', '408', '114', '923', '169', '654', '318', '12'] 142 ['64', '12', '318', '302', '127', '479', '603', '178', '480', '357'] 177 ['134', '474', '114', '285', '408', '603', '427', '923', '480', '185'] 203 ['483', '114', '318', '408', '98', '313', '169', '511', '174', '272'] 21 ['134', '318', '357', '180', '23', '191', '474', '238', '251', '192'] 197 ['173', '28', '64', '12', '204', '169', '318', '520', '191', '114'] 134 ['318', '169', '178', '134', '603', '98', '114', '168', '136', '496'] 180 ['647', '661', '408', '89', '23', '484', '114', '178', '1512', '507'] 236 ['408', '48', '357', '923', '197', '511', '513', '169', '603', '480'] 263 ['178', '493', '169', '251', '302', '513', '313', '12', '923', '114'] 109 ['300', '313', '114', '272', '1063', '746', '651', '515', '316', '611'] 64 ['408', '483', '169', '178', '192', '479', '134', '474', '657', '223'] 114 ['114', '408', '169', '302', '657', '603', '170', '963', '134', '513'] 239 ['285', '525', '60', '487', '813', '408', '709', '59', '169', '1194'] 117 ['127', '318', '89', '114', '408', '190', '513', '169', '59', '515'] 65 ['408', '169', '480', '272', '496', '189', '114', '483', '57', '657'] 137 ['98', '603', '196', '480', '483', '408', '318', '189', '357', '484'] 257 ['134', '318', '474', '114', '483', '302', '408', '513', '48', '197'] 111 ['318', '357', '64', '12', '483', '408', '474', '603', '480', '169'] 285 ['408', '50', '272', '483', '474', '114', '187', '511', '173', '318'] 96 ['313', '272', '302', '114', '651', '169', '22', '513', '199', '515'] 116 ['64', '114', '483', '192', '923', '169', '182', '208', '178', '134'] 73 ['169', '134', '408', '185', '513', '654', '45', '182', '963', '641'] 221 ['603', '114', '134', '89', '169', '654', '657', '357', '408', '963'] 235 ['169', '408', '641', '187', '134', '114', '180', '479', '178', '199'] 164 ['302', '98', '427', '272', '178', '318', '64', '484', '169', '12'] 281 ['169', '64', '318', '98', '408', '191', '963', '483', '272', '1'] 182 ['408', '483', '173', '478', '12', '480', '64', '513', '187', '192'] 129 ['408', '515', '483', '285', '357', '14', '427', '480', '603', '275'] 45 ['169', '318', '64', '272', '251', '408', '173', '12', '134', '313'] 131 ['169', '318', '12', '408', '134', '603', '178', '192', '191', '223'] 230 ['318', '520', '705', '272', '169', '735', '88', '178', '966', '194'] 126 ['318', '357', '408', '603', '285', '480', '178', '64', '483', '114'] 231 ['318', '169', '64', '515', '357', '603', '963', '98', '272', '480'] 280 ['169', '285', '603', '488', '657', '178', '134', '475', '45', '607'] 288 ['483', '496', '709', '603', '963', '285', '315', '705', '657', '655'] 152 ['302', '603', '427', '208', '174', '479', '172', '408', '178', '127'] 217 ['318', '705', '237', '484', '98', '178', '8', '611', '603', '515'] 79 ['318', '408', '178', '98', '657', '498', '963', '735', '479', '603'] 75 ['318', '603', '320', '178', '185', '195', '98', '480', '60', '23'] 245 ['169', '318', '657', '496', '64', '114', '173', '603', '98', '498'] 282 ['318', '169', '488', '79', '178', '483', '657', '272', '174', '923'] 78 ['318', '169', '603', '12', '408', '178', '98', '484', '64', '515'] 118 ['242', '302', '144', '194', '196', '515', '165', '480', '195', '479'] 283 ['98', '272', '357', '114', '603', '169', '64', '483', '12', '318'] 171 ['50', '169', '12', '114', '172', '408', '178', '483', '318', '480'] 107 ['127', '408', '483', '427', '318', '705', '654', '173', '12', '603'] 226 ['313', '603', '318', '64', '272', '496', '529', '173', '483', '515'] 306 ['483', '64', '302', '515', '12', '134', '127', '169', '272', '114'] 173 ['474', '603', '515', '275', '479', '272', '483', '408', '178', '318'] 185 ['483', '496', '408', '98', '603', '12', '64', '170', '136', '169'] 150 ['172', '483', '318', '169', '408', '515', '168', '174', '694', '357'] 274 ['603', '515', '408', '127', '64', '357', '169', '511', '513', '963'] 188 ['496', '258', '169', '12', '963', '520', '603', '479', '408', '114'] 48 ['64', '12', '114', '169', '318', '272', '127', '173', '408', '23'] 311 ['169', '114', '408', '313', '190', '316', '963', '246', '531', '57'] 165 ['483', '408', '114', '59', '480', '657', '189', '357', '313', '56'] 208 ['50', '173', '64', '318', '169', '174', '178', '483', '496', '22'] 2 ['318', '357', '169', '114', '483', '408', '165', '64', '657', '515'] 205 ['127', '98', '64', '169', '318', '511', '56', '12', '357', '408'] 248 ['408', '56', '134', '357', '603', '12', '192', '654', '318', '528'] 93 ['483', '318', '408', '178', '606', '513', '169', '313', '603', '709'] 159 ['169', '313', '513', '189', '98', '135', '181', '114', '519', '315'] 146 ['169', '98', '408', '519', '357', '511', '603', '478', '611', '64'] 29 ['408', '134', '64', '515', '169', '603', '483', '172', '318', '48'] 156 ['483', '114', '513', '427', '408', '511', '199', '603', '498', '316'] 37 ['408', '318', '114', '427', '64', '479', '199', '98', '178', '12'] 141 ['316', '241', '478', '513', '57', '98', '136', '660', '493', '59'] 195 ['318', '174', '64', '178', '114', '50', '172', '357', '302', '511'] 108 ['318', '408', '483', '178', '657', '169', '480', '488', '603', '64'] 47 ['427', '483', '318', '408', '606', '143', '114', '170', '136', '272'] 255 ['156', '127', '134', '318', '192', '205', '475', '114', '64', '408'] 89 ['318', '186', '520', '205', '178', '484', '659', '204', '134', '199'] 140 ['318', '483', '603', '657', '479', '408', '285', '178', '174', '169'] 190 ['169', '98', '318', '408', '12', '189', '174', '64', '190', '1449'] 24 ['408', '169', '50', '187', '114', '657', '483', '251', '189', '474'] 17 ['318', '169', '190', '474', '178', '189', '12', '23', '136', '408'] 313 ['302', '408', '513', '12', '169', '963', '709', '530', '656', '179'] 53 ['318', '408', '357', '483', '496', '427', '603', '12', '479', '657'] 124 ['169', '134', '513', '408', '923', '114', '603', '1449', '178', '511'] 149 ['408', '169', '483', '50', '189', '511', '205', '187', '318', '64'] 176 ['408', '427', '169', '480', '657', '474', '488', '603', '318', '9'] 106 ['408', '603', '357', '169', '173', '515', '272', '114', '483', '170'] 312 ['318', '12', '127', '302', '1194', '251', '192', '129', '64', '963'] 175 ['427', '408', '318', '174', '173', '480', '963', '169', '479', '482'] 153 ['318', '603', '275', '285', '408', '98', '185', '513', '923', '657'] 220 ['318', '408', '169', '963', '657', '483', '484', '12', '114', '8'] 143 ['496', '178', '169', '603', '50', '480', '12', '98', '479', '173'] 199 ['127', '50', '483', '357', '12', '603', '187', '318', '1142', '173'] 202 ['169', '12', '513', '50', '511', '181', '272', '178', '98', '519'] 277 ['114', '56', '483', '408', '357', '169', '64', '318', '98', '657'] 206 ['191', '318', '187', '192', '50', '408', '652', '480', '651', '487'] 76 ['483', '45', '134', '187', '114', '127', '132', '515', '408', '251'] 314 ['265', '272', '79', '174', '315', '222', '515', '181', '82', '313'] 136 ['408', '483', '178', '923', '480', '513', '12', '657', '114', '285'] 179 ['114', '742', '357', '12', '64', '183', '515', '169', '318', '659'] 4 ['474', '98', '603', '100', '515', '498', '23', '479', '272', '48'] 304 ['98', '318', '316', '56', '603', '483', '114', '496', '484', '100'] 3 ['483', '178', '285', '427', '408', '50', '1449', '603', '169', '315'] 227 ['483', '114', '134', '169', '135', '64', '408', '170', '603', '519'] 252 ['302', '272', '172', '483', '408', '127', '318', '134', '169', '50'] 212 ['408', '169', '251', '64', '12', '513', '923', '57', '114', '189'] 310 ['127', '318', '408', '114', '137', '480', '603', '190', '316', '357'] 35 ['174', '318', '22', '169', '408', '176', '8', '114', '515', '480'] 147 ['408', '318', '64', '480', '513', '169', '357', '657', '132', '12'] 105 ['64', '22', '172', '483', '408', '479', '127', '50', '515', '174'] 34 ['302', '474', '603', '427', '479', '408', '178', '318', '89', '64'] 71 ['408', '318', '169', '114', '603', '515', '963', '427', '272', '661'] 51 ['114', '313', '169', '272', '174', '408', '12', '651', '22', '318'] 204 ['483', '127', '100', '603', '427', '511', '963', '98', '474', '479'] 315 ['408', '169', '313', '59', '474', '134', '320', '199', '483', '251'] 31 ['408', '483', '474', '169', '12', '50', '178', '132', '316', '133'] 316 ['408', '56', '285', '315', '136', '603', '320', '606', '124', '705'] 103 ['318', '513', '192', '408', '511', '178', '114', '124', '12', '641'] 318 ['515', '496', '178', '316', '603', '251', '59', '258', '272', '483'] 30 ['64', '318', '100', '515', '483', '496', '169', '603', '127', '196'] 120 ['178', '657', '64', '169', '285', '12', '408', '114', '318', '480'] 46 ['408', '169', '12', '251', '483', '923', '64', '114', '513', '318'] 289 ['169', '641', '408', '528', '114', '357', '189', '64', '200', '190'] 209 ['187', '12', '64', '169', '483', '511', '172', '174', '513', '357'] 261 ['194', '603', '181', '427', '480', '498', '23', '496', '132', '174'] 88 ['480', '513', '483', '197', '168', '114', '59', '318', '408', '498'] 9 ['408', '923', '657', '98', '132', '1194', '493', '59', '194', '357'] 247 ['483', '408', '169', '89', '480', '318', '114', '172', '12', '528'] 321 ['114', '408', '169', '963', '98', '183', '189', '520', '313', '488'] 266 ['483', '513', '127', '480', '169', '114', '511', '488', '963', '191'] 74 ['50', '169', '483', '172', '114', '408', '174', '181', '963', '515'] 238 ['127', '169', '474', '318', '408', '513', '511', '187', '134', '302'] 319 ['169', '12', '178', '408', '89', '318', '59', '357', '64', '190'] 323 ['12', '187', '357', '427', '114', '474', '483', '408', '195', '174'] 67 ['318', '483', '513', '408', '923', '963', '57', '474', '98', '511'] 211 ['483', '50', '318', '408', '530', '169', '48', '427', '134', '315'] 98 ['64', '408', '513', '657', '169', '511', '191', '483', '496', '427'] 12 ['64', '169', '313', '603', '427', '657', '515', '178', '165', '483'] 40 ['318', '408', '64', '169', '313', '515', '480', '496', '50', '114'] 258 ['169', '114', '318', '408', '963', '195', '56', '60', '174', '1449'] 228 ['488', '169', '408', '318', '963', '189', '114', '12', '513', '50'] 325 ['427', '603', '56', '479', '709', '513', '663', '615', '633', '315'] 320 ['114', '515', '408', '251', '923', '513', '316', '169', '483', '168'] 326 ['607', '313', '189', '169', '661', '12', '303', '143', '408', '487'] 327 ['483', '187', '134', '124', '488', '511', '641', '963', '656', '213'] 183 ['318', '169', '313', '114', '514', '408', '603', '515', '1449', '657'] 328 ['169', '923', '173', '114', '1', '735', '178', '484', '208', '242'] 322 ['187', '408', '191', '169', '480', '178', '474', '481', '316', '488'] 330 ['302', '272', '483', '408', '178', '705', '191', '169', '611', '251'] 27 ['64', '12', '169', '22', '606', '1449', '603', '408', '318', '427'] 331 ['272', '169', '89', '408', '318', '12', '320', '313', '196', '114'] 332 ['603', '515', '496', '272', '408', '318', '153', '189', '357', '191'] 329 ['114', '408', '64', '511', '357', '318', '315', '132', '285', '513'] 86 ['483', '474', '98', '50', '513', '603', '493', '316', '114', '408'] 139 ['408', '134', '513', '603', '709', '114', '923', '474', '272', '189'] 300 ['14', '483', '408', '318', '64', '657', '285', '515', '272', '603'] 163 ['50', '114', '172', '408', '169', '603', '515', '181', '513', '174'] 333 ['169', '408', '12', '603', '64', '318', '114', '272', '251', '606'] 334 ['178', '654', '114', '199', '923', '251', '136', '613', '661', '646'] 39 ['318', '12', '408', '483', '8', '169', '657', '64', '56', '194'] 324 ['302', '474', '144', '98', '603', '100', '515', '427', '512', '165'] 132 ['318', '169', '64', '408', '180', '357', '98', '114', '603', '488'] 336 ['98', '12', '64', '169', '174', '318', '205', '963', '172', '195'] 335 ['98', '169', '318', '657', '408', '180', '114', '96', '603', '178'] 169 ['173', '12', '318', '408', '114', '169', '89', '641', '357', '272'] 338 ['318', '302', '209', '89', '199', '285', '114', '652', '165', '357'] 339 ['408', '285', '641', '318', '169', '114', '251', '1142', '48', '659'] 309 ['173', '318', '169', '187', '12', '50', '313', '408', '479', '192'] 342 ['474', '1', '302', '127', '272', '180', '64', '187', '1039', '705'] 340 ['313', '64', '22', '318', '98', '12', '191', '169', '513', '316'] 317 ['515', '408', '64', '318', '169', '483', '302', '12', '513', '114'] 341 ['98', '515', '174', '272', '408', '127', '318', '56', '64', '357'] 343 ['178', '285', '14', '603', '433', '479', '151', '172', '430', '173'] 344 ['474', '483', '134', '23', '603', '480', '168', '427', '48', '654'] 345 ['483', '127', '427', '654', '134', '169', '114', '357', '657', '603'] 346 ['258', '408', '194', '69', '228', '82', '114', '205', '483', '59'] 347 ['963', '64', '169', '651', '205', '190', '483', '114', '272', '313'] 273 ['408', '12', '173', '318', '169', '64', '657', '603', '251', '479'] 55 ['483', '172', '408', '12', '64', '198', '705', '318', '96', '272'] 349 ['408', '169', '12', '657', '603', '480', '318', '474', '483', '427'] 348 ['408', '169', '12', '963', '316', '272', '198', '98', '114', '187'] 354 ['923', '408', '474', '641', '856', '921', '647', '525', '132', '493'] 351 ['515', '272', '172', '408', '318', '64', '191', '183', '430', '134'] 358 ['98', '427', '483', '480', '169', '1449', '479', '530', '497', '515'] 352 ['169', '313', '285', '318', '357', '275', '613', '483', '408', '189'] 360 ['513', '408', '923', '318', '169', '178', '694', '98', '48', '180'] 363 ['357', '165', '484', '199', '211', '607', '513', '633', '480', '318'] 355 ['98', '603', '480', '174', '172', '483', '408', '178', '127', '318'] 362 ['483', '12', '98', '127', '191', '64', '190', '603', '134', '357'] 357 ['242', '302', '427', '165', '496', '174', '479', '272', '79', '172'] 356 ['64', '169', '318', '98', '496', '357', '603', '174', '114', '408'] 361 ['318', '483', '134', '474', '408', '64', '511', '427', '127', '480'] 365 ['318', '357', '603', '178', '251', '64', '69', '11', '480', '189'] 350 ['408', '318', '114', '474', '357', '641', '513', '169', '134', '923'] 367 ['474', '603', '427', '480', '132', '479', '48', '483', '408', '178'] 368 ['408', '169', '654', '657', '513', '483', '178', '603', '528', '198'] 371 ['515', '56', '318', '272', '169', '963', '187', '313', '28', '408'] 373 ['408', '313', '272', '902', '513', '223', '515', '1194', '484', '98'] 370 ['408', '474', '513', '318', '169', '178', '515', '272', '488', '519'] 374 ['496', '300', '275', '208', '313', '169', '484', '520', '298', '378'] 372 ['222', '603', '1137', '181', '515', '480', '174', '96', '479', '272'] 337 ['169', '408', '483', '12', '168', '318', '64', '657', '480', '178'] 378 ['483', '187', '127', '408', '169', '603', '190', '114', '192', '208'] 366 ['302', '474', '603', '515', '480', '498', '479', '272', '483', '408'] 377 ['169', '318', '408', '12', '64', '114', '187', '515', '483', '50'] 375 ['64', '12', '318', '127', '191', '98', '173', '169', '22', '483'] 359 ['318', '12', '169', '172', '114', '189', '963', '178', '89', '96'] 379 ['483', '313', '408', '484', '48', '482', '318', '493', '479', '923'] 380 ['178', '657', '513', '484', '408', '285', '603', '169', '173', '606'] 381 ['474', '515', '408', '603', '12', '302', '114', '488', '199', '136'] 385 ['709', '45', '124', '963', '856', '641', '923', '302', '1194', '519'] 382 ['45', '169', '656', '963', '483', '285', '251', '515', '132', '178'] 387 ['134', '654', '276', '221', '1194', '137', '709', '657', '302', '483'] 364 ['408', '169', '64', '318', '12', '114', '98', '515', '474', '483'] 369 ['408', '169', '963', '318', '174', '12', '178', '98', '64', '89'] 388 ['181', '480', '174', '79', '172', '408', '178', '318', '520', '64'] 386 ['169', '114', '318', '185', '657', '12', '64', '480', '483', '357'] 389 ['408', '511', '180', '169', '528', '606', '647', '357', '114', '1194'] 383 ['408', '318', '12', '515', '114', '190', '64', '136', '511', '589'] 390 ['318', '64', '133', '483', '408', '114', '313', '22', '178', '520'] 393 ['493', '251', '478', '853', '408', '513', '515', '601', '199', '520'] 392 ['427', '483', '132', '606', '187', '124', '22', '64', '498', '479'] 376 ['483', '318', '169', '408', '657', '64', '174', '178', '479', '474'] 394 ['318', '64', '483', '100', '357', '127', '268', '408', '199', '258'] 391 ['302', '408', '169', '315', '519', '515', '178', '14', '285', '124'] 398 ['143', '318', '313', '114', '651', '316', '190', '272', '515', '83'] 397 ['98', '511', '114', '64', '603', '187', '923', '169', '203', '191'] 399 ['1449', '604', '306', '272', '923', '921', '485', '652', '313', '316'] 396 ['114', '98', '318', '64', '169', '272', '313', '174', '515', '611'] 401 ['79', '98', '427', '408', '190', '496', '320', '234', '129', '480'] 402 ['318', '114', '64', '134', '191', '56', '178', '514', '603', '133'] 384 ['408', '169', '513', '474', '318', '511', '923', '114', '357', '479'] 395 ['169', '114', '357', '520', '483', '12', '963', '195', '173', '480'] 353 ['483', '169', '98', '318', '12', '64', '923', '408', '513', '114'] 403 ['313', '64', '169', '498', '174', '194', '430', '484', '483', '963'] 405 ['144', '496', '705', '1142', '237', '223', '283', '1134', '835', '165'] 400 ['50', '12', '172', '174', '64', '408', '169', '22', '183', '483'] 406 ['530', '302', '484', '178', '169', '114', '408', '963', '189', '659'] 407 ['64', '114', '12', '318', '170', '178', '251', '192', '285', '187'] 409 ['408', '114', '185', '641', '513', '64', '656', '182', '1203', '177'] 404 ['64', '199', '98', '169', '603', '479', '408', '513', '519', '318'] 413 ['318', '169', '483', '408', '174', '127', '64', '98', '496', '479'] 416 ['57', '169', '408', '208', '48', '923', '1063', '114', '489', '963'] 408 ['318', '50', '484', '316', '483', '408', '513', '191', '178', '169'] 410 ['169', '50', '1449', '474', '134', '408', '515', '174', '114', '520'] 411 ['12', '408', '64', '483', '251', '178', '478', '357', '316', '480'] 417 ['318', '114', '480', '408', '313', '9', '482', '478', '603', '272'] 412 ['530', '272', '60', '483', '79', '134', '313', '963', '513', '192'] 420 ['318', '64', '169', '357', '427', '185', '98', '178', '483', '474'] 422 ['357', '285', '408', '603', '169', '64', '56', '318', '12', '251'] 425 ['23', '116', '179', '246', '114', '963', '603', '919', '496', '408'] 419 ['498', '178', '172', '483', '251', '603', '64', '169', '136', '496'] 415 ['408', '318', '12', '50', '114', '357', '169', '64', '313', '89'] 423 ['408', '169', '64', '12', '318', '603', '187', '98', '178', '483'] 429 ['408', '661', '169', '251', '272', '187', '135', '515', '615', '1142'] 428 ['22', '190', '483', '318', '79', '64', '98', '132', '963', '127'] 427 ['474', '86', '257', '144', '246', '98', '194', '603', '1137', '181'] 418 ['408', '169', '64', '318', '185', '178', '12', '480', '205', '484'] 424 ['483', '603', '513', '316', '98', '641', '963', '197', '285', '1449'] 432 ['98', '318', '357', '64', '272', '483', '515', '427', '603', '651'] 421 ['318', '515', '408', '114', '169', '189', '480', '316', '483', '357'] 435 ['475', '205', '508', '192', '134', '320', '223', '272', '340', '483'] 433 ['285', '169', '408', '528', '134', '242', '483', '127', '603', '427'] 426 ['302', '114', '272', '169', '408', '165', '611', '179', '479', '124'] 436 ['408', '318', '56', '114', '272', '69', '169', '79', '589', '12'] 430 ['483', '408', '480', '427', '114', '496', '603', '357', '657', '197'] 434 ['169', '114', '408', '265', '318', '496', '174', '64', '515', '516'] 437 ['408', '285', '251', '199', '194', '178', '357', '427', '242', '528'] 438 ['169', '64', '272', '187', '313', '320', '408', '173', '189', '478'] 431 ['169', '408', '483', '114', '513', '515', '50', '357', '272', '56'] 442 ['408', '151', '173', '50', '169', '478', '246', '357', '474', '114'] 440 ['302', '100', '427', '132', '48', '408', '178', '318', '64', '169'] 445 ['182', '427', '318', '483', '134', '651', '190', '169', '357', '114'] 447 ['64', '357', '603', '318', '114', '480', '172', '530', '408', '178'] 449 ['169', '427', '199', '318', '191', '156', '98', '408', '50', '134'] 450 ['408', '515', '615', '165', '641', '189', '57', '963', '116', '1007'] 446 ['169', '408', '483', '64', '511', '22', '50', '272', '187', '427'] 439 ['190', '169', '408', '963', '64', '515', '178', '174', '483', '318'] 451 ['313', '192', '272', '98', '298', '89', '114', '187', '186', '180'] 452 ['313', '511', '357', '169', '178', '484', '114', '1007', '493', '215'] 454 ['739', '265', '208', '192', '187', '515', '966', '320', '684', '408'] 453 ['408', '173', '169', '96', '134', '64', '192', '114', '127', '603'] 414 ['318', '64', '12', '187', '191', '98', '513', '192', '479', '132'] 455 ['483', '316', '408', '169', '187', '963', '357', '114', '208', '661'] 444 ['357', '169', '427', '474', '98', '603', '302', '513', '114', '127'] 448 ['483', '408', '313', '178', '479', '498', '169', '251', '127', '474'] 457 ['408', '480', '427', '736', '1142', '520', '496', '963', '498', '30'] 456 ['408', '137', '285', '318', '488', '134', '89', '654', '340', '178'] 458 ['641', '657', '923', '59', '661', '1143', '60', '114', '479', '177'] 462 ['169', '318', '98', '174', '189', '511', '64', '483', '530', '408'] 459 ['316', '313', '963', '176', '59', '315', '513', '1449', '118', '483'] 460 ['408', '134', '514', '498', '60', '169', '89', '511', '923', '603'] 461 ['357', '64', '318', '480', '408', '114', '427', '98', '14', '603'] 467 ['483', '64', '12', '285', '134', '187', '191', '168', '178', '603'] 468 ['483', '474', '187', '169', '408', '114', '480', '479', '482', '484'] 466 ['474', '408', '493', '603', '483', '318', '427', '190', '169', '480'] 472 ['302', '515', '165', '514', '479', '408', '178', '89', '430', '134'] 465 ['168', '480', '313', '178', '483', '316', '923', '89', '493', '83'] 463 ['89', '64', '318', '169', '12', '483', '174', '272', '408', '134'] 471 ['173', '251', '169', '318', '178', '513', '515', '479', '60', '603'] 474 ['169', '114', '153', '408', '272', '189', '512', '429', '887', '223'] 469 ['480', '408', '657', '169', '114', '316', '12', '178', '89', '357'] 464 ['64', '169', '496', '178', '408', '98', '1039', '963', '275', '318'] 476 ['318', '114', '64', '98', '484', '178', '603', '483', '496', '169'] 478 ['285', '127', '275', '603', '483', '474', '83', '136', '59', '480'] 473 ['318', '114', '64', '174', '169', '408', '603', '272', '12', '187'] 470 ['318', '357', '169', '185', '174', '313', '98', '483', '408', '527'] 480 ['408', '357', '488', '199', '285', '124', '318', '189', '963', '474'] 441 ['64', '483', '169', '114', '173', '603', '515', '657', '427', '178'] 479 ['64', '191', '194', '12', '251', '515', '513', '124', '134', '114'] 484 ['64', '483', '496', '511', '515', '169', '302', '480', '178', '479'] 486 ['98', '223', '56', '483', '318', '488', '357', '603', '8', '496'] 487 ['169', '515', '520', '963', '435', '511', '519', '496', '124', '316'] 482 ['483', '114', '515', '169', '480', '408', '318', '199', '498', '484'] 481 ['483', '114', '641', '408', '511', '478', '302', '172', '513', '963'] 492 ['169', '50', '408', '12', '427', '22', '484', '513', '251', '357'] 493 ['8', '169', '313', '963', '153', '64', '189', '484', '114', '187'] 490 ['114', '513', '12', '511', '483', '172', '488', '169', '174', '408'] 489 ['603', '169', '488', '114', '178', '963', '318', '79', '746', '611'] 483 ['483', '172', '316', '169', '408', '251', '478', '178', '64', '479'] 496 ['134', '187', '200', '408', '654', '45', '59', '963', '513', '169'] 494 ['318', '185', '169', '12', '272', '172', '313', '195', '408', '22'] 495 ['474', '483', '169', '315', '963', '59', '316', '136', '484', '603'] 477 ['302', '474', '265', '144', '1', '98', '194', '603', '1137', '100'] 497 ['272', '483', '59', '318', '199', '963', '198', '64', '302', '513'] 488 ['313', '12', '316', '272', '735', '204', '114', '282', '121', '169'] 498 ['285', '178', '408', '169', '529', '921', '1194', '114', '189', '45'] 499 ['169', '408', '515', '64', '705', '320', '603', '302', '114', '170'] 491 ['169', '318', '963', '87', '64', '98', '89', '57', '302', '189'] 500 ['488', '603', '654', '209', '513', '23', '173', '484', '192', '525'] 502 ['169', '12', '251', '178', '611', '50', '496', '320', '530', '488'] 503 ['474', '209', '483', '179', '408', '647', '513', '320', '56', '511'] 504 ['513', '923', '272', '64', '169', '408', '87', '83', '114', '265'] 505 ['143', '318', '603', '15', '657', '483', '64', '485', '306', '272'] 506 ['313', '64', '22', '318', '144', '127', '143', '483', '169', '237'] 443 ['318', '64', '169', '408', '190', '22', '50', '79', '187', '192'] 507 ['242', '474', '265', '86', '1', '98', '193', '194', '603', '1137'] 514 ['427', '705', '480', '606', '1449', '519', '736', '187', '657', '513'] 508 ['100', '408', '657', '169', '483', '127', '178', '302', '641', '12'] 511 ['603', '408', '127', '318', '56', '64', '357', '169', '170', '132'] 515 ['114', '169', '12', '963', '195', '515', '603', '64', '183', '190'] 512 ['515', '480', '174', '483', '408', '269', '169', '603', '657', '357'] 513 ['98', '174', '272', '483', '318', '64', '134', '169', '190', '511'] 475 ['169', '427', '963', '408', '174', '22', '513', '357', '98', '511'] 523 ['515', '427', '275', '174', '479', '79', '483', '178', '127', '318'] 518 ['313', '496', '174', '318', '169', '50', '408', '96', '114', '64'] 509 ['12', '178', '172', '185', '169', '474', '657', '114', '318', '251'] 516 ['318', '483', '408', '114', '1449', '313', '923', '633', '189', '520'] 510 ['408', '169', '114', '515', '318', '22', '64', '272', '513', '657'] 524 ['408', '45', '223', '659', '114', '694', '1137', '166', '963', '357'] 501 ['474', '318', '285', '408', '178', '169', '64', '657', '136', '513'] 525 ['169', '483', '318', '408', '98', '64', '174', '50', '89', '515'] 521 ['408', '483', '654', '192', '318', '169', '198', '519', '513', '205'] 520 ['318', '408', '246', '64', '272', '268', '12', '89', '496', '313'] 519 ['242', '302', '474', '265', '486', '95', '234', '98', '194', '603'] 528 ['408', '114', '172', '963', '318', '98', '169', '64', '165', '513'] 532 ['144', '174', '69', '64', '114', '657', '223', '651', '963', '185'] 530 ['515', '318', '272', '496', '603', '498', '12', '657', '251', '923'] 531 ['318', '64', '169', '133', '483', '408', '178', '481', '657', '190'] 529 ['114', '169', '313', '173', '318', '408', '515', '190', '272', '199'] 517 ['408', '169', '511', '199', '318', '513', '133', '64', '493', '12'] 527 ['519', '114', '166', '251', '408', '199', '272', '430', '641', '483'] 485 ['408', '318', '483', '12', '114', '64', '50', '100', '127', '169'] 533 ['520', '173', '963', '114', '657', '79', '515', '316', '272', '750'] 535 ['408', '251', '175', '208', '57', '169', '1007', '641', '694', '647'] 536 ['12', '64', '173', '127', '520', '526', '651', '178', '463', '479'] 526 ['483', '169', '134', '170', '654', '12', '223', '114', '511', '528'] 537 ['408', '114', '659', '251', '154', '589', '522', '856', '223', '169'] 534 ['98', '603', '181', '515', '480', '498', '496', '174', '485', '172'] 541 ['169', '64', '357', '272', '515', '318', '520', '12', '513', '114'] 538 ['114', '408', '313', '482', '64', '302', '603', '178', '427', '657'] 542 ['408', '515', '483', '114', '135', '474', '603', '480', '170', '98'] 545 ['408', '318', '313', '64', '272', '178', '114', '169', '1449', '316'] 539 ['169', '173', '64', '479', '1', '408', '498', '515', '318', '251'] 547 ['318', '114', '187', '64', '50', '357', '178', '169', '513', '174'] 543 ['483', '963', '657', '127', '408', '285', '606', '589', '498', '641'] 548 ['174', '96', '195', '144', '169', '22', '210', '513', '173', '318'] 546 ['496', '479', '513', '174', '603', '178', '79', '408', '127', '474'] 522 ['357', '483', '603', '178', '169', '127', '474', '513', '408', '498'] 551 ['114', '923', '298', '633', '60', '1143', '474', '611', '657', '306'] 544 ['318', '427', '169', '480', '64', '357', '114', '483', '603', '408'] 553 ['169', '12', '408', '114', '318', '923', '302', '493', '316', '272'] 552 ['318', '169', '408', '483', '316', '657', '178', '498', '64', '513'] 540 ['483', '318', '408', '127', '12', '178', '657', '169', '56', '192'] 554 ['64', '313', '408', '169', '205', '483', '114', '493', '12', '963'] 550 ['64', '169', '173', '12', '408', '318', '127', '174', '134', '172'] 556 ['511', '285', '408', '50', '169', '357', '8', '657', '199', '89'] 559 ['474', '114', '408', '530', '498', '484', '313', '169', '134', '480'] 560 ['357', '474', '408', '180', '169', '272', '511', '199', '251', '172'] 561 ['134', '408', '169', '647', '654', '192', '60', '641', '525', '519'] 563 ['318', '408', '272', '483', '285', '528', '187', '169', '64', '189'] 566 ['276', '185', '169', '60', '190', '603', '513', '661', '923', '475'] 557 ['318', '357', '169', '657', '408', '114', '515', '64', '603', '496'] 558 ['169', '64', '50', '603', '178', '408', '12', '483', '657', '172'] 564 ['483', '173', '114', '408', '174', '318', '923', '64', '493', '172'] 565 ['194', '603', '480', '496', '483', '408', '178', '318', '484', '191'] 573 ['408', '56', '511', '169', '114', '647', '923', '251', '474', '187'] 549 ['64', '318', '12', '169', '272', '173', '408', '114', '178', '479'] 567 ['923', '529', '408', '285', '921', '169', '652', '515', '516', '315'] 569 ['408', '318', '496', '64', '483', '427', '515', '357', '170', '83'] 562 ['12', '178', '22', '498', '64', '705', '265', '484', '530', '192'] 576 ['64', '318', '169', '173', '313', '408', '515', '174', '12', '114'] 577 ['272', '169', '487', '528', '199', '923', '315', '124', '178', '408'] 579 ['64', '178', '272', '12', '483', '318', '251', '479', '519', '515'] 574 ['64', '483', '174', '318', '169', '208', '191', '127', '528', '96'] 555 ['408', '483', '191', '178', '515', '187', '313', '170', '427', '64'] 572 ['483', '603', '479', '480', '187', '515', '318', '127', '408', '313'] 575 ['408', '169', '515', '12', '657', '170', '190', '480', '114', '134'] 584 ['318', '408', '64', '169', '515', '59', '178', '496', '174', '127'] 588 ['513', '59', '478', '408', '190', '516', '507', '736', '187', '1512'] 587 ['98', '64', '603', '127', '178', '408', '963', '12', '185', '519'] 568 ['408', '302', '98', '513', '357', '170', '251', '480', '318', '607'] 586 ['513', '251', '272', '963', '515', '408', '114', '478', '483', '1007'] 585 ['483', '408', '64', '318', '22', '272', '174', '89', '173', '515'] 582 ['127', '474', '12', '191', '98', '169', '603', '408', '178', '285'] 591 ['178', '169', '483', '515', '657', '199', '408', '513', '134', '318'] 581 ['178', '483', '12', '408', '513', '709', '963', '603', '480', '479'] 592 ['208', '190', '641', '693', '651', '474', '136', '661', '435', '584'] 580 ['408', '12', '483', '178', '172', '1142', '528', '488', '272', '694'] 590 ['408', '169', '427', '12', '603', '178', '474', '318', '657', '64'] 593 ['64', '186', '520', '197', '187', '114', '89', '651', '178', '519'] 583 ['98', '318', '64', '484', '603', '479', '173', '136', '185', '169'] 596 ['318', '169', '483', '480', '12', '173', '408', '178', '659', '191'] 570 ['169', '114', '483', '530', '408', '657', '474', '183', '603', '64'] 599 ['114', '173', '272', '98', '318', '659', '313', '603', '515', '408'] 589 ['22', '50', '513', '1449', '174', '265', '178', '169', '483', '98'] 594 ['64', '318', '169', '963', '408', '98', '178', '427', '272', '513'] 597 ['169', '64', '318', '12', '427', '963', '98', '603', '408', '357'] 578 ['408', '169', '1449', '483', '513', '511', '480', '189', '657', '178'] 601 ['175', '408', '654', '209', '169', '134', '7', '192', '180', '187'] 602 ['318', '114', '480', '178', '483', '191', '313', '79', '180', '187'] 600 ['64', '114', '169', '190', '357', '272', '180', '313', '318', '12'] 605 ['56', '134', '199', '511', '515', '169', '285', '178', '45', '479'] 603 ['169', '408', '178', '483', '114', '480', '318', '963', '316', '302'] 595 ['357', '64', '318', '169', '285', '408', '12', '483', '172', '173'] 606 ['318', '357', '114', '408', '603', '272', '170', '482', '515', '657'] 608 ['496', '170', '203', '963', '12', '652', '657', '922', '1137', '57'] 607 ['127', '318', '408', '114', '169', '192', '89', '134', '654', '357'] 610 ['408', '513', '132', '515', '192', '64', '306', '114', '641', '169'] 611 ['64', '483', '357', '963', '59', '169', '12', '127', '408', '603'] 617 ['318', '513', '12', '165', '603', '633', '483', '114', '199', '408'] 618 ['511', '520', '169', '480', '114', '205', '272', '604', '316', '435'] 614 ['127', '272', '50', '251', '169', '474', '114', '134', '408', '172'] 609 ['318', '114', '169', '272', '98', '64', '57', '275', '427', '511'] 615 ['64', '169', '408', '50', '173', '313', '114', '98', '285', '657'] 616 ['169', '64', '408', '98', '318', '483', '285', '114', '513', '515'] 620 ['515', '483', '178', '286', '96', '64', '523', '22', '187', '143'] 571 ['318', '474', '127', '408', '169', '12', '178', '488', '513', '613'] 619 ['484', '64', '318', '210', '12', '169', '204', '661', '223', '483'] 613 ['169', '408', '199', '251', '134', '178', '180', '313', '114', '647'] 622 ['134', '114', '187', '657', '318', '182', '357', '272', '813', '513'] 621 ['408', '318', '114', '513', '169', '12', '251', '654', '178', '474'] 604 ['173', '174', '64', '483', '202', '172', '313', '50', '479', '79'] 624 ['318', '64', '178', '408', '12', '169', '528', '191', '613', '657'] 612 ['408', '169', '318', '657', '114', '474', '189', '515', '694', '185'] 627 ['603', '169', '408', '483', '488', '515', '480', '168', '127', '641'] 623 ['64', '174', '480', '427', '169', '114', '318', '515', '513', '172'] 628 ['474', '86', '98', '603', '32', '1137', '100', '515', '427', '165'] 625 ['12', '178', '318', '427', '474', '98', '64', '496', '657', '132'] 629 ['318', '313', '169', '408', '8', '114', '520', '496', '483', '97'] 633 ['316', '64', '313', '483', '513', '22', '357', '191', '174', '659'] 632 ['408', '169', '190', '511', '520', '178', '114', '427', '251', '493'] 631 ['408', '480', '513', '178', '169', '12', '64', '603', '709', '647'] 634 ['169', '191', '59', '177', '12', '430', '64', '178', '176', '483'] 639 ['408', '185', '603', '480', '251', '302', '114', '133', '515', '657'] 630 ['313', '189', '114', '318', '923', '408', '603', '178', '169', '611'] 642 ['483', '251', '513', '59', '1143', '1449', '161', '272', '86', '187'] 637 ['87', '318', '483', '114', '511', '133', '657', '480', '427', '89'] 640 ['603', '181', '480', '483', '408', '484', '199', '50', '513', '114'] 626 ['98', '483', '169', '64', '12', '661', '285', '513', '318', '484'] 643 ['134', '318', '302', '251', '654', '285', '657', '64', '513', '199'] 598 ['318', '169', '408', '64', '483', '48', '511', '170', '285', '423'] 638 ['318', '64', '205', '483', '143', '79', '316', '178', '408', '480'] 635 ['408', '132', '483', '114', '50', '272', '478', '1137', '251', '198'] 644 ['302', '483', '408', '169', '114', '313', '515', '963', '488', '64'] 636 ['98', '181', '496', '174', '172', '483', '408', '178', '318', '64'] 645 ['408', '603', '169', '657', '269', '285', '127', '114', '661', '480'] 648 ['313', '300', '258', '8', '378', '12', '129', '611', '51', '515'] 647 ['12', '178', '513', '963', '169', '641', '408', '657', '478', '64'] 650 ['408', '169', '318', '12', '114', '513', '165', '611', '64', '190'] 651 ['408', '169', '318', '483', '513', '657', '603', '134', '251', '64'] 654 ['427', '178', '480', '515', '657', '603', '202', '483', '328', '378'] 653 ['750', '114', '963', '169', '1019', '173', '12', '483', '498', '515'] 655 ['199', '661', '641', '1449', '488', '169', '189', '114', '530', '484'] 649 ['64', '169', '357', '513', '191', '603', '22', '12', '45', '272'] 658 ['357', '64', '187', '191', '272', '427', '180', '480', '285', '483'] 656 ['169', '479', '474', '357', '318', '64', '199', '23', '127', '114'] 660 ['641', '114', '199', '169', '12', '178', '647', '59', '408', '187'] 659 ['169', '12', '251', '114', '923', '318', '100', '408', '302', '515'] 646 ['408', '169', '114', '513', '515', '64', '318', '251', '603', '98'] 663 ['511', '172', '22', '114', '408', '483', '651', '169', '427', '963'] 664 ['511', '357', '474', '199', '178', '493', '114', '488', '316', '512'] 657 ['483', '318', '98', '313', '64', '427', '480', '515', '408', '603'] 665 ['408', '515', '169', '513', '512', '474', '189', '480', '318', '493'] 666 ['285', '408', '589', '60', '641', '475', '652', '512', '178', '14'] 661 ['483', '114', '12', '479', '513', '511', '659', '519', '474', '302'] 662 ['408', '318', '528', '12', '127', '316', '923', '187', '199', '603'] 667 ['169', '408', '511', '170', '474', '89', '180', '302', '165', '663'] 641 ['302', '641', '963', '474', '187', '654', '488', '127', '318', '1449'] 668 ['519', '513', '169', '12', '178', '603', '187', '963', '127', '483'] 673 ['408', '169', '98', '285', '56', '483', '1449', '50', '603', '276'] 671 ['143', '318', '64', '133', '251', '483', '302', '408', '95', '429'] 669 ['478', '199', '285', '513', '251', '165', '178', '318', '923', '606'] 676 ['96', '228', '408', '498', '427', '178', '511', '515', '19', '89'] 674 ['318', '272', '64', '114', '169', '9', '89', '483', '22', '357'] 652 ['408', '483', '318', '114', '694', '498', '169', '197', '484', '603'] 677 ['302', '272', '484', '251', '169', '488', '514', '483', '611', '519'] 682 ['408', '483', '1143', '478', '1142', '198', '169', '480', '116', '496'] 679 ['408', '480', '127', '515', '603', '12', '114', '190', '136', '474'] 684 ['313', '169', '496', '189', '318', '114', '515', '651', '603', '493'] 685 ['513', '483', '199', '178', '603', '169', '114', '923', '192', '12'] 683 ['408', '50', '357', '528', '12', '318', '134', '169', '483', '64'] 691 ['408', '427', '483', '169', '515', '197', '12', '657', '134', '480'] 672 ['427', '318', '64', '178', '603', '408', '12', '496', '59', '89'] 692 ['169', '408', '178', '64', '251', '272', '174', '12', '657', '603'] 690 ['50', '313', '172', '181', '659', '515', '251', '318', '272', '531'] 689 ['174', '302', '318', '513', '64', '313', '191', '192', '79', '272'] 686 ['483', '408', '169', '251', '316', '606', '479', '482', '114', '165'] 693 ['603', '251', '657', '169', '474', '513', '515', '114', '408', '83'] 688 ['242', '51', '346', '474', '265', '86', '222', '144', '1', '95'] 697 ['64', '169', '603', '190', '318', '134', '313', '483', '408', '136'] 698 ['318', '272', '12', '589', '531', '963', '657', '178', '169', '114'] 670 ['302', '100', '169', '50', '318', '172', '127', '408', '173', '64'] 694 ['169', '114', '12', '64', '408', '963', '515', '313', '192', '173'] 680 ['173', '483', '168', '114', '134', '12', '603', '89', '923', '178'] 705 ['513', '316', '320', '251', '735', '57', '1103', '408', '493', '272'] 701 ['98', '515', '483', '408', '318', '357', '169', '603', '427', '172'] 699 ['408', '657', '169', '89', '134', '114', '923', '483', '474', '603'] 704 ['474', '114', '483', '408', '513', '530', '169', '179', '12', '357'] 707 ['56', '320', '408', '114', '169', '223', '513', '357', '646', '189'] 700 ['408', '172', '134', '313', '89', '483', '12', '64', '114', '190'] 687 ['169', '127', '408', '427', '318', '114', '483', '357', '272', '480'] 695 ['127', '603', '479', '134', '488', '483', '12', '513', '223', '178'] 675 ['483', '474', '479', '178', '64', '480', '513', '603', '657', '641'] 708 ['96', '174', '173', '64', '318', '408', '89', '169', '172', '515'] 709 ['963', '169', '511', '408', '513', '488', '114', '100', '479', '12'] 711 ['14', '357', '285', '603', '9', '657', '179', '1194', '124', '589'] 710 ['474', '511', '114', '515', '480', '513', '498', '427', '408', '482'] 712 ['64', '318', '408', '357', '657', '496', '118', '274', '22', '169'] 715 ['187', '515', '127', '114', '313', '511', '408', '750', '651', '357'] 713 ['169', '480', '483', '357', '98', '197', '12', '408', '285', '64'] 716 ['316', '285', '272', '15', '315', '169', '408', '71', '1449', '170'] 681 ['483', '318', '169', '515', '498', '603', '657', '96', '427', '513'] 678 ['318', '64', '169', '59', '313', '48', '272', '178', '484', '124'] 719 ['169', '479', '963', '519', '196', '197', '190', '1449', '133', '192'] 702 ['50', '98', '408', '181', '64', '174', '132', '357', '199', '114'] 721 ['114', '96', '408', '169', '313', '659', '186', '480', '316', '210'] 714 ['174', '318', '64', '313', '316', '79', '172', '515', '173', '520'] 717 ['173', '515', '498', '483', '216', '479', '64', '318', '408', '170'] 718 ['318', '64', '603', '190', '313', '12', '169', '357', '272', '22'] 696 ['483', '408', '474', '197', '513', '1137', '170', '269', '528', '511'] 722 ['50', '408', '12', '318', '483', '127', '174', '191', '272', '269'] 724 ['318', '64', '174', '98', '144', '651', '173', '143', '357', '186'] 727 ['272', '313', '966', '484', '190', '223', '661', '134', '963', '237'] 725 ['408', '318', '192', '923', '190', '657', '480', '169', '134', '603'] 706 ['483', '318', '174', '172', '408', '169', '114', '923', '59', '272'] 720 ['114', '474', '12', '408', '514', '513', '98', '134', '357', '516'] 729 ['513', '64', '408', '169', '603', '178', '480', '659', '318', '251'] 726 ['174', '50', '169', '190', '12', '1449', '963', '408', '172', '199'] 728 ['318', '64', '169', '178', '603', '127', '657', '483', '408', '357'] 703 ['313', '64', '174', '318', '169', '408', '357', '79', '480', '96'] 738 ['114', '12', '515', '520', '480', '132', '272', '483', '963', '1194'] 736 ['98', '483', '408', '169', '12', '64', '427', '69', '190', '603'] 734 ['64', '169', '178', '641', '189', '12', '408', '185', '513', '659'] 730 ['64', '483', '12', '134', '174', '178', '59', '318', '1449', '186'] 743 ['127', '64', '251', '12', '178', '474', '483', '169', '357', '56'] 742 ['169', '178', '64', '251', '357', '98', '427', '187', '603', '191'] 737 ['134', '603', '114', '513', '199', '479', '408', '607', '483', '641'] 733 ['408', '60', '483', '134', '514', '528', '169', '114', '168', '902'] 745 ['318', '513', '223', '357', '208', '173', '408', '251', '199', '22'] 740 ['169', '318', '483', '657', '963', '603', '114', '408', '320', '923'] 735 ['408', '318', '169', '64', '98', '357', '114', '12', '56', '205'] 747 ['191', '963', '114', '484', '518', '1449', '165', '656', '641', '613'] 723 ['318', '408', '12', '64', '474', '56', '513', '127', '98', '302'] 739 ['169', '189', '963', '408', '173', '1019', '174', '272', '474', '516'] 749 ['318', '12', '408', '493', '607', '28', '963', '169', '165', '1007'] 748 ['313', '12', '127', '191', '178', '480', '134', '484', '430', '611'] 746 ['313', '169', '408', '173', '114', '318', '480', '98', '483', '272'] 731 ['98', '651', '272', '423', '692', '22', '174', '169', '172', '181'] 750 ['318', '98', '169', '1449', '313', '408', '64', '59', '483', '272'] 741 ['318', '12', '127', '64', '169', '603', '408', '185', '511', '285'] 751 ['318', '408', '114', '657', '169', '511', '64', '357', '513', '132'] 756 ['114', '408', '474', '318', '657', '169', '1194', '483', '515', '513'] 757 ['318', '169', '408', '12', '603', '320', '114', '483', '272', '302'] 752 ['134', '603', '89', '12', '179', '514', '488', '408', '611', '318'] 758 ['178', '169', '251', '519', '429', '136', '194', '516', '86', '606'] 732 ['318', '169', '12', '50', '98', '408', '285', '64', '114', '657'] 762 ['169', '408', '134', '528', '189', '318', '209', '483', '285', '205'] 744 ['100', '12', '64', '408', '169', '183', '251', '134', '318', '357'] 754 ['169', '480', '318', '100', '513', '114', '483', '484', '603', '357'] 753 ['318', '169', '603', '496', '216', '408', '498', '1007', '478', '12'] 763 ['169', '134', '114', '603', '285', '513', '963', '178', '488', '408'] 764 ['963', '265', '169', '515', '427', '498', '193', '114', '513', '480'] 767 ['169', '474', '178', '50', '408', '603', '191', '173', '134', '114'] 769 ['318', '169', '483', '1449', '64', '22', '174', '657', '408', '189'] 755 ['169', '408', '114', '313', '64', '89', '318', '190', '427', '272'] 771 ['178', '272', '483', '513', '57', '430', '657', '479', '963', '923'] 768 ['64', '318', '169', '22', '79', '651', '178', '251', '174', '191'] 773 ['199', '528', '276', '223', '177', '285', '134', '137', '478', '511'] 765 ['357', '408', '318', '134', '12', '168', '64', '89', '483', '603'] 772 ['408', '318', '64', '357', '169', '114', '173', '178', '427', '663'] 766 ['408', '251', '169', '56', '114', '1103', '515', '513', '320', '12'] 774 ['60', '483', '474', '134', '478', '285', '1194', '603', '32', '315'] 760 ['318', '408', '1449', '23', '192', '272', '513', '189', '923', '663'] 761 ['168', '114', '318', '190', '173', '89', '169', '79', '408', '902'] 777 ['408', '1449', '169', '483', '318', '480', '487', '12', '529', '189'] 759 ['318', '169', '313', '272', '210', '515', '189', '657', '64', '114'] 776 ['484', '172', '313', '513', '100', '408', '12', '64', '480', '1143'] 780 ['181', '114', '483', '173', '169', '408', '480', '205', '191', '178'] 779 ['172', '169', '408', '187', '192', '272', '318', '114', '12', '173'] 778 ['408', '134', '474', '513', '169', '185', '182', '641', '114', '302'] 782 ['318', '15', '408', '513', '87', '114', '169', '199', '613', '657'] 786 ['64', '408', '483', '963', '12', '657', '603', '515', '114', '313'] 784 ['318', '169', '114', '408', '178', '191', '64', '174', '134', '205'] 770 ['98', '603', '196', '515', '498', '496', '479', '216', '483', '408'] 788 ['408', '169', '242', '272', '313', '208', '515', '173', '513', '923'] 789 ['603', '134', '169', '483', '187', '513', '191', '174', '318', '223'] 790 ['64', '318', '272', '12', '114', '513', '178', '169', '185', '316'] 787 ['318', '513', '408', '50', '168', '483', '178', '59', '174', '64'] 783 ['408', '318', '169', '515', '603', '64', '272', '313', '251', '98'] 785 ['513', '483', '408', '603', '178', '114', '963', '169', '98', '493'] 794 ['603', '318', '169', '316', '64', '12', '408', '657', '199', '114'] 781 ['480', '479', '408', '132', '173', '513', '963', '603', '427', '511'] 796 ['1016', '310', '169', '837', '647', '73', '392', '1448', '114', '601'] 795 ['199', '178', '318', '483', '114', '408', '134', '285', '124', '603'] 793 ['408', '474', '178', '483', '12', '302', '515', '251', '134', '603'] 798 ['513', '316', '169', '22', '192', '302', '300', '320', '199', '187'] 791 ['114', '313', '408', '169', '513', '64', '272', '318', '316', '483'] 802 ['64', '313', '187', '174', '178', '12', '169', '22', '114', '651'] 800 ['169', '408', '528', '318', '302', '483', '178', '12', '513', '251'] 804 ['313', '169', '484', '430', '114', '1194', '165', '189', '300', '166'] 803 ['56', '483', '528', '98', '408', '923', '191', '187', '285', '357'] 775 ['169', '408', '318', '963', '174', '64', '12', '483', '520', '178'] 792 ['169', '513', '603', '178', '408', '483', '190', '480', '709', '302'] 799 ['114', '169', '408', '12', '318', '64', '515', '657', '89', '357'] 805 ['474', '408', '134', '114', '251', '921', '285', '57', '513', '923'] 806 ['64', '134', '178', '478', '657', '963', '603', '1039', '194', '132'] 807 ['1449', '189', '513', '963', '316', '272', '114', '302', '178', '192'] 797 ['172', '174', '408', '318', '98', '8', '223', '251', '173', '187'] 801 ['174', '258', '172', '408', '318', '64', '169', '12', '50', '22'] 809 ['408', '114', '89', '318', '64', '963', '190', '169', '127', '515'] 815 ['64', '22', '169', '12', '520', '126', '493', '408', '286', '648'] 817 ['169', '12', '657', '483', '114', '408', '496', '64', '83', '427'] 821 ['515', '479', '169', '603', '316', '474', '408', '187', '178', '480'] 818 ['56', '98', '479', '318', '64', '12', '603', '483', '134', '114'] 814 ['318', '169', '408', '50', '12', '172', '479', '313', '603', '357'] 812 ['169', '64', '98', '176', '114', '313', '318', '272', '96', '144'] 823 ['169', '114', '313', '192', '276', '511', '185', '483', '285', '357'] 825 ['22', '59', '96', '265', '430', '313', '611', '483', '513', '69'] 827 ['169', '474', '603', '100', '515', '479', '483', '134', '357', '513'] 829 ['483', '169', '173', '199', '64', '191', '50', '285', '479', '603'] 811 ['603', '480', '275', '174', '483', '408', '318', '89', '133', '520'] 830 ['178', '318', '64', '483', '143', '493', '515', '513', '302', '659'] 826 ['64', '169', '408', '12', '318', '114', '178', '603', '483', '474'] 831 ['408', '172', '483', '511', '513', '474', '169', '318', '114', '357'] 819 ['12', '357', '178', '127', '603', '9', '22', '64', '313', '483'] 828 ['483', '408', '98', '474', '134', '530', '56', '48', '657', '603'] 808 ['174', '172', '483', '408', '318', '89', '169', '50', '511', '513'] 835 ['64', '178', '657', '603', '483', '114', '493', '498', '478', '963'] 833 ['276', '582', '114', '513', '60', '408', '709', '856', '86', '607'] 836 ['427', '64', '408', '480', '169', '114', '483', '515', '197', '651'] 816 ['474', '98', '603', '408', '178', '318', '56', '357', '169', '12'] 838 ['483', '178', '64', '493', '285', '515', '427', '192', '479', '651'] 839 ['480', '408', '483', '963', '114', '923', '513', '603', '169', '488'] 840 ['318', '114', '408', '136', '488', '633', '427', '694', '178', '1449'] 832 ['169', '98', '114', '408', '318', '515', '480', '603', '189', '89'] 810 ['474', '144', '1', '98', '193', '194', '603', '100', '181', '515'] 844 ['408', '483', '169', '603', '657', '98', '272', '64', '285', '923'] 843 ['408', '169', '189', '134', '156', '12', '114', '513', '272', '318'] 834 ['135', '169', '12', '174', '603', '318', '98', '408', '64', '511'] 846 ['114', '313', '156', '169', '1449', '166', '923', '1143', '408', '1073'] 837 ['483', '515', '923', '50', '114', '408', '315', '64', '318', '178'] 813 ['169', '318', '192', '408', '179', '483', '98', '512', '187', '603'] 842 ['192', '172', '187', '12', '318', '513', '357', '603', '134', '483'] 847 ['64', '87', '478', '268', '12', '315', '493', '22', '169', '497'] 848 ['114', '313', '64', '408', '659', '251', '178', '611', '169', '87'] 822 ['64', '318', '313', '178', '302', '479', '98', '59', '657', '611'] 852 ['318', '483', '187', '173', '178', '64', '313', '429', '169', '174'] 851 ['181', '169', '187', '190', '528', '114', '177', '189', '520', '513'] 849 ['302', '265', '86', '257', '486', '98', '194', '603', '1137', '100'] 854 ['169', '408', '192', '474', '480', '657', '178', '201', '209', '114'] 850 ['64', '313', '1', '143', '483', '603', '657', '169', '489', '191'] 858 ['64', '318', '483', '12', '480', '408', '50', '178', '657', '357'] 853 ['408', '513', '114', '134', '313', '169', '50', '199', '98', '64'] 855 ['318', '316', '64', '187', '114', '302', '197', '963', '408', '651'] 824 ['511', '134', '513', '98', '127', '114', '199', '483', '641', '169'] 845 ['318', '408', '12', '169', '657', '114', '64', '174', '483', '427'] 841 ['169', '12', '357', '657', '89', '64', '603', '114', '408', '100'] 859 ['169', '64', '98', '603', '479', '173', '515', '408', '12', '480'] 862 ['272', '408', '178', '318', '189', '169', '513', '963', '30', '156'] 856 ['169', '64', '318', '12', '603', '114', '657', '513', '515', '178'] 820 ['64', '408', '272', '12', '318', '654', '185', '187', '169', '483'] 863 ['318', '89', '64', '197', '208', '963', '59', '173', '169', '190'] 860 ['318', '483', '190', '64', '484', '178', '169', '963', '603', '1194'] 857 ['318', '496', '408', '515', '272', '64', '178', '169', '479', '59'] 864 ['515', '187', '963', '272', '313', '528', '657', '479', '251', '923'] 865 ['12', '89', '173', '357', '603', '657', '199', '318', '183', '64'] 868 ['175', '357', '657', '165', '654', '302', '137', '134', '483', '251'] 867 ['408', '178', '169', '165', '187', '127', '357', '923', '513', '285'] 861 ['12', '64', '169', '251', '603', '641', '515', '657', '98', '408'] 870 ['408', '114', '963', '285', '14', '529', '515', '187', '525', '482'] 871 ['318', '12', '114', '64', '483', '169', '408', '178', '430', '657'] 875 ['199', '178', '483', '182', '127', '408', '493', '482', '475', '272'] 876 ['408', '169', '603', '64', '357', '114', '427', '657', '480', '515'] 872 ['1449', '302', '318', '169', '189', '603', '79', '513', '427', '484'] 866 ['483', '479', '318', '1449', '408', '22', '98', '923', '64', '515'] 877 ['315', '496', '64', '127', '272', '318', '357', '408', '603', '320'] 873 ['64', '408', '272', '169', '178', '318', '515', '498', '480', '15'] 880 ['427', '251', '198', '408', '528', '813', '178', '480', '134', '496'] 878 ['408', '313', '192', '657', '603', '114', '483', '493', '169', '480'] 869 ['169', '135', '483', '89', '474', '12', '185', '480', '408', '190'] 881 ['169', '496', '272', '607', '963', '966', '210', '657', '313', '316'] 879 ['357', '64', '427', '272', '56', '408', '515', '513', '483', '169'] 883 ['641', '923', '187', '169', '114', '963', '478', '474', '654', '507'] 882 ['480', '318', '64', '313', '169', '408', '114', '651', '12', '520'] 884 ['169', '483', '302', '134', '603', '114', '654', '170', '272', '427'] 886 ['114', '169', '513', '408', '511', '313', '480', '302', '661', '199'] 885 ['313', '64', '114', '496', '480', '657', '515', '408', '165', '385'] 889 ['114', '478', '272', '1019', '963', '429', '143', '251', '316', '136'] 874 ['178', '657', '187', '64', '408', '513', '318', '251', '169', '12'] 892 ['169', '272', '474', '427', '963', '302', '408', '603', '513', '114'] 890 ['408', '178', '318', '12', '14', '242', '60', '320', '1142', '64'] 893 ['169', '98', '357', '483', '408', '313', '480', '189', '427', '64'] 887 ['15', '750', '79', '408', '705', '59', '516', '316', '313', '205'] 891 ['242', '98', '603', '515', '272', '483', '408', '178', '318', '64'] 894 ['169', '357', '657', '408', '1449', '963', '603', '89', '64', '480'] 896 ['357', '169', '114', '520', '170', '194', '750', '408', '197', '193'] 897 ['302', '114', '169', '191', '513', '64', '357', '511', '178', '272'] 901 ['143', '313', '79', '302', '480', '603', '483', '515', '173', '114'] 899 ['489', '169', '87', '480', '178', '313', '136', '482', '656', '657'] 903 ['169', '408', '190', '511', '474', '134', '513', '200', '285', '654'] 904 ['87', '408', '318', '8', '528', '515', '272', '64', '923', '251'] 907 ['242', '302', '474', '265', '193', '603', '1137', '196', '515', '526'] 905 ['357', '408', '187', '64', '318', '127', '483', '480', '1137', '606'] 902 ['169', '408', '178', '427', '98', '22', '603', '511', '705', '190'] 898 ['169', '408', '12', '483', '50', '187', '528', '603', '923', '285'] 895 ['357', '483', '127', '318', '64', '98', '114', '963', '515', '408'] 906 ['483', '169', '56', '64', '114', '357', '98', '480', '318', '12'] 900 ['408', '191', '170', '174', '134', '427', '114', '357', '135', '169'] 908 ['178', '169', '408', '64', '285', '272', '513', '189', '474', '59'] 916 ['127', '169', '285', '408', '191', '515', '357', '661', '603', '1142'] 911 ['169', '64', '963', '408', '657', '178', '318', '513', '12', '170'] 912 ['480', '178', '169', '603', '657', '285', '923', '202', '493', '189'] 914 ['408', '169', '657', '483', '515', '275', '114', '480', '178', '923'] 918 ['285', '272', '528', '813', '61', '408', '513', '30', '479', '511'] 919 ['169', '485', '603', '178', '963', '251', '45', '213', '479', '408'] 921 ['186', '318', '114', '22', '64', '302', '12', '520', '204', '963'] 910 ['318', '515', '483', '603', '474', '64', '408', '427', '479', '136'] 913 ['511', '134', '190', '275', '272', '515', '170', '114', '521', '1039'] 915 ['169', '483', '64', '12', '603', '50', '318', '484', '98', '408'] 922 ['483', '169', '408', '493', '199', '178', '709', '750', '515', '64'] 923 ['98', '603', '515', '427', '480', '272', '79', '483', '408', '178'] 928 ['242', '302', '474', '603', '32', '432', '515', '480', '498', '272'] 927 ['313', '114', '318', '272', '302', '483', '603', '657', '169', '408'] 924 ['603', '483', '190', '169', '657', '1449', '474', '22', '479', '520'] 929 ['302', '64', '114', '513', '357', '511', '427', '169', '251', '408'] 931 ['318', '64', '98', '496', '963', '114', '169', '174', '603', '56'] 917 ['318', '169', '357', '408', '603', '316', '1449', '313', '189', '59'] 932 ['318', '408', '963', '694', '59', '1073', '923', '512', '1194', '12'] 909 ['302', '474', '603', '427', '480', '272', '483', '408', '127', '89'] 934 ['98', '169', '272', '511', '114', '408', '187', '318', '165', '189'] 933 ['114', '963', '185', '479', '513', '856', '521', '656', '134', '427'] 935 ['169', '408', '318', '114', '923', '12', '611', '189', '493', '427'] 938 ['64', '174', '520', '265', '302', '199', '79', '483', '496', '659'] 940 ['169', '483', '408', '178', '603', '320', '423', '318', '1', '59'] 888 ['483', '408', '169', '603', '318', '480', '515', '114', '657', '251'] 925 ['318', '313', '178', '316', '603', '496', '191', '114', '657', '169'] 942 ['98', '64', '178', '169', '963', '408', '483', '132', '22', '603'] 937 ['318', '169', '603', '480', '98', '114', '64', '657', '427', '606'] 926 ['318', '357', '169', '408', '50', '285', '174', '483', '178', '251'] 943 ['223', '198', '45', '269', '357', '272', '531', '483', '963', '134'] 939 ['302', '474', '144', '95', '98', '603', '100', '181', '515', '427'] 936 ['483', '408', '134', '474', '89', '480', '169', '12', '114', '1449'] 930 ['98', '357', '603', '318', '511', '197', '480', '285', '408', '606'] 920 ['318', '483', '79', '174', '408', '178', '169', '657', '603', '114'] 941 ['50', '64', '127', '169', '483', '114', '603', '12', '318', '98']
In [25]:
#알고리즘 평가
import surprise
from surprise import Dataset
data = Dataset.load_builtin('ml-100k')
data.split(n_folds=3)
In [ ]:
# RMSE(평균 제곱근 편차) # 추정 값 또는 모델이 예측한 값과 실제 환경에서 관찰되는 값의 차이를 다룰 때 사용하는 측도 # 정밀도를 표현하는데 적합하다. # 값이 커질수록 예측에 어느정도 오류가 있는지 알 수 있다. # MAE(평균 절대 오차) # 두 개의 연속 변수 간의 차리를 측정한 것 # 예측 결과와 최종결과를 비교하는 방법 중 하나. #RMSE vs MAE # RMSE 가 MAE보다 이상치에 민감 # 하지만 이상치가 매우 드물 경우 RMSE가 잘 맞아 일반적으로 널리 사용됨
In [26]:
sim_options = {'name': 'msd'}
algo = surprise.KNNBasic(sim_options = sim_options)
surprise.evaluate(algo, data)
/Users/apple/anaconda3/lib/python3.6/site-packages/surprise/evaluate.py:66: UserWarning: The evaluate() method is deprecated. Please use model_selection.cross_validate() instead. 'model_selection.cross_validate() instead.', UserWarning) /Users/apple/anaconda3/lib/python3.6/site-packages/surprise/dataset.py:193: UserWarning: Using data.split() or using load_from_folds() without using a CV iterator is now deprecated. UserWarning)
Evaluating RMSE, MAE of algorithm KNNBasic. ------------ Fold 1 Computing the msd similarity matrix... Done computing similarity matrix. RMSE: 0.9886 MAE: 0.7802 ------------ Fold 2 Computing the msd similarity matrix... Done computing similarity matrix. RMSE: 0.9915 MAE: 0.7840 ------------ Fold 3 Computing the msd similarity matrix... Done computing similarity matrix. RMSE: 0.9866 MAE: 0.7817 ------------ ------------ Mean RMSE: 0.9889 Mean MAE : 0.7820 ------------ ------------
Out[26]:
CaseInsensitiveDefaultDict(list,
{'mae': [0.7802189976083773,
0.783998616236742,
0.7816694414058579],
'rmse': [0.988581860543401,
0.9915342547625395,
0.9866376586861513]})
In [27]:
sim_options = {'name' : 'cosine'}
algo = surprise.KNNBasic(sim_options=sim_options)
surprise.evaluate(algo,data)
/Users/apple/anaconda3/lib/python3.6/site-packages/surprise/evaluate.py:66: UserWarning: The evaluate() method is deprecated. Please use model_selection.cross_validate() instead. 'model_selection.cross_validate() instead.', UserWarning) /Users/apple/anaconda3/lib/python3.6/site-packages/surprise/dataset.py:193: UserWarning: Using data.split() or using load_from_folds() without using a CV iterator is now deprecated. UserWarning)
Evaluating RMSE, MAE of algorithm KNNBasic. ------------ Fold 1 Computing the cosine similarity matrix... Done computing similarity matrix. RMSE: 1.0205 MAE: 0.8060 ------------ Fold 2 Computing the cosine similarity matrix... Done computing similarity matrix. RMSE: 1.0246 MAE: 0.8120 ------------ Fold 3 Computing the cosine similarity matrix... Done computing similarity matrix. RMSE: 1.0195 MAE: 0.8091 ------------ ------------ Mean RMSE: 1.0215 Mean MAE : 0.8090 ------------ ------------
Out[27]:
CaseInsensitiveDefaultDict(list,
{'mae': [0.806030797261509,
0.8120439158026675,
0.8090582538793538],
'rmse': [1.020463034428536,
1.0246269391772709,
1.0194786302184546]})
In [28]:
sim_options = {'name' : 'pearson_baseline'}
algo = surprise.KNNBasic(sim_options=sim_options)
surprise.evaluate(algo,data)
/Users/apple/anaconda3/lib/python3.6/site-packages/surprise/evaluate.py:66: UserWarning: The evaluate() method is deprecated. Please use model_selection.cross_validate() instead. 'model_selection.cross_validate() instead.', UserWarning) /Users/apple/anaconda3/lib/python3.6/site-packages/surprise/dataset.py:193: UserWarning: Using data.split() or using load_from_folds() without using a CV iterator is now deprecated. UserWarning)
Evaluating RMSE, MAE of algorithm KNNBasic. ------------ Fold 1 Estimating biases using als... Computing the pearson_baseline similarity matrix... Done computing similarity matrix. RMSE: 1.0099 MAE: 0.7982 ------------ Fold 2 Estimating biases using als... Computing the pearson_baseline similarity matrix... Done computing similarity matrix. RMSE: 1.0140 MAE: 0.8030 ------------ Fold 3 Estimating biases using als... Computing the pearson_baseline similarity matrix... Done computing similarity matrix. RMSE: 1.0070 MAE: 0.7987 ------------ ------------ Mean RMSE: 1.0103 Mean MAE : 0.8000 ------------ ------------
Out[28]:
CaseInsensitiveDefaultDict(list,
{'mae': [0.7982412446965477,
0.802987846170291,
0.798665233328668],
'rmse': [1.009867180638949,
1.013983308014259,
1.0070221348855233]})
In [29]:
algo = surprise.SVD() surprise.evaluate(algo,data)
/Users/apple/anaconda3/lib/python3.6/site-packages/surprise/evaluate.py:66: UserWarning: The evaluate() method is deprecated. Please use model_selection.cross_validate() instead. 'model_selection.cross_validate() instead.', UserWarning) /Users/apple/anaconda3/lib/python3.6/site-packages/surprise/dataset.py:193: UserWarning: Using data.split() or using load_from_folds() without using a CV iterator is now deprecated. UserWarning)
Evaluating RMSE, MAE of algorithm SVD. ------------ Fold 1 RMSE: 0.9467 MAE: 0.7461 ------------ Fold 2 RMSE: 0.9476 MAE: 0.7470 ------------ Fold 3 RMSE: 0.9430 MAE: 0.7454 ------------ ------------ Mean RMSE: 0.9458 Mean MAE : 0.7461 ------------ ------------
Out[29]:
CaseInsensitiveDefaultDict(list,
{'mae': [0.7461152162961537,
0.7469725668755399,
0.7453529954592129],
'rmse': [0.9466899057105319,
0.9475860673879669,
0.9430403257419846]})