Обучение модели нейронной сети

2020-11-02T20:06:14Z

Категория статьи: Нейронные сети

Процесс обучения модели. Параметры обучения. Сохранение модели

Всем привет!

В этой статье речь пойдет об обучении модели.

В процессе обучения нам понадобится уменьшать скорость обучения (learning rate) и на каждой эпохе будем сохранять модель, если ее метрика стала лучше по отношению к предыдущим эпохам. Это можно сделать используя callback функции.

from tensorflow.keras.callbacks import ReduceLROnPlateau, ModelCheckpoint

# Для уменьшения скорости обучения
callback_RLR = ReduceLROnPlateau(monitor='val_accuracy',
                        factor=0.5,
                        patience=5,
                        verbose=1,
                        mode='auto',
                        min_delta=0.0001,
                        cooldown=0,
                        min_lr=0)

# Имя файла для сохранения модели
model_filename = str(cl_count).zfill(max_len_dirname) + ".hdf5"

# Сохранение модели с лучшей метрикой
callback_MC = ModelCheckpoint(os.path.join(model_dir, model_filename),
                              monitor='val_accuracy',
                              verbose=1,
                              save_best_only=True,
                              save_weights_only=False,
                              mode='auto')

Определим размер партии (batch_size) и создадим два генератора: для обучения (train_gen) и для проверки (val_gen).

batch_size = 400

train_gen = gen_batch(train_dir, batch_size)
val_gen = gen_batch(val_dir, batch_size)

Определим количество эпох, шагов за эпоху для обучения и для проверки. Создадим таймер для определения времени обучение. Запускаем!!!

timer1 = MeasuringRunTime.Timer()
history = model.fit(
    train_gen,
    steps_per_epoch=8,
    epochs=60,
    validation_data=(val_gen),
    validation_steps=8,
    callbacks=[callback_RLR, callback_MC],
    verbose=1
    )
print(f"Общее время обучения {timer1.Stop()} сек. ( ~ {int(timer1.Stop() // 60)} мин. )")

Можно отключить ModelCheckpoint - сохранение лучшей модели на каждой эпохе это ускорит процесс обучения, но если произойдет какой-либо сбой, то придется обучать модель заново. Для этого просто удалите ModelCheckpoint из списка callbacks метода модели fit.

Epoch 1/60
8/8 [==============================] - 17s 2s/step - loss: 3.8608 - accuracy: 0.1078 - val_loss: 545.5818 - val_accuracy: 0.0338
Epoch 2/60
8/8 [==============================] - 21s 3s/step - loss: 3.1756 - accuracy: 0.1806 - val_loss: 154.8965 - val_accuracy: 0.0622
Epoch 3/60
8/8 [==============================] - 24s 3s/step - loss: 2.7437 - accuracy: 0.2784 - val_loss: 66.4634 - val_accuracy: 0.0691
Epoch 4/60
8/8 [==============================] - 28s 4s/step - loss: 2.3348 - accuracy: 0.3625 - val_loss: 47.2417 - val_accuracy: 0.0466
Epoch 5/60
8/8 [==============================] - 32s 4s/step - loss: 1.9039 - accuracy: 0.4525 - val_loss: 32.7056 - val_accuracy: 0.0366
Epoch 6/60
8/8 [==============================] - 32s 4s/step - loss: 1.5286 - accuracy: 0.5572 - val_loss: 25.3254 - val_accuracy: 0.0534
Epoch 7/60
8/8 [==============================] - 28s 4s/step - loss: 1.3385 - accuracy: 0.5987 - val_loss: 23.8004 - val_accuracy: 0.0413
Epoch 8/60
8/8 [==============================] - 18s 2s/step - loss: 1.0778 - accuracy: 0.6716 - val_loss: 13.5953 - val_accuracy: 0.1381
Epoch 9/60
8/8 [==============================] - 18s 2s/step - loss: 0.8656 - accuracy: 0.7359 - val_loss: 13.7603 - val_accuracy: 0.1097
Epoch 10/60
8/8 [==============================] - 21s 3s/step - loss: 0.7403 - accuracy: 0.7625 - val_loss: 8.9256 - val_accuracy: 0.2200
Epoch 11/60
8/8 [==============================] - 24s 3s/step - loss: 0.6791 - accuracy: 0.7869 - val_loss: 7.9027 - val_accuracy: 0.2375
Epoch 12/60
8/8 [==============================] - 31s 4s/step - loss: 0.5866 - accuracy: 0.8134 - val_loss: 7.4308 - val_accuracy: 0.2431
Epoch 13/60
8/8 [==============================] - 32s 4s/step - loss: 0.6051 - accuracy: 0.8109 - val_loss: 4.6100 - val_accuracy: 0.3625
Epoch 14/60
8/8 [==============================] - 24s 3s/step - loss: 0.5324 - accuracy: 0.8413 - val_loss: 5.2812 - val_accuracy: 0.3303
Epoch 15/60
8/8 [==============================] - 21s 3s/step - loss: 0.5200 - accuracy: 0.8419 - val_loss: 3.3817 - val_accuracy: 0.4363
Epoch 16/60
8/8 [==============================] - 14s 2s/step - loss: 0.3360 - accuracy: 0.8978 - val_loss: 3.2014 - val_accuracy: 0.4531
Epoch 17/60
8/8 [==============================] - 17s 2s/step - loss: 0.3040 - accuracy: 0.9116 - val_loss: 2.7252 - val_accuracy: 0.5131
Epoch 18/60
8/8 [==============================] - 23s 3s/step - loss: 0.2650 - accuracy: 0.9228 - val_loss: 2.8983 - val_accuracy: 0.5019
Epoch 19/60
8/8 [==============================] - 27s 3s/step - loss: 0.2468 - accuracy: 0.9306 - val_loss: 2.3681 - val_accuracy: 0.5516
Epoch 20/60
8/8 [==============================] - 31s 4s/step - loss: 0.2285 - accuracy: 0.9312 - val_loss: 1.8135 - val_accuracy: 0.6272
Epoch 21/60
8/8 [==============================] - 21s 3s/step - loss: 0.2125 - accuracy: 0.9328 - val_loss: 1.8626 - val_accuracy: 0.5991
Epoch 22/60
8/8 [==============================] - 23s 3s/step - loss: 0.2230 - accuracy: 0.9312 - val_loss: 1.2885 - val_accuracy: 0.6934
Epoch 23/60
8/8 [==============================] - 20s 2s/step - loss: 0.1755 - accuracy: 0.9466 - val_loss: 1.2572 - val_accuracy: 0.6975
Epoch 24/60
8/8 [==============================] - 18s 2s/step - loss: 0.1221 - accuracy: 0.9641 - val_loss: 0.8783 - val_accuracy: 0.7588
Epoch 25/60
8/8 [==============================] - 23s 3s/step - loss: 0.1230 - accuracy: 0.9634 - val_loss: 0.9080 - val_accuracy: 0.7531
Epoch 26/60
8/8 [==============================] - 25s 3s/step - loss: 0.0969 - accuracy: 0.9734 - val_loss: 0.7368 - val_accuracy: 0.7884
Epoch 27/60
8/8 [==============================] - 26s 3s/step - loss: 0.0855 - accuracy: 0.9769 - val_loss: 0.6356 - val_accuracy: 0.8159
Epoch 28/60
8/8 [==============================] - 20s 2s/step - loss: 0.0871 - accuracy: 0.9762 - val_loss: 0.6328 - val_accuracy: 0.8181
Epoch 29/60
8/8 [==============================] - 24s 3s/step - loss: 0.0872 - accuracy: 0.9741 - val_loss: 0.4666 - val_accuracy: 0.8612
Epoch 30/60
8/8 [==============================] - 26s 3s/step - loss: 0.0846 - accuracy: 0.9737 - val_loss: 0.5280 - val_accuracy: 0.8534
Epoch 31/60
8/8 [==============================] - 20s 2s/step - loss: 0.0498 - accuracy: 0.9859 - val_loss: 0.4329 - val_accuracy: 0.8619
Epoch 32/60
8/8 [==============================] - 22s 3s/step - loss: 0.0404 - accuracy: 0.9894 - val_loss: 0.3951 - val_accuracy: 0.8822
Epoch 33/60
8/8 [==============================] - 26s 3s/step - loss: 0.0528 - accuracy: 0.9862 - val_loss: 0.4171 - val_accuracy: 0.8769
Epoch 34/60
8/8 [==============================] - 22s 3s/step - loss: 0.0537 - accuracy: 0.9866 - val_loss: 0.4298 - val_accuracy: 0.8769
Epoch 35/60
8/8 [==============================] - 20s 3s/step - loss: 0.0566 - accuracy: 0.9834 - val_loss: 0.3815 - val_accuracy: 0.8869
Epoch 36/60
8/8 [==============================] - 24s 3s/step - loss: 0.0752 - accuracy: 0.9819 - val_loss: 0.3856 - val_accuracy: 0.8791
Epoch 37/60
8/8 [==============================] - 28s 4s/step - loss: 0.0748 - accuracy: 0.9812 - val_loss: 0.3436 - val_accuracy: 0.9028
Epoch 38/60
8/8 [==============================] - 25s 3s/step - loss: 0.0699 - accuracy: 0.9806 - val_loss: 0.3897 - val_accuracy: 0.8878
Epoch 39/60
8/8 [==============================] - 23s 3s/step - loss: 0.0380 - accuracy: 0.9906 - val_loss: 0.3333 - val_accuracy: 0.8972
Epoch 40/60
8/8 [==============================] - 26s 3s/step - loss: 0.0343 - accuracy: 0.9919 - val_loss: 0.3133 - val_accuracy: 0.8988
Epoch 41/60
8/8 [==============================] - 17s 2s/step - loss: 0.0420 - accuracy: 0.9866 - val_loss: 0.2928 - val_accuracy: 0.9059
Epoch 42/60
8/8 [==============================] - 21s 3s/step - loss: 0.0320 - accuracy: 0.9928 - val_loss: 0.3107 - val_accuracy: 0.9081
Epoch 43/60
8/8 [==============================] - 25s 3s/step - loss: 0.0300 - accuracy: 0.9922 - val_loss: 0.2738 - val_accuracy: 0.9156
Epoch 44/60
8/8 [==============================] - 30s 4s/step - loss: 0.0468 - accuracy: 0.9897 - val_loss: 0.2625 - val_accuracy: 0.9237
Epoch 45/60
8/8 [==============================] - 30s 4s/step - loss: 0.0357 - accuracy: 0.9887 - val_loss: 0.2542 - val_accuracy: 0.9203
Epoch 46/60
8/8 [==============================] - 22s 3s/step - loss: 0.0243 - accuracy: 0.9944 - val_loss: 0.2753 - val_accuracy: 0.9178
Epoch 47/60
8/8 [==============================] - 21s 3s/step - loss: 0.0332 - accuracy: 0.9909 - val_loss: 0.3277 - val_accuracy: 0.9013
Epoch 48/60
8/8 [==============================] - 17s 2s/step - loss: 0.0345 - accuracy: 0.9900 - val_loss: 0.2464 - val_accuracy: 0.9203
Epoch 49/60
8/8 [==============================] - 20s 3s/step - loss: 0.0286 - accuracy: 0.9919 - val_loss: 0.2898 - val_accuracy: 0.9144
Epoch 50/60
8/8 [==============================] - 26s 3s/step - loss: 0.0290 - accuracy: 0.9931 - val_loss: 0.2363 - val_accuracy: 0.9272
Epoch 51/60
8/8 [==============================] - 30s 4s/step - loss: 0.0288 - accuracy: 0.9925 - val_loss: 0.2360 - val_accuracy: 0.9309
Epoch 52/60
8/8 [==============================] - 31s 4s/step - loss: 0.0261 - accuracy: 0.9941 - val_loss: 0.2430 - val_accuracy: 0.9278
Epoch 53/60
8/8 [==============================] - 27s 3s/step - loss: 0.0178 - accuracy: 0.9947 - val_loss: 0.1963 - val_accuracy: 0.9428
Epoch 54/60
8/8 [==============================] - 17s 2s/step - loss: 0.0149 - accuracy: 0.9966 - val_loss: 0.2368 - val_accuracy: 0.9284
Epoch 55/60
8/8 [==============================] - 16s 2s/step - loss: 0.0240 - accuracy: 0.9934 - val_loss: 0.2196 - val_accuracy: 0.9312
Epoch 56/60
8/8 [==============================] - 18s 2s/step - loss: 0.0264 - accuracy: 0.9925 - val_loss: 0.2661 - val_accuracy: 0.9256
Epoch 57/60
8/8 [==============================] - 24s 3s/step - loss: 0.0389 - accuracy: 0.9897 - val_loss: 0.2156 - val_accuracy: 0.9359
Epoch 58/60
8/8 [==============================] - ETA: 0s - loss: 0.0270 - accuracy: 0.9922
Epoch 00058: ReduceLROnPlateau reducing learning rate to 0.004999999888241291.
8/8 [==============================] - 29s 4s/step - loss: 0.0270 - accuracy: 0.9922 - val_loss: 0.2421 - val_accuracy: 0.9297
Epoch 59/60
8/8 [==============================] - 31s 4s/step - loss: 0.0267 - accuracy: 0.9922 - val_loss: 0.1958 - val_accuracy: 0.9350
Epoch 60/60
8/8 [==============================] - 31s 4s/step - loss: 0.0256 - accuracy: 0.9947 - val_loss: 0.1804 - val_accuracy: 0.9428
Общее время обучения 1434.2035085779994 сек. ( ~ 23 мин. )

Я отключил сохранение модели ModelCheckpoint при этом обучение 60 эпох длилось около 23 минут. Точность на обучающем наборе достигла 99,47%, а на проверочном наборе 94,28%.

Выведем историю обучения модели на график для метрик loss и accuracy.

Из графиков видно, что модель обучается нормально, переобучение отсутствует. Нужно еще запустить на обучение несколько эпох и следить за метриками.

Сохраним модель на Google диск

model.save(os.path.join(model_dir, model_filename))

Я запустил обучение модели еще на 60 эпох. Точность на обучающем наборе достигла 100%, а точность на проверочном наборе 96,88%. Видим, что появляется переобучение!

На этом этапе можно остановить процесс обучения.

Для большей точности можно попробовать подобрать другую архитектуру модели, например использовать свёрточные слои.