[2023-01-05 01:15:41][1] Register OctaneLive
[2023-01-05 01:15:41][0] Loaded Octane DLL:OctaneRender Studio 2022.1 RC 3  version:12000009  SDK:2022.1 RC 3
[2023-01-05 01:15:41][0] OctaneRender for Cinema 4D 2022.1-RC3  built for:260   C4D vers:26107
[2023-01-05 01:15:41][0] C4D:26107  comp:[DESKTOP-MIF8A62]  OS:[WINDOWS 64 BIT] Windows 10, 64 Bit, Professional Edition (build 19041)  versType:10
[2023-01-05 01:15:41][0] CPU:[GenuineIntel, stepping 5, model 5, instruction family 6]  speed:3.8 GHz  numOfCpu:16Ht
[2023-01-05 01:15:41][0] Plugins:[cadexchangeconfig substanceconfig 3dquakersR25 addons advanced_render alembic archigrass assetbrowser bitmapfilter bpexchange browser bugslife_client c4d_emulation c4d_viewport_render c4dOctane-R26 c4dtoa ca ca2 cadexchange clothilde collada14 collada15 colorchoosergui compositing cyctypes deltamush deployment dropzone dwg dwgobjects dynamics exchanges ExpansionPack1R25 ExpansionPack2R25 expressiontag fbx ForesterR25 FumeFX FumeFXArnold gltf gorillacam goz greyscalegorillahub hair hdrilink hot4d i_skp interop interop_forger io_forger io_obj lightkit3 lighttypes metrics mkmodeler mocca model mograph motioncam motiontracker nbo nbp newman nodeeditor2 objects onlineupdater pbd_simulations physx_hybrid pluslibrary projectassetinspector python realflow_25 redshift relaxuv scenemanager2 sculpt sculptbrushes shader signal sketch sky sla spacemouse substance teamrender thinking_particles tool_system_hybrid topcoat tpoperators usd uvviewport volumes VRayForCinema xpressocore xtensions redshift4c4d ]
[2023-01-05 01:15:41][0] NVAPI initializing...
[2023-01-05 01:15:41][0] NVAPI Version: NVidia Complete Version 1.10
[2023-01-05 01:15:41][0] Nvidia driver version:527.56  [r527_32]
[2023-01-05 01:15:41][1] LoadOslScripts()
[2023-01-05 01:15:41][0] Loaded 175 osl scripts from dir:C:\Program Files\Maxon Cinema 4D R26\plugins\hot4d\c4doctane\res\osl_scripts
[2023-01-05 01:15:41][1] LoadCustomLuts()
[2023-01-05 01:16:00][1] <markMaterials>
[2023-01-05 01:16:00][1] <markMaterials>
[2023-01-05 01:16:26][1] initOctaneLive()
[2023-01-05 01:16:26][1] Init OctaneLive...[2023-01-05 01:16:28][1] Plugin initialized.[2023-01-05 01:16:28][1] Plugin activation...[2023-01-05 01:16:32][1] Plugin activation OK.
[2023-01-05 01:16:32][1] Activation time=5.935 sec.
[2023-01-05 01:16:32][0] readDeviceSettings   E:0 P:0 T:0 D:0
[2023-01-05 01:16:32][0] VRAM used/free/max:0Kb/12Gb/12Gb  Out-of-core used:0Kb  RAM used:8.341Gb total:63.858Gb  OpenGL free/total:0/0
[2023-01-05 01:16:32][1] Device:0  [NVIDIA GeForce RTX 3060]<Dis>[][8.6]  Memory used/free/total:0Kb/12Gb/12Gb  outOfCoreMem:0Kb  p2pBytesUsed:0  runtimeDataSize:0Kb temperature:30
[2023-01-05 01:16:32][1] <markMaterials>
[2023-01-05 01:16:33][1] >> ProcessUpdate(flags=G cAsyn=0 dDelay=1
[2023-01-05 01:16:33][1] >> ProcessUpdate(flags=G cAsyn=0 dDelay=1
[2023-01-05 01:16:33][1] >> ProcessUpdate(flags=G cAsyn=0 dDelay=1
[2023-01-05 01:16:33][1] >> ProcessUpdate(flags=G cAsyn=0 dDelay=1
[2023-01-05 01:16:33][1] >> ProcessUpdate(flags=G cAsyn=0 dDelay=1
[2023-01-05 01:16:33][1] >> ProcessUpdate(flags=G cAsyn=0 dDelay=1
[2023-01-05 01:16:33][1] >> ProcessUpdate(flags=G cAsyn=0 dDelay=1
[2023-01-05 01:16:33][1] <markMaterials>
[2023-01-05 01:16:33][1] Generating voxels for object:Volume  cnt=75460[2023-01-05 01:16:33][1] Generating voxels for object:Volume  cnt=75460[2023-01-05 01:16:33][1] Generating voxels for object:Volume  cnt=75460[2023-01-05 01:16:33][1] Generating voxels for object:Volume  cnt=75460[2023-01-05 01:16:33][1] Generating voxels for object:Volume  cnt=75460[2023-01-05 01:16:33][1] Generating voxels for object:Volume  cnt=75460[2023-01-05 01:16:33][1] Generating voxels for object:Volume  cnt=75460[2023-01-05 01:16:47][1] checkNewVersions()
[2023-01-05 01:16:49][0] restoreDeviceStates status:2
[2023-01-05 01:16:49][0] setDevicesActivity  E:0 P:0 T:0 D:0
[2023-01-05 01:16:49][0] save
[2023-01-05 01:16:50][0] AllDevicesOff
[2023-01-05 01:17:23][0] AllDevicesOff
[2023-01-05 01:17:24][0] AllDevicesOff
[2023-01-05 01:17:24][0] AllDevicesOff
[2023-01-05 01:17:24][0] AllDevicesOff
[2023-01-05 01:17:25][0] AllDevicesOff
[2023-01-05 01:17:25][0] AllDevicesOff
[2023-01-05 01:17:25][0] AllDevicesOff
[2023-01-05 01:17:26][0] AllDevicesOff
[2023-01-05 01:17:26][0] AllDevicesOff
[2023-01-05 01:17:27][0] AllDevicesOff
[2023-01-05 01:17:27][0] AllDevicesOff
[2023-01-05 01:17:27][0] AllDevicesOff
[2023-01-05 01:17:28][0] AllDevicesOff
[2023-01-05 01:17:28][0] AllDevicesOff
[2023-01-05 01:17:29][0] AllDevicesOff
[2023-01-05 01:17:29][0] AllDevicesOff
[2023-01-05 01:17:29][0] AllDevicesOff
[2023-01-05 01:17:30][0] AllDevicesOff
[2023-01-05 01:17:30][0] AllDevicesOff
[2023-01-05 01:17:31][0] AllDevicesOff
[2023-01-05 01:17:31][0] AllDevicesOff
[2023-01-05 01:17:31][0] AllDevicesOff
[2023-01-05 01:17:31][0] AllDevicesOff
[2023-01-05 01:17:32][0] AllDevicesOff
[2023-01-05 01:17:32][0] AllDevicesOff
[2023-01-05 01:17:33][0] AllDevicesOff
[2023-01-05 01:17:33][0] AllDevicesOff
[2023-01-05 01:17:33][0] AllDevicesOff
[2023-01-05 01:17:34][0] AllDevicesOff
[2023-01-05 01:17:34][0] AllDevicesOff
[2023-01-05 01:17:35][0] AllDevicesOff
[2023-01-05 01:17:35][0] AllDevicesOff
[2023-01-05 01:17:35][0] AllDevicesOff
[2023-01-05 01:17:36][0] AllDevicesOff
[2023-01-05 01:17:36][0] AllDevicesOff
[2023-01-05 01:17:36][0] AllDevicesOff
[2023-01-05 01:17:37][0] AllDevicesOff
[2023-01-05 01:17:37][0] AllDevicesOff
[2023-01-05 01:17:37][0] AllDevicesOff
[2023-01-05 01:17:37][0] AllDevicesOff
[2023-01-05 01:17:38][0] AllDevicesOff
[2023-01-05 01:17:38][0] AllDevicesOff
[2023-01-05 01:17:38][0] AllDevicesOff
[2023-01-05 01:17:39][0] AllDevicesOff
[2023-01-05 01:17:39][0] AllDevicesOff
[2023-01-05 01:17:39][0] AllDevicesOff
[2023-01-05 01:17:40][0] AllDevicesOff
[2023-01-05 01:17:40][0] AllDevicesOff
[2023-01-05 01:17:40][0] AllDevicesOff
[2023-01-05 01:17:41][0] AllDevicesOff
[2023-01-05 01:17:41][0] AllDevicesOff
[2023-01-05 01:17:42][0] AllDevicesOff
[2023-01-05 01:17:42][0] AllDevicesOff
[2023-01-05 01:17:58][0] AllDevicesOff
[2023-01-05 01:18:02][0] AllDevicesOff
[2023-01-05 01:18:12][0] restoreDeviceStates status:2
[2023-01-05 01:18:12][0] setDevicesActivity  E:0 P:0 T:0 D:0
[2023-01-05 01:18:12][0] save
[2023-01-05 01:19:20][0] restoreDeviceStates status:4
[2023-01-05 01:19:20][0] setDevicesActivity  E:0 P:0 T:0 D:0
[2023-01-05 01:19:20][0] save
[2023-01-05 01:19:20][0] 111[2023-01-05 01:20:15][0] restoreDeviceStates status:4
[2023-01-05 01:20:15][0] setDevicesActivity  E:0 P:0 T:0 D:0
[2023-01-05 01:20:15][0] save
[2023-01-05 01:20:15][0] 111[2023-01-05 01:23:12][0] AllDevicesOff
[2023-01-05 01:23:18][0] AllDevicesOff
[2023-01-05 01:23:37][0] restoreDeviceStates status:4
[2023-01-05 01:23:37][0] setDevicesActivity  E:0 P:0 T:0 D:0
[2023-01-05 01:23:37][0] save
[2023-01-05 01:23:37][0] 111[2023-01-05 09:57:45][0] AllDevicesOff
[2023-01-05 09:58:42][0] AllDevicesOff
[2023-01-05 09:58:43][0] AllDevicesOff
[2023-01-05 09:58:57][0] AllDevicesOff
[2023-01-05 09:58:58][0] AllDevicesOff
[2023-01-05 09:59:29][0] AllDevicesOff
[2023-01-05 09:59:30][0] AllDevicesOff
[2023-01-05 10:00:17][0] AllDevicesOff
[2023-01-05 10:00:17][0] AllDevicesOff
[2023-01-05 10:00:46][0] AllDevicesOff
[2023-01-05 10:00:49][0] AllDevicesOff
[2023-01-05 10:01:04][0] restoreDeviceStates status:4
[2023-01-05 10:01:04][0] setDevicesActivity  E:0 P:0 T:0 D:0
[2023-01-05 10:01:04][0] save
[2023-01-05 10:01:04][0] 111[2023-01-05 10:02:33][0] AllDevicesOff
[2023-01-05 10:02:35][0] AllDevicesOff
[2023-01-05 10:02:35][0] AllDevicesOff
[2023-01-05 10:02:36][0] AllDevicesOff
[2023-01-05 10:02:37][0] AllDevicesOff
[2023-01-05 10:02:38][0] AllDevicesOff
[2023-01-05 10:02:39][0] AllDevicesOff
[2023-01-05 10:02:40][0] AllDevicesOff
[2023-01-05 10:02:43][0] AllDevicesOff
[2023-01-05 10:02:47][0] AllDevicesOff
[2023-01-05 10:02:48][0] AllDevicesOff
[2023-01-05 10:02:48][0] AllDevicesOff
[2023-01-05 10:02:49][0] AllDevicesOff
[2023-01-05 10:02:50][0] AllDevicesOff
[2023-01-05 10:02:51][0] AllDevicesOff
[2023-01-05 10:03:08][0] AllDevicesOff
[2023-01-05 10:03:09][0] AllDevicesOff
[2023-01-05 10:03:12][0] AllDevicesOff
[2023-01-05 10:03:12][0] AllDevicesOff
[2023-01-05 10:03:13][0] AllDevicesOff
[2023-01-05 10:03:14][0] AllDevicesOff
[2023-01-05 10:03:15][0] AllDevicesOff
[2023-01-05 10:05:49][0] AllDevicesOff
[2023-01-05 10:05:53][0] AllDevicesOff
[2023-01-05 10:05:54][0] AllDevicesOff
[2023-01-05 10:36:45][0] AllDevicesOff
[2023-01-05 10:36:47][0] AllDevicesOff
[2023-01-05 10:36:47][0] AllDevicesOff
[2023-01-05 10:36:48][0] AllDevicesOff
[2023-01-05 10:36:48][0] AllDevicesOff
[2023-01-05 10:36:49][0] AllDevicesOff
[2023-01-05 10:36:49][0] AllDevicesOff
[2023-01-05 10:36:49][0] AllDevicesOff
[2023-01-05 10:36:52][0] AllDevicesOff
[2023-01-05 10:36:53][0] AllDevicesOff
[2023-01-05 10:36:53][0] AllDevicesOff
[2023-01-05 10:36:53][0] AllDevicesOff
[2023-01-05 10:36:54][0] AllDevicesOff
[2023-01-05 10:37:03][0] AllDevicesOff
[2023-01-05 10:37:04][0] AllDevicesOff
[2023-01-05 10:37:04][0] AllDevicesOff
[2023-01-05 10:37:09][0] AllDevicesOff
[2023-01-05 10:37:09][0] AllDevicesOff
[2023-01-05 10:37:09][0] AllDevicesOff
[2023-01-05 10:37:14][0] AllDevicesOff
[2023-01-05 10:37:15][0] AllDevicesOff
[2023-01-05 10:37:15][0] AllDevicesOff
[2023-01-05 10:37:18][0] AllDevicesOff
[2023-01-05 10:37:19][0] AllDevicesOff
[2023-01-05 10:37:19][0] AllDevicesOff
[2023-01-05 10:37:23][0] AllDevicesOff
[2023-01-05 10:37:24][0] AllDevicesOff
[2023-01-05 10:37:26][0] AllDevicesOff
[2023-01-05 10:37:27][0] AllDevicesOff
[2023-01-05 10:37:40][0] AllDevicesOff
[2023-01-05 10:38:09][0] restoreDeviceStates status:4
[2023-01-05 10:38:09][0] setDevicesActivity  E:0 P:0 T:0 D:0
[2023-01-05 10:38:09][0] save
[2023-01-05 10:38:09][0] 111[2023-01-05 10:39:00][0] AllDevicesOff
[2023-01-05 10:39:01][0] AllDevicesOff
[2023-01-05 10:39:01][0] AllDevicesOff
[2023-01-05 10:39:01][0] AllDevicesOff
[2023-01-05 10:39:02][0] AllDevicesOff
[2023-01-05 10:39:02][0] AllDevicesOff
[2023-01-05 10:39:02][0] AllDevicesOff
[2023-01-05 10:39:04][0] AllDevicesOff
[2023-01-05 10:39:05][0] AllDevicesOff
[2023-01-05 10:39:06][0] AllDevicesOff
[2023-01-05 10:39:08][0] AllDevicesOff
[2023-01-05 10:39:09][0] AllDevicesOff
[2023-01-05 10:39:09][0] AllDevicesOff
[2023-01-05 10:39:18][0] restoreDeviceStates status:4
[2023-01-05 10:39:18][0] setDevicesActivity  E:0 P:0 T:0 D:0
[2023-01-05 10:39:18][0] save
[2023-01-05 10:39:18][0] 111[2023-01-05 10:51:01][0] AllDevicesOff
[2023-01-05 10:51:27][0] AllDevicesOff
[2023-01-05 10:54:25][0] AllDevicesOff
[2023-01-05 10:54:28][0] AllDevicesOff
[2023-01-05 10:55:05][0] AllDevicesOff
[2023-01-05 10:55:13][0] AllDevicesOff
[2023-01-05 10:56:28][0] AllDevicesOff
[2023-01-05 10:56:41][0] AllDevicesOff
[2023-01-05 10:56:47][0] AllDevicesOff
[2023-01-05 10:56:48][0] AllDevicesOff
[2023-01-05 10:56:51][0] AllDevicesOff
[2023-01-05 10:57:24][0] AllDevicesOff
[2023-01-05 10:57:44][0] AllDevicesOff
[2023-01-05 10:58:45][0] AllDevicesOff
[2023-01-05 10:58:45][0] AllDevicesOff
[2023-01-05 10:58:45][0] AllDevicesOff
[2023-01-05 10:58:46][0] AllDevicesOff
[2023-01-05 10:58:48][0] AllDevicesOff
[2023-01-05 10:58:49][0] AllDevicesOff
[2023-01-05 10:58:49][0] AllDevicesOff
[2023-01-05 10:58:49][0] AllDevicesOff
[2023-01-05 10:58:54][0] AllDevicesOff
[2023-01-05 10:58:55][0] AllDevicesOff
[2023-01-05 10:59:29][0] AllDevicesOff
[2023-01-05 10:59:31][0] AllDevicesOff
[2023-01-05 10:59:32][0] AllDevicesOff
[2023-01-05 10:59:32][0] AllDevicesOff
[2023-01-05 10:59:32][0] AllDevicesOff
[2023-01-05 10:59:33][0] AllDevicesOff
[2023-01-05 10:59:33][0] AllDevicesOff
[2023-01-05 10:59:34][0] AllDevicesOff
[2023-01-05 10:59:35][0] AllDevicesOff
[2023-01-05 10:59:37][0] AllDevicesOff
[2023-01-05 10:59:43][0] AllDevicesOff
[2023-01-05 10:59:45][0] AllDevicesOff
[2023-01-05 10:59:46][0] AllDevicesOff
[2023-01-05 10:59:46][0] AllDevicesOff
[2023-01-05 10:59:46][0] AllDevicesOff
[2023-01-05 10:59:47][0] AllDevicesOff
[2023-01-05 10:59:47][0] AllDevicesOff
[2023-01-05 10:59:48][0] AllDevicesOff
[2023-01-05 10:59:50][0] AllDevicesOff
[2023-01-05 10:59:51][0] AllDevicesOff
[2023-01-05 10:59:51][0] AllDevicesOff
[2023-01-05 11:00:07][0] AllDevicesOff
[2023-01-05 11:00:07][0] AllDevicesOff
[2023-01-05 11:00:15][0] AllDevicesOff
[2023-01-05 11:14:06][0] restoreDeviceStates status:4
[2023-01-05 11:14:07][0] setDevicesActivity  E:0 P:0 T:0 D:0
[2023-01-05 11:14:07][0] save
[2023-01-05 11:14:07][0] 111[2023-01-05 11:17:36][0] restoreDeviceStates status:4
[2023-01-05 11:17:36][0] setDevicesActivity  E:0 P:0 T:0 D:0
[2023-01-05 11:17:36][0] save
[2023-01-05 11:17:36][0] 111[2023-01-05 11:19:41][0] restoreDeviceStates status:4
[2023-01-05 11:19:42][0] setDevicesActivity  E:0 P:0 T:0 D:0
[2023-01-05 11:19:42][0] save
[2023-01-05 11:19:42][0] 111[2023-01-05 11:21:05][0] AllDevicesOff
[2023-01-05 11:21:08][0] AllDevicesOff
[2023-01-05 11:21:19][0] AllDevicesOff
[2023-01-05 11:21:22][0] AllDevicesOff
[2023-01-05 11:21:29][0] AllDevicesOff
[2023-01-05 11:26:47][0] AllDevicesOff
[2023-01-05 11:26:51][0] AllDevicesOff
[2023-01-05 11:26:55][0] AllDevicesOff
[2023-01-05 11:27:45][0] AllDevicesOff
[2023-01-05 11:27:47][0] AllDevicesOff
[2023-01-05 11:27:48][0] AllDevicesOff
[2023-01-05 11:27:48][0] AllDevicesOff
[2023-01-05 11:27:49][0] AllDevicesOff
[2023-01-05 11:27:49][0] AllDevicesOff
[2023-01-05 11:28:03][0] AllDevicesOff
[2023-01-05 11:28:06][0] AllDevicesOff
[2023-01-05 11:28:06][0] AllDevicesOff
[2023-01-05 11:28:08][0] AllDevicesOff
[2023-01-05 11:28:08][0] AllDevicesOff
[2023-01-05 11:28:09][0] AllDevicesOff
[2023-01-05 11:28:13][0] AllDevicesOff
[2023-01-05 11:28:14][0] AllDevicesOff
[2023-01-05 11:28:15][0] AllDevicesOff
[2023-01-05 11:28:15][0] AllDevicesOff
[2023-01-05 11:28:31][0] AllDevicesOff
[2023-01-05 11:28:33][0] AllDevicesOff
[2023-01-05 11:28:34][0] AllDevicesOff
[2023-01-05 11:28:34][0] AllDevicesOff
[2023-01-05 11:28:57][0] AllDevicesOff
[2023-01-05 11:28:58][0] AllDevicesOff
[2023-01-05 11:28:59][0] AllDevicesOff
[2023-01-05 11:28:59][0] AllDevicesOff
[2023-01-05 11:28:59][0] AllDevicesOff
[2023-01-05 11:29:00][0] AllDevicesOff
[2023-01-05 11:29:00][0] AllDevicesOff
[2023-01-05 11:29:07][0] AllDevicesOff
[2023-01-05 11:29:08][0] AllDevicesOff
[2023-01-05 11:29:08][0] AllDevicesOff
[2023-01-05 11:29:13][0] AllDevicesOff
[2023-01-05 11:29:13][0] AllDevicesOff
[2023-01-05 11:29:14][0] AllDevicesOff
[2023-01-05 11:29:17][0] AllDevicesOff
[2023-01-05 11:29:18][0] AllDevicesOff
[2023-01-05 11:29:18][0] AllDevicesOff
[2023-01-05 11:29:21][0] AllDevicesOff
[2023-01-05 11:29:22][0] AllDevicesOff
[2023-01-05 11:29:22][0] AllDevicesOff
[2023-01-05 11:29:28][0] AllDevicesOff
[2023-01-05 11:29:40][0] AllDevicesOff
[2023-01-05 11:30:33][0] restoreDeviceStates status:4
[2023-01-05 11:30:34][0] setDevicesActivity  E:0 P:0 T:0 D:0
[2023-01-05 11:30:34][0] save
[2023-01-05 11:30:34][0] 111[2023-01-05 11:36:47][0] AllDevicesOff
[2023-01-05 11:39:32][0] AllDevicesOff
[2023-01-05 11:39:32][0] AllDevicesOff
[2023-01-05 11:39:33][0] AllDevicesOff
[2023-01-05 11:39:37][0] AllDevicesOff
[2023-01-05 11:39:37][0] AllDevicesOff
[2023-01-05 11:39:37][0] AllDevicesOff
[2023-01-05 11:39:47][0] AllDevicesOff
[2023-01-05 11:39:47][0] AllDevicesOff
[2023-01-05 11:40:21][0] AllDevicesOff
[2023-01-05 11:40:24][0] AllDevicesOff
[2023-01-05 11:40:25][0] AllDevicesOff
[2023-01-05 11:40:36][0] AllDevicesOff
[2023-01-05 11:40:36][0] AllDevicesOff
[2023-01-05 11:40:51][0] AllDevicesOff
[2023-01-05 11:40:51][0] AllDevicesOff
[2023-01-05 11:40:53][0] AllDevicesOff
[2023-01-05 11:40:57][0] AllDevicesOff
[2023-01-05 11:41:00][0] AllDevicesOff
[2023-01-05 11:41:00][0] AllDevicesOff
[2023-01-05 11:41:05][0] AllDevicesOff
[2023-01-05 11:41:08][0] AllDevicesOff
[2023-01-05 11:41:08][0] AllDevicesOff
[2023-01-05 11:41:12][0] AllDevicesOff
[2023-01-05 11:41:16][0] AllDevicesOff
[2023-01-05 11:41:20][0] AllDevicesOff
[2023-01-05 11:41:50][0] AllDevicesOff
[2023-01-05 11:41:51][0] AllDevicesOff
[2023-01-05 11:41:51][0] AllDevicesOff
[2023-01-05 11:41:52][0] AllDevicesOff
[2023-01-05 11:41:52][0] AllDevicesOff
[2023-01-05 11:41:52][0] AllDevicesOff
[2023-01-05 11:41:53][0] AllDevicesOff
[2023-01-05 11:41:53][0] AllDevicesOff
[2023-01-05 11:41:53][0] AllDevicesOff
[2023-01-05 11:41:55][0] AllDevicesOff
[2023-01-05 11:41:55][0] AllDevicesOff
[2023-01-05 11:41:55][0] AllDevicesOff
[2023-01-05 11:41:56][0] AllDevicesOff
[2023-01-05 11:41:56][0] AllDevicesOff
[2023-01-05 11:41:57][0] AllDevicesOff
[2023-01-05 11:41:57][0] AllDevicesOff
[2023-01-05 11:41:57][0] AllDevicesOff
[2023-01-05 11:41:58][0] AllDevicesOff
[2023-01-05 11:42:39][0] AllDevicesOff
[2023-01-05 11:42:39][0] AllDevicesOff
[2023-01-05 11:42:39][0] AllDevicesOff
[2023-01-05 11:42:40][0] AllDevicesOff
[2023-01-05 11:42:40][0] AllDevicesOff
[2023-01-05 11:42:40][0] AllDevicesOff
[2023-01-05 11:42:41][0] AllDevicesOff
[2023-01-05 11:42:41][0] AllDevicesOff
[2023-01-05 11:42:42][0] AllDevicesOff
[2023-01-05 11:42:42][0] AllDevicesOff
[2023-01-05 11:42:43][0] AllDevicesOff
[2023-01-05 11:42:43][0] AllDevicesOff
[2023-01-05 11:42:46][0] AllDevicesOff
[2023-01-05 11:42:46][0] AllDevicesOff
[2023-01-05 11:42:46][0] AllDevicesOff
[2023-01-05 11:42:47][0] AllDevicesOff
[2023-01-05 11:42:47][0] AllDevicesOff
[2023-01-05 11:42:48][0] AllDevicesOff
[2023-01-05 11:42:48][0] AllDevicesOff
[2023-01-05 11:42:48][0] AllDevicesOff
[2023-01-05 11:42:49][0] AllDevicesOff
[2023-01-05 11:42:49][0] AllDevicesOff
[2023-01-05 11:42:52][0] AllDevicesOff
[2023-01-05 11:42:53][0] AllDevicesOff
[2023-01-05 11:42:53][0] AllDevicesOff
[2023-01-05 11:42:53][0] AllDevicesOff
[2023-01-05 11:42:53][0] AllDevicesOff
[2023-01-05 11:42:54][0] AllDevicesOff
[2023-01-05 11:42:54][0] AllDevicesOff
[2023-01-05 11:42:57][0] AllDevicesOff
[2023-01-05 11:42:58][0] AllDevicesOff
[2023-01-05 11:42:58][0] AllDevicesOff
[2023-01-05 11:42:58][0] AllDevicesOff
[2023-01-05 11:42:59][0] AllDevicesOff
[2023-01-05 11:42:59][0] AllDevicesOff
[2023-01-05 11:42:59][0] AllDevicesOff
[2023-01-05 11:43:00][0] AllDevicesOff
[2023-01-05 11:43:01][0] AllDevicesOff
[2023-01-05 11:43:01][0] AllDevicesOff
[2023-01-05 11:43:02][0] AllDevicesOff
[2023-01-05 11:43:02][0] AllDevicesOff
[2023-01-05 11:43:02][0] AllDevicesOff
[2023-01-05 11:43:02][0] AllDevicesOff
[2023-01-05 11:43:08][0] AllDevicesOff
[2023-01-05 11:43:08][0] AllDevicesOff
[2023-01-05 11:43:09][0] AllDevicesOff
[2023-01-05 11:43:17][0] AllDevicesOff
[2023-01-05 11:43:40][0] AllDevicesOff
[2023-01-05 11:43:40][0] AllDevicesOff
[2023-01-05 11:43:40][0] AllDevicesOff
[2023-01-05 11:43:41][0] AllDevicesOff
[2023-01-05 11:43:41][0] AllDevicesOff
[2023-01-05 11:43:41][0] AllDevicesOff
[2023-01-05 11:43:42][0] AllDevicesOff
[2023-01-05 11:43:42][0] AllDevicesOff
[2023-01-05 11:43:42][0] AllDevicesOff
[2023-01-05 11:43:43][0] AllDevicesOff
[2023-01-05 11:43:47][0] AllDevicesOff
[2023-01-05 11:43:48][0] AllDevicesOff
[2023-01-05 11:43:48][0] AllDevicesOff
[2023-01-05 11:43:48][0] AllDevicesOff
[2023-01-05 11:43:49][0] AllDevicesOff
[2023-01-05 11:43:49][0] AllDevicesOff
[2023-01-05 11:43:49][0] AllDevicesOff
[2023-01-05 11:43:50][0] AllDevicesOff
[2023-01-05 11:43:50][0] AllDevicesOff
[2023-01-05 11:43:51][0] AllDevicesOff
[2023-01-05 11:44:04][0] AllDevicesOff
[2023-01-05 11:44:04][0] AllDevicesOff
[2023-01-05 11:44:05][0] AllDevicesOff
[2023-01-05 11:44:05][0] AllDevicesOff
[2023-01-05 11:44:05][0] AllDevicesOff
[2023-01-05 11:44:06][0] AllDevicesOff
[2023-01-05 11:44:06][0] AllDevicesOff
[2023-01-05 11:44:06][0] AllDevicesOff
[2023-01-05 11:44:07][0] AllDevicesOff
[2023-01-05 11:44:07][0] AllDevicesOff
[2023-01-05 11:44:21][0] AllDevicesOff
[2023-01-05 11:44:21][0] AllDevicesOff
[2023-01-05 11:44:21][0] AllDevicesOff
[2023-01-05 11:44:22][0] AllDevicesOff
[2023-01-05 11:44:22][0] AllDevicesOff
[2023-01-05 11:44:23][0] AllDevicesOff
[2023-01-05 11:44:23][0] AllDevicesOff
[2023-01-05 11:44:23][0] AllDevicesOff
[2023-01-05 11:44:24][0] AllDevicesOff
[2023-01-05 11:44:24][0] AllDevicesOff
[2023-01-05 11:44:26][0] AllDevicesOff
[2023-01-05 11:44:26][0] AllDevicesOff
[2023-01-05 11:44:27][0] AllDevicesOff
[2023-01-05 11:44:27][0] AllDevicesOff
[2023-01-05 11:44:27][0] AllDevicesOff
[2023-01-05 11:44:28][0] AllDevicesOff
[2023-01-05 11:44:28][0] AllDevicesOff
[2023-01-05 11:44:28][0] AllDevicesOff
[2023-01-05 11:44:29][0] AllDevicesOff
[2023-01-05 11:44:29][0] AllDevicesOff
[2023-01-05 11:44:30][0] AllDevicesOff
[2023-01-05 11:44:30][0] AllDevicesOff
[2023-01-05 11:44:30][0] AllDevicesOff
[2023-01-05 11:44:31][0] AllDevicesOff
[2023-01-05 11:44:38][0] AllDevicesOff
[2023-01-05 11:44:54][0] restoreDeviceStates status:4
[2023-01-05 11:44:55][0] setDevicesActivity  E:0 P:0 T:0 D:0
[2023-01-05 11:44:55][0] save
[2023-01-05 11:44:55][0] 111[2023-01-05 11:47:01][0] AllDevicesOff
[2023-01-05 11:54:54][0] AllDevicesOff
[2023-01-05 11:54:55][0] AllDevicesOff
[2023-01-05 11:54:56][0] AllDevicesOff
[2023-01-05 11:54:56][0] AllDevicesOff
[2023-01-05 11:54:56][0] AllDevicesOff
[2023-01-05 11:54:57][0] AllDevicesOff
[2023-01-05 11:54:57][0] AllDevicesOff
[2023-01-05 11:54:57][0] AllDevicesOff
[2023-01-05 11:54:58][0] AllDevicesOff
[2023-01-05 11:54:58][0] AllDevicesOff
[2023-01-05 11:55:01][0] AllDevicesOff
[2023-01-05 11:55:02][0] AllDevicesOff
[2023-01-05 11:55:02][0] AllDevicesOff
[2023-01-05 11:55:02][0] AllDevicesOff
[2023-01-05 11:55:06][0] AllDevicesOff
[2023-01-05 11:55:08][0] AllDevicesOff
[2023-01-05 11:55:08][0] AllDevicesOff
[2023-01-05 11:55:09][0] AllDevicesOff
[2023-01-05 11:55:14][0] AllDevicesOff
[2023-01-05 11:55:14][0] AllDevicesOff
[2023-01-05 11:55:15][0] AllDevicesOff
[2023-01-05 11:55:16][0] AllDevicesOff
[2023-01-05 11:55:16][0] AllDevicesOff
[2023-01-05 11:55:16][0] AllDevicesOff
[2023-01-05 11:55:25][0] AllDevicesOff
[2023-01-05 11:55:25][0] AllDevicesOff
[2023-01-05 11:55:26][0] AllDevicesOff
[2023-01-05 11:55:36][0] restoreDeviceStates status:4
[2023-01-05 11:55:36][0] setDevicesActivity  E:0 P:0 T:0 D:0
[2023-01-05 11:55:36][0] save
[2023-01-05 11:55:36][0] 111[2023-01-05 14:14:01][0] AllDevicesOff
[2023-01-05 14:14:01][0] AllDevicesOff
[2023-01-05 14:14:02][0] AllDevicesOff
[2023-01-05 14:14:04][0] AllDevicesOff
[2023-01-05 14:14:05][0] AllDevicesOff
[2023-01-05 14:14:05][0] AllDevicesOff
[2023-01-05 14:14:09][0] AllDevicesOff
[2023-01-05 14:14:10][0] AllDevicesOff
[2023-01-05 14:14:10][0] AllDevicesOff
[2023-01-05 14:14:11][0] AllDevicesOff
[2023-01-05 14:14:11][0] AllDevicesOff
[2023-01-05 14:14:12][0] AllDevicesOff
[2023-01-05 14:14:12][0] AllDevicesOff
[2023-01-05 14:14:12][0] AllDevicesOff
[2023-01-05 14:14:12][0] AllDevicesOff
[2023-01-05 14:14:13][0] AllDevicesOff
[2023-01-05 14:14:16][0] AllDevicesOff
[2023-01-05 14:14:17][0] AllDevicesOff
[2023-01-05 14:14:17][0] AllDevicesOff
[2023-01-05 14:14:18][0] AllDevicesOff
[2023-01-05 14:14:23][0] restoreDeviceStates status:4
[2023-01-05 14:14:24][0] setDevicesActivity  E:0 P:0 T:0 D:0
[2023-01-05 14:14:24][0] save
[2023-01-05 14:14:24][0] 111[2023-01-05 14:17:36][0] AllDevicesOff
[2023-01-05 14:17:36][0] AllDevicesOff
[2023-01-05 14:17:37][0] AllDevicesOff
[2023-01-05 14:17:38][0] AllDevicesOff
[2023-01-05 14:17:38][0] AllDevicesOff
[2023-01-05 14:17:40][0] AllDevicesOff
[2023-01-05 14:22:28][0] AllDevicesOff
[2023-01-05 14:22:28][0] AllDevicesOff
[2023-01-05 14:22:28][0] AllDevicesOff
[2023-01-05 14:22:30][0] AllDevicesOff
[2023-01-05 14:22:31][0] AllDevicesOff
[2023-01-05 14:22:32][0] AllDevicesOff
[2023-01-05 14:22:40][0] AllDevicesOff
[2023-01-05 14:22:41][0] AllDevicesOff
[2023-01-05 14:22:42][0] AllDevicesOff
[2023-01-05 14:22:43][0] AllDevicesOff
[2023-01-05 14:22:44][0] AllDevicesOff
[2023-01-05 14:23:07][0] AllDevicesOff
[2023-01-05 14:23:07][0] AllDevicesOff
[2023-01-05 14:23:08][0] AllDevicesOff
[2023-01-05 14:26:26][0] AllDevicesOff
[2023-01-05 14:26:30][0] AllDevicesOff
[2023-01-05 14:37:34][0] restoreDeviceStates status:3
[2023-01-05 14:37:34][0] setDevicesActivity  E:0 P:0 T:0 D:0
[2023-01-05 14:37:34][0] save
[2023-01-05 14:38:19][0] restoreDeviceStates status:3
[2023-01-05 14:38:19][0] setDevicesActivity  E:0 P:0 T:0 D:0
[2023-01-05 14:38:19][0] save
[2023-01-05 14:38:19][0] 111[2023-01-05 16:00:24][0] restoreDeviceStates status:7
[2023-01-05 16:00:25][0] setDevicesActivity  E:0 P:0 T:0 D:0
[2023-01-05 16:00:25][0] save
[2023-01-05 16:00:25][0] restoreDeviceStates status:7
[2023-01-05 16:00:25][0] setDevicesActivity  E:0 P:0 T:0 D:0
[2023-01-05 16:00:25][0] save
[2023-01-05 16:00:37][0] restoreDeviceStates status:4
[2023-01-05 16:00:37][0] setDevicesActivity  E:0 P:0 T:0 D:0
[2023-01-05 16:00:37][0] save
[2023-01-05 16:00:37][0] 111[2023-01-05 16:03:18][0] restoreDeviceStates status:4
[2023-01-05 16:03:18][0] setDevicesActivity  E:0 P:0 T:0 D:0
[2023-01-05 16:03:18][0] save
[2023-01-05 16:03:18][0] 111[2023-01-05 16:19:16][0] restoreDeviceStates status:4
[2023-01-05 16:19:17][0] setDevicesActivity  E:0 P:0 T:0 D:0
[2023-01-05 16:19:17][0] save
[2023-01-05 16:19:17][0] 111[2023-01-05 16:30:15][1] ----------------------------------------------------------------------------------------
[2023-01-05 16:30:15][1] <<< Render failure detected!!! >>>
Please check render statistics to solve the problem.
[2023-01-05 16:30:15][1] OCT: MB:0/0  ST/MOV:0/570  Nodes:1190  Tris:15.283m  DispTris:30.078m  Hairs:0  Meshes:25k
[2023-01-05 16:30:15][0]  MB:0/0  ST/MOV:0/570  Nodes:1190  Tris:15.283m  DispTris:30.078m  Hairs:0  Meshes:25k
[2023-01-05 16:30:15][1] OCT: Textures Grey8/16:2/0  Rgb32/64:20/1
[2023-01-05 16:30:15][0]  Textures Grey8/16:2/0  Rgb32/64:20/1
[2023-01-05 16:30:15][1] Device:0  [NVIDIA GeForce RTX 3060]<En>[PDT][8.6]  Memory used/free/total:4.037Gb/479Mb/12Gb  outOfCoreMem:9.287Gb  p2pBytesUsed:0  runtimeDataSize:1.097Gb temperature:35
[2023-01-05 16:30:15][1] ApiUpdate Wait:0 Caller:needApiUpdate tm:0.584ms.  [OK]
[2023-01-05 16:30:15][1] Render Failure...Please check Octane log window from 'Settings->Other->Show Render Log' button...[2023-01-05 16:30:24][1] <markMaterials>
[2023-01-05 16:30:24][1] <markMaterials>
[2023-01-05 16:30:24][1] >> ProcessUpdate(flags=CMOG cAsyn=1 dDelay=1
[2023-01-05 16:30:24][1] >> ProcessUpdate end res:0 tm:0.002ms. 
[2023-01-05 16:30:54][1] <markMaterials>
[2023-01-05 16:30:54][1] >> ProcessUpdate(flags=CM cAsyn=0 dDelay=1
[2023-01-05 16:30:54][1] >> ProcessUpdate end res:0 tm:0.002ms. 
[2023-01-05 16:31:03][1] <markMaterials>
[2023-01-05 16:31:03][1]   convert mat:[wood_sidingSVGDirty]  tip:1029501
[2023-01-05 16:31:03][0] AllDevicesOff
[2023-01-05 16:31:03][1] >> ProcessUpdate(flags=CM cAsyn=0 dDelay=1
[2023-01-05 16:31:03][1] >> ProcessUpdate end res:0 tm:0.002ms. 
[2023-01-05 16:31:04][1] <markMaterials>
[2023-01-05 16:31:04][1] >> ProcessUpdate(flags=CM cAsyn=0 dDelay=1
[2023-01-05 16:31:04][1] >> ProcessUpdate end res:0 tm:0.002ms. 
[2023-01-05 16:31:04][1] <markMaterials>
[2023-01-05 16:31:04][1] >> ProcessUpdate(flags=CM cAsyn=0 dDelay=1
[2023-01-05 16:31:04][1] >> ProcessUpdate end res:0 tm:0.002ms. 
[2023-01-05 16:31:05][1] LoadOslScripts()
[2023-01-05 16:31:05][1] <markMaterials>
[2023-01-05 16:31:05][1] >> ProcessUpdate(flags=M cAsyn=0 dDelay=0
[2023-01-05 16:31:05][1] >> ProcessUpdate end res:0 tm:0.002ms. 
[2023-01-05 16:31:05][1]   convert mat:[wood_sidingSVGDirty]  tip:1029501
[2023-01-05 16:31:05][0] AllDevicesOff
[2023-01-05 16:31:05][1] >> ProcessUpdate(flags=M cAsyn=0 dDelay=0
[2023-01-05 16:31:05][1] >> ProcessUpdate end res:0 tm:0.002ms. 
[2023-01-05 16:31:05][1] >> ProcessUpdate(flags=CM cAsyn=0 dDelay=1
[2023-01-05 16:31:05][1] >> ProcessUpdate end res:0 tm:0.002ms. 
[2023-01-05 16:31:05][1]   convert mat:[wood_sidingSVGDirty]  tip:1029501
[2023-01-05 16:31:05][0] AllDevicesOff
[2023-01-05 16:31:05][0] AllDevicesOff
[2023-01-05 16:31:05][1] Opengl preview:wood_sidingSVGDirty  size:256*256[2023-01-05 16:31:05][1]   convert mat:[wood_sidingSVGDirty]  tip:1029501
[2023-01-05 16:31:05][0] AllDevicesOff
[2023-01-05 16:31:05][1] <markMaterials>
[2023-01-05 16:31:05][0] AllDevicesOff
[2023-01-05 16:31:05][1] >> ProcessUpdate(flags=CM cAsyn=0 dDelay=1
[2023-01-05 16:31:05][1] >> ProcessUpdate end res:0 tm:0.002ms. 
[2023-01-05 16:31:05][0] AllDevicesOff
[2023-01-05 16:31:05][0] AllDevicesOff
[2023-01-05 16:31:05][0] AllDevicesOff
[2023-01-05 16:31:06][0] AllDevicesOff
[2023-01-05 16:31:06][1] <markMaterials>
[2023-01-05 16:31:06][1] >> ProcessUpdate(flags=CM cAsyn=0 dDelay=1
[2023-01-05 16:31:06][1] >> ProcessUpdate end res:0 tm:0.002ms. 
[2023-01-05 16:31:06][1] Opengl preview:wood_sidingSVGDirty  size:256*256[2023-01-05 16:31:06][1] <markMaterials>
[2023-01-05 16:31:06][1] >> ProcessUpdate(flags=CM cAsyn=0 dDelay=1
[2023-01-05 16:31:06][1] >> ProcessUpdate end res:0 tm:0.001ms. 
[2023-01-05 16:31:06][1] Opengl preview:wood_sidingSVGDirty  size:256*256[2023-01-05 16:31:06][1] <markMaterials>
[2023-01-05 16:31:06][1] >> ProcessUpdate(flags=CM cAsyn=0 dDelay=1
[2023-01-05 16:31:06][1] >> ProcessUpdate end res:0 tm:0.002ms. 
[2023-01-05 16:31:08][1] <markMaterials>
[2023-01-05 16:31:08][1] >> ProcessUpdate(flags=CM cAsyn=0 dDelay=1
[2023-01-05 16:31:08][1] >> ProcessUpdate end res:0 tm:0.002ms. 
[2023-01-05 16:31:13][1] >> ProcessUpdate(flags=M cAsyn=0 dDelay=0
[2023-01-05 16:31:13][1] >> ProcessUpdate end res:0 tm:0.002ms. 
[2023-01-05 16:31:13][1] >> ProcessUpdate(flags=M cAsyn=0 dDelay=0
[2023-01-05 16:31:13][1] >> ProcessUpdate end res:0 tm:0.001ms. 
[2023-01-05 16:31:13][1] >> ProcessUpdate(flags=M cAsyn=0 dDelay=0
[2023-01-05 16:31:13][1] >> ProcessUpdate end res:0 tm:0.001ms. 
[2023-01-05 16:31:13][1] >> ProcessUpdate(flags=M cAsyn=0 dDelay=0
[2023-01-05 16:31:13][1] >> ProcessUpdate end res:0 tm:0.001ms. 
[2023-01-05 16:31:13][1] >> ProcessUpdate(flags=M cAsyn=0 dDelay=0
[2023-01-05 16:31:13][1] >> ProcessUpdate end res:0 tm:0.002ms. 
[2023-01-05 16:31:13][1] >> ProcessUpdate(flags=M cAsyn=0 dDelay=0
[2023-01-05 16:31:13][1] >> ProcessUpdate end res:0 tm:0.001ms. 
[2023-01-05 16:31:13][1] <markMaterials>
[2023-01-05 16:31:13][1]   convert mat:[wood_sidingSVGDirty]  tip:1029501
[2023-01-05 16:31:13][0] AllDevicesOff
[2023-01-05 16:31:13][1] >> ProcessUpdate(flags=M cAsyn=0 dDelay=0
[2023-01-05 16:31:13][1] >> ProcessUpdate end res:0 tm:0.005ms. 
[2023-01-05 16:31:13][1] >> ProcessUpdate(flags=CM cAsyn=0 dDelay=1
[2023-01-05 16:31:13][1] >> ProcessUpdate end res:0 tm:0.002ms. 
[2023-01-05 16:31:13][1]   convert mat:[wood_sidingSVGDirty]  tip:1029501
[2023-01-05 16:31:13][0] AllDevicesOff
[2023-01-05 16:31:13][0] AllDevicesOff
[2023-01-05 16:31:13][1] Opengl preview:wood_sidingSVGDirty  size:256*256[2023-01-05 16:31:13][1]   convert mat:[wood_sidingSVGDirty]  tip:1029501
[2023-01-05 16:31:13][1] <markMaterials>
[2023-01-05 16:31:13][1] >> ProcessUpdate(flags=CM cAsyn=0 dDelay=1
[2023-01-05 16:31:13][1] >> ProcessUpdate end res:0 tm:0.001ms. 
[2023-01-05 16:31:14][0] AllDevicesOff
[2023-01-05 16:31:14][1] <markMaterials>
[2023-01-05 16:31:14][1] >> ProcessUpdate(flags=CM cAsyn=0 dDelay=1
[2023-01-05 16:31:14][1] >> ProcessUpdate end res:0 tm:0.002ms. 
[2023-01-05 16:31:14][1] Opengl preview:wood_sidingSVGDirty  size:256*256[2023-01-05 16:31:14][1] <markMaterials>
[2023-01-05 16:31:14][1] >> ProcessUpdate(flags=CM cAsyn=0 dDelay=1
[2023-01-05 16:31:14][1] >> ProcessUpdate end res:0 tm:0.002ms. 
[2023-01-05 16:31:14][1] Opengl preview:wood_sidingSVGDirty  size:256*256[2023-01-05 16:31:14][1] <markMaterials>
[2023-01-05 16:31:14][1] >> ProcessUpdate(flags=CM cAsyn=0 dDelay=1
[2023-01-05 16:31:14][1] >> ProcessUpdate end res:0 tm:0.002ms. 
[2023-01-05 16:31:16][1] NE AskClose OK
[2023-01-05 16:31:16][1] closeNodeviewDialog[2023-01-05 16:31:16][1] NE deleteAll OK
 [OK]
[2023-01-05 16:31:17][1] <markMaterials>
[2023-01-05 16:31:17][1] <markMaterials>
[2023-01-05 16:31:17][1] >> ProcessUpdate(flags=CM cAsyn=0 dDelay=1
[2023-01-05 16:31:17][1] >> ProcessUpdate end res:0 tm:0.002ms. 
[2023-01-05 16:31:18][1] >> ProcessUpdate(flags=CM cAsyn=0 dDelay=1
[2023-01-05 16:31:18][1] >> ProcessUpdate end res:0 tm:0.002ms. 
[2023-01-05 16:31:18][1] <markMaterials>
[2023-01-05 16:31:18][1] >> ProcessUpdate(flags=CM cAsyn=0 dDelay=1
[2023-01-05 16:31:18][1] >> ProcessUpdate end res:0 tm:0.002ms. 
[2023-01-05 16:31:19][1] <markMaterials>
[2023-01-05 16:31:19][1] <markMaterials>
[2023-01-05 16:31:19][1] >> ProcessUpdate(flags=CM cAsyn=0 dDelay=1
[2023-01-05 16:31:19][1] >> ProcessUpdate end res:0 tm:0.002ms. 
[2023-01-05 16:31:21][1] LoadOslScripts()
[2023-01-05 16:31:21][1] <markMaterials>
[2023-01-05 16:31:21][1] >> ProcessUpdate(flags=M cAsyn=0 dDelay=0
[2023-01-05 16:31:21][1] >> ProcessUpdate end res:0 tm:0.001ms. 
[2023-01-05 16:31:21][1]   convert mat:[wood_sidingSVGDirty]  tip:1029501
[2023-01-05 16:31:21][0] AllDevicesOff
[2023-01-05 16:31:21][1] >> ProcessUpdate(flags=M cAsyn=0 dDelay=0
[2023-01-05 16:31:21][1] >> ProcessUpdate end res:0 tm:0.002ms. 
[2023-01-05 16:31:21][1]   convert mat:[wood_sidingSVGDirty]  tip:1029501
[2023-01-05 16:31:21][1] >> ProcessUpdate(flags=CM cAsyn=0 dDelay=1
[2023-01-05 16:31:21][1] >> ProcessUpdate end res:0 tm:0.001ms. 
[2023-01-05 16:31:21][0] AllDevicesOff
[2023-01-05 16:31:21][0] AllDevicesOff
[2023-01-05 16:31:21][0] AllDevicesOff
[2023-01-05 16:31:21][0] AllDevicesOff
[2023-01-05 16:31:21][0] AllDevicesOff
[2023-01-05 16:31:21][0] AllDevicesOff
[2023-01-05 16:31:21][0] AllDevicesOff
[2023-01-05 16:31:21][1] Opengl preview:wood_sidingSVGDirty  size:256*256[2023-01-05 16:31:21][1]   convert mat:[wood_sidingSVGDirty]  tip:1029501
[2023-01-05 16:31:21][1] <markMaterials>
[2023-01-05 16:31:21][1] >> ProcessUpdate(flags=CM cAsyn=0 dDelay=1
[2023-01-05 16:31:21][1] >> ProcessUpdate end res:0 tm:0.002ms. 
[2023-01-05 16:31:22][0] AllDevicesOff
[2023-01-05 16:31:22][1] <markMaterials>
[2023-01-05 16:31:22][1] >> ProcessUpdate(flags=CM cAsyn=0 dDelay=1
[2023-01-05 16:31:22][1] >> ProcessUpdate end res:0 tm:0.002ms. 
[2023-01-05 16:31:22][1] Opengl preview:wood_sidingSVGDirty  size:256*256[2023-01-05 16:31:22][1] <markMaterials>
[2023-01-05 16:31:22][1] >> ProcessUpdate(flags=CM cAsyn=0 dDelay=1
[2023-01-05 16:31:22][1] >> ProcessUpdate end res:0 tm:0.002ms. 
[2023-01-05 16:31:22][1] Opengl preview:wood_sidingSVGDirty  size:256*256[2023-01-05 16:31:22][1] <markMaterials>
[2023-01-05 16:31:22][1] >> ProcessUpdate(flags=CM cAsyn=0 dDelay=1
[2023-01-05 16:31:22][1] >> ProcessUpdate end res:0 tm:0.002ms. 
[2023-01-05 16:31:28][1] >> ProcessUpdate(flags=M cAsyn=0 dDelay=0
[2023-01-05 16:31:28][1] >> ProcessUpdate end res:0 tm:0.002ms. 
[2023-01-05 16:31:28][1] <markMaterials>
[2023-01-05 16:31:28][1]   convert mat:[wood_sidingSVGDirty]  tip:1029501
[2023-01-05 16:31:28][0] AllDevicesOff
[2023-01-05 16:31:28][1] >> ProcessUpdate(flags=M cAsyn=0 dDelay=0
[2023-01-05 16:31:28][1] >> ProcessUpdate end res:0 tm:0.001ms. 
[2023-01-05 16:31:28][1] >> ProcessUpdate(flags=CM cAsyn=0 dDelay=1
[2023-01-05 16:31:28][1] >> ProcessUpdate end res:0 tm:0.002ms. 
[2023-01-05 16:31:28][1]   convert mat:[wood_sidingSVGDirty]  tip:1029501
[2023-01-05 16:31:28][0] AllDevicesOff
[2023-01-05 16:31:28][0] AllDevicesOff
[2023-01-05 16:31:28][0] AllDevicesOff
[2023-01-05 16:31:28][1] Opengl preview:wood_sidingSVGDirty  size:256*256[2023-01-05 16:31:28][1]   convert mat:[wood_sidingSVGDirty]  tip:1029501
[2023-01-05 16:31:28][1] <markMaterials>
[2023-01-05 16:31:28][1] >> ProcessUpdate(flags=CM cAsyn=0 dDelay=1
[2023-01-05 16:31:28][1] >> ProcessUpdate end res:0 tm:0.002ms. 
[2023-01-05 16:31:29][0] AllDevicesOff
[2023-01-05 16:31:29][1] <markMaterials>
[2023-01-05 16:31:29][1] >> ProcessUpdate(flags=CM cAsyn=0 dDelay=1
[2023-01-05 16:31:29][1] >> ProcessUpdate end res:0 tm:0.002ms. 
[2023-01-05 16:31:29][1] Opengl preview:wood_sidingSVGDirty  size:256*256[2023-01-05 16:31:29][1] <markMaterials>
[2023-01-05 16:31:29][1] >> ProcessUpdate(flags=CM cAsyn=0 dDelay=1
[2023-01-05 16:31:29][1] >> ProcessUpdate end res:0 tm:0.002ms. 
[2023-01-05 16:31:29][1] Opengl preview:wood_sidingSVGDirty  size:256*256[2023-01-05 16:31:29][1] <markMaterials>
[2023-01-05 16:31:29][1] >> ProcessUpdate(flags=CM cAsyn=0 dDelay=1
[2023-01-05 16:31:29][1] >> ProcessUpdate end res:0 tm:0.002ms. 
[2023-01-05 16:31:31][1] NE AskClose OK
[2023-01-05 16:31:31][1] closeNodeviewDialog[2023-01-05 16:31:31][1] NE deleteAll OK
 [OK]
[2023-01-05 16:31:32][1] <markMaterials>
[2023-01-05 16:31:32][1] <markMaterials>
[2023-01-05 16:31:32][1] >> ProcessUpdate(flags=CM cAsyn=0 dDelay=1
[2023-01-05 16:31:32][1] >> ProcessUpdate end res:0 tm:0.002ms. 
[2023-01-05 16:31:36][1] setInteractive(0)[OK]  [OK]
[2023-01-05 16:31:36][0] restoreDeviceStates status:4
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to destroy CUDA event
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to destroy CUDA event
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to destroy CUDA event
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate pinned memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate pinned memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device array
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device array
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device array
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to destroy texture objects
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to destroy texture objects
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to destroy texture objects
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to destroy texture objects
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to destroy texture objects
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:Failed to free page-locked memory
[2023-01-05 16:31:36][1] OCT:Failed to free page-locked memory
[2023-01-05 16:31:36][1] OCT:Failed to free page-locked memory
[2023-01-05 16:31:36][1] OCT:Failed to free page-locked memory
[2023-01-05 16:31:36][1] OCT:Failed to free page-locked memory
[2023-01-05 16:31:36][1] OCT:Failed to free page-locked memory
[2023-01-05 16:31:36][1] OCT:Failed to free page-locked memory
[2023-01-05 16:31:36][1] OCT:Failed to free page-locked memory
[2023-01-05 16:31:36][1] OCT:Failed to free page-locked memory
[2023-01-05 16:31:36][1] OCT:Failed to free page-locked memory
[2023-01-05 16:31:36][1] OCT:Failed to free page-locked memory
[2023-01-05 16:31:36][1] OCT:Failed to free page-locked memory
[2023-01-05 16:31:36][1] OCT:Failed to free page-locked memory
[2023-01-05 16:31:36][1] OCT:Failed to free page-locked memory
[2023-01-05 16:31:36][1] OCT:Failed to free page-locked memory
[2023-01-05 16:31:36][1] OCT:Failed to free page-locked memory
[2023-01-05 16:31:36][1] OCT:Failed to free page-locked memory
[2023-01-05 16:31:36][1] OCT:Failed to free page-locked memory
[2023-01-05 16:31:36][1] OCT:Failed to free page-locked memory
[2023-01-05 16:31:36][1] OCT:Failed to free page-locked memory
[2023-01-05 16:31:36][1] OCT:Failed to free page-locked memory
[2023-01-05 16:31:36][1] OCT:Failed to free page-locked memory
[2023-01-05 16:31:36][1] OCT:Failed to free page-locked memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to destroy texture objects
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device array
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to destroy texture objects
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to destroy texture objects
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to destroy texture objects
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to destroy texture objects
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to destroy texture objects
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to destroy texture objects
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to destroy texture objects
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to destroy texture objects
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device array
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device array
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device array
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device array
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device array
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device array
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device array
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device array
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> could not get memory info
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:36][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:36][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:Failed to free page-locked memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:Failed to free page-locked memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:Failed to free page-locked memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:Failed to free page-locked memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:Failed to free page-locked memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:Failed to free page-locked memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:Failed to free page-locked memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:Failed to free page-locked memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:Failed to free page-locked memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:Failed to free page-locked memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:Failed to free page-locked memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:Failed to free page-locked memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:Failed to free page-locked memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:Failed to free page-locked memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:Failed to free page-locked memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:Failed to free page-locked memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:Failed to free page-locked memory
[2023-01-05 16:31:37][1] OCT:Failed to free page-locked memory
[2023-01-05 16:31:37][1] OCT:Failed to free page-locked memory
[2023-01-05 16:31:37][1] OCT:Failed to free page-locked memory
[2023-01-05 16:31:37][1] OCT:Failed to free page-locked memory
[2023-01-05 16:31:37][1] OCT:Failed to free page-locked memory
[2023-01-05 16:31:37][1] OCT:Failed to free page-locked memory
[2023-01-05 16:31:37][1] OCT:Failed to free page-locked memory
[2023-01-05 16:31:37][1] OCT:Failed to free page-locked memory
[2023-01-05 16:31:37][1] OCT:Failed to free page-locked memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:Failed to free page-locked memory
[2023-01-05 16:31:37][1] OCT:Failed to free page-locked memory
[2023-01-05 16:31:37][1] OCT:Failed to free page-locked memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:Failed to free page-locked memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:Failed to free page-locked memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:Failed to free page-locked memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:Failed to free page-locked memory
[2023-01-05 16:31:37][1] OCT:Failed to free page-locked memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:Failed to free page-locked memory
[2023-01-05 16:31:37][1] OCT:Failed to free page-locked memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:Failed to free page-locked memory
[2023-01-05 16:31:37][1] OCT:Failed to free page-locked memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:Failed to free page-locked memory
[2023-01-05 16:31:37][1] OCT:Failed to free page-locked memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:Failed to free page-locked memory
[2023-01-05 16:31:37][1] OCT:Failed to free page-locked memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:Failed to free page-locked memory
[2023-01-05 16:31:37][1] OCT:Failed to free page-locked memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to unload module
[2023-01-05 16:31:37][1] OCT:OptiX error for device 0: [ERROR] Error synching on OptixPipeline event (CUDA error string: an illegal memory access was encountered, CUDA error code: 700)
Error destroying OptixPipeline event (CUDA error string: an illegal memory access was encountered, CUDA error code: 700)
[2023-01-05 16:31:37][1] OCT:OptiX API error for device 0: CUDA error
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:OptiX error for device 0: [ERROR] Error synching on OptixPipeline event (CUDA error string: an illegal memory access was encountered, CUDA error code: 700)
Error destroying OptixPipeline event (CUDA error string: an illegal memory access was encountered, CUDA error code: 700)
[2023-01-05 16:31:37][1] OCT:OptiX API error for device 0: CUDA error
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:OptiX error for device 0: [ERROR] Error synching on OptixPipeline event (CUDA error string: an illegal memory access was encountered, CUDA error code: 700)
Error destroying OptixPipeline event (CUDA error string: an illegal memory access was encountered, CUDA error code: 700)
[2023-01-05 16:31:37][1] OCT:OptiX API error for device 0: CUDA error
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:OptiX error for device 0: [ERROR] Error synching on OptixPipeline event (CUDA error string: an illegal memory access was encountered, CUDA error code: 700)
Error destroying OptixPipeline event (CUDA error string: an illegal memory access was encountered, CUDA error code: 700)
[2023-01-05 16:31:37][1] OCT:OptiX API error for device 0: CUDA error
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:OptiX error for device 0: [ERROR] Error synching on OptixPipeline event (CUDA error string: an illegal memory access was encountered, CUDA error code: 700)
Error destroying OptixPipeline event (CUDA error string: an illegal memory access was encountered, CUDA error code: 700)
[2023-01-05 16:31:37][1] OCT:OptiX API error for device 0: CUDA error
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:OptiX error for device 0: [ERROR] Failed to destroy launch resources (CUDA error string: an illegal memory access was encountered, CUDA error code: 700)
[2023-01-05 16:31:37][1] OCT:OptiX API error for device 0: CUDA error
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to destroy CUDA event
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to destroy CUDA event
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to destroy CUDA event
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to destroy CUDA event
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to destroy CUDA event
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to destroy CUDA event
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to destroy CUDA event
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to destroy CUDA event
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to destroy CUDA event
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to destroy CUDA event
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to destroy CUDA event
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to destroy CUDA event
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to destroy CUDA event
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to destroy CUDA event
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to destroy CUDA event
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to destroy CUDA event
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to destroy CUDA event
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to destroy CUDA event
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to destroy CUDA event
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to destroy CUDA event
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to destroy CUDA event
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to destroy CUDA event
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to destroy CUDA event
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to destroy CUDA event
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to destroy CUDA event
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to destroy CUDA event
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to destroy CUDA event
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to destroy CUDA event
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to destroy CUDA event
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to destroy CUDA event
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to destroy CUDA event
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to destroy CUDA event
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> failed to deallocate device memory
[2023-01-05 16:31:37][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:37][1] OCT:  -> could not get memory info
[2023-01-05 16:31:52][0] setDevicesActivity  E:0 P:0 T:0 D:0
[2023-01-05 16:31:52][0] save
[2023-01-05 16:31:52][1] OCT:CUDA error 700 on device 0: an illegal memory access was encountered
[2023-01-05 16:31:52][1] OCT:  -> failed to bind device to current thread
[2023-01-05 16:31:52][1] OCT:device 0: failed to initialize context
[2023-01-05 16:31:52][1] OCT:device 0: failed to initialize render thread 000001AE83702130
[2023-01-05 16:31:53][0] 111[2023-01-05 16:31:53][1] initOctaneLive()
[2023-01-05 16:31:53][1] Sending scene data.....[2023-01-05 16:31:53][1] OCT:--------------------------- EXPORT LOG ---------------------------
[2023-01-05 16:31:53][0] --------------------------- EXPORT LOG ---------------------------
[2023-01-05 16:31:53][1] ResetAndInit 1
[2023-01-05 16:31:53][1] OCT:CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
[2023-01-05 16:31:53][1] OCT:denoiserThread0 : Exited with a error. Restart required
[2023-01-05 16:31:53][1] ----------------------------------------------------------------------------------------
[2023-01-05 16:31:53][1] <<< Render failure detected!!! >>>
Please check render statistics to solve the problem.
[2023-01-05 16:31:53][1] OCT: MB:0/0  ST/MOV:0/570  Nodes:1191  Tris:0  DispTris:0  Hairs:0  Meshes:0
[2023-01-05 16:31:53][0]  MB:0/0  ST/MOV:0/570  Nodes:1191  Tris:0  DispTris:0  Hairs:0  Meshes:0
[2023-01-05 16:31:53][1] OCT: Textures Grey8/16:0/0  Rgb32/64:0/0
[2023-01-05 16:31:53][0]  Textures Grey8/16:0/0  Rgb32/64:0/0
[2023-01-05 16:31:53][1] Device:0  [NVIDIA GeForce RTX 3060]<En>[PDT][8.6]  Memory used/free/total:0Kb/479Mb/12Gb  outOfCoreMem:0Kb  p2pBytesUsed:0  runtimeDataSize:0Kb temperature:32
[2023-01-05 16:31:53][1] RESET PROJECT[2023-01-05 16:31:53][1] Init defaults[2023-01-05 16:31:53][1] createOctaneDefaults [2023-01-05 16:31:53][1] ResetCaches [OK]
[2023-01-05 16:31:53][1] ResetAndInit [OK]

